Azure Event Hubs for real-time data ingestion

Posted on by Panther

Azure Event Hubs is a cloud-based event processing service that enables real-time data ingestion and processing in the Azure environment. It provides a scalable and reliable platform for collecting and analyzing streaming data from multiple sources.

In this tutorial, we will walk you through the steps to configure and use Azure Event Hubs for real-time data ingestion. We will cover the following topics:

  • Creating an Azure Event Hubs namespace
  • Configuring Event Hubs and its components
  • Creating an Event Hub
  • Sending data to Event Hub
  • Receiving and processing data from Event Hub
  • Monitoring and troubleshooting Event Hubs

Prerequisites

  • An active Microsoft Azure account
  • A basic understanding of Azure services
  • Microsoft Visual Studio or Azure Functions CLI

Creating an Azure Event Hubs Namespace

The first step to using Azure Event Hubs is to create an event hubs namespace. A namespace is a container that holds related event hubs, and it provides a scope for assigning management policies and tracking usage metrics.

To create a namespace for Azure Event Hubs, follow the steps below:

  1. Log in to your Azure portal account and navigate to the Azure Event Hubs service.

  2. In the Event Hubs service, click the “Add” button to create a new Event Hubs instance.

  3. In the “Create Namespace” dialog box, provide a unique name for your namespace.

  4. Select the pricing tier, which determines the number of event hubs you can create in this namespace.

  5. Choose the subscription and resource group for your namespace, and select the region where you want to deploy the namespace.

  6. Click the “Create” button to create your new namespace.

Once the namespace is created, you can configure it and create an event hub within it.

Configuring Event Hubs and its Components

After creating the Event Hubs namespace, you need to configure it and its related components to start ingesting and processing data. The primary components of Event Hubs are:

  • Event Hubs
  • Event Hub policies
  • Event Hub consumers
  • Event Hub partitions

Let’s look at each of these components in detail.

Event Hubs

An Event Hub is a high-throughput data streaming platform that can accept and process millions of events per second. Each Event Hub instance consists of one or more partitions, which are the logical containers that can store a subset of the event data.

To create an Event Hub within your namespace, follow the steps below:

  1. Navigate to the Azure portal and open your Event Hubs namespace.

  2. In the namespace dashboard, click on the “+ Event Hub” button to create a new Event Hub instance.

  3. Provide a unique name for your Event Hub, select the number of partitions you want to create and a retention policy.

  4. Click the create button to create your new Event Hub.

Event Hub Policies

Event Hub policies are used to manage access to Event Hubs resources. Policies define the permissions that are granted to specific users, groups, or applications that access the Event Hub.

To create an Event Hub policy, follow the steps below:

  1. Open the Namespace dashboard in your Azure portal and select the Event Hub for which you want to create a policy.

  2. In the Event Hub dashboard, select the “Shared access policies” option from the menu, then click the “Add” button.

  3. Provide a unique name for your new policy and select the appropriate permissions you want to grant.

  4. Click the “Create” button to create your new policy.

Event Hub Consumers

Event Hub consumers are responsible for reading and processing data from the Event Hub partitions. You can use various methods to read data from Event Hubs, including:

  • Azure Functions
  • Azure Stream Analytics
  • Azure Logic Apps
  • Azure Data Factory

In this tutorial, we will use Azure Functions to consume data from Event Hubs.

Event Hub Partitions

Event Hub partitions are the logical containers in which data is stored. Each partition is an ordered sequence of events that is stored independently and separately from other partitions. This architecture allows for greater scalability since each partition can be processed independently.

Creating an Azure Function to Send Data to Event Hub

The first step in ingesting data into Azure Event Hubs is to create an Azure Function that will send data to the Event Hub. We will be using Visual Studio to create and deploy our Azure Function.

To create an Azure Function, follow the steps below:

  1. Open Visual Studio and create a new project.

  2. Select “Cloud” from the project types, and then select “Azure Functions.”

  3. Select the “Azure Functions v3” template and provide a name for your project.

  4. Click on the “Create” button to create your new project.

  5. Wait for Visual Studio to create your Azure Function project template.

  6. Open the “Function1.cs” file and replace the code with the following:

    using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
using Microsoft.Azure.EventHubs;
using System.Text;
using System;

public static class Function1
{
   [FunctionName("Function1")]
   public static async System.Threading.Tasks.Task RunAsync([TimerTrigger("*/5 * * * * *")]TimerInfo myTimer,
       ILogger log)
   {
       var connectionStringBuilder = new EventHubsConnectionStringBuilder(Environment.GetEnvironmentVariable("EventHubConnectionString"))
       {
           EntityPath = Environment.GetEnvironmentVariable("EventHubName")
       };
       var eventHubClient = EventHubClient.CreateFromConnectionString(connectionStringBuilder.ToString());

       try
       {
           log.LogInformation($"Sending message: {myTimer}");
           await eventHubClient.SendAsync(new EventData(Encoding.UTF8.GetBytes(myTimer.ToString())));
       }
       catch (Exception e)
       {
           log.LogError($"Error: {e.Message}");
       }
   }
}
  7. In the “local.settings.json” file, add the following configuration settings: 
    {
   "IsEncrypted": false,
   "Values": {
       "AzureWebJobsStorage": "",
       "FUNCTIONS_WORKER_RUNTIME": "dotnet",
       "EventHubConnectionString": "<your-event-hub-connection-string>",
       "EventHubName": "<your-event-hub-name>"
   }
}
  8. Replace your-event-hub-connection-string and your-event-hub-name with the appropriate values.

  9. Build and run your Azure function.

Once your Azure Function is running, it will send messages to your Event Hub every 5 seconds.

Creating an Azure Function to Consume Data from Event Hub

The next step is to create an Azure Function that will consume data from the Event Hub and process it. For this tutorial, we will be using Visual Studio to create and deploy the Azure Function.

To create an Azure Function to consume the data, follow these steps:

  1. Open your Azure Function project created in the previous step.

  2. Right-click on the project in the Solution Explorer and select “Add” -> “New Item.”

  3. In the dialog box, select “Azure Function” and give it a name.

  4. Under “Templates,” select the “Event Hub trigger” option.

  5. Click the “Add” button to create your new function.

  6. In your new function, replace the existing code with the following:

    using System.Threading.Tasks;
using Microsoft.Azure.EventHubs;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;

public static class Function2
{
   [FunctionName("Function2")]
   public static async Task RunAsync([EventHubTrigger("<your-event-hub-name>", Connection = "EventHubConnectionAppSetting")] EventData[] events, ILogger log)
   {
       foreach (EventData eventData in events)
       {
           string messageBody = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
           log.LogInformation($"C# Event Hub trigger function processed a message: {messageBody}");
       }
   }
}
  7. In the “local.settings.json” file, add the following configuration settings: 
    {
   "IsEncrypted": false,
   "Values": {
       "AzureWebJobsStorage": "",
       "FUNCTIONS_WORKER_RUNTIME": "dotnet",
       "EventHubConnectionAppSetting": "<your-event-hub-connection-string>",
       "EventHubName": "<your-event-hub-name>"
   }
}
  8. Replace your-event-hub-connection-string and your-event-hub-name with the appropriate values.

  9. Build and run your Azure Function.

Once your Azure Function is running, it will consume the messages from your Event Hub and log their contents.

Monitoring and Troubleshooting Event Hubs

To monitor and troubleshoot your Event Hubs, you can use the Azure Portal or Azure Monitor. With Azure Monitor, you can view metrics and logs for Event Hub operations, such as the number of messages ingested, the number of partition reads and writes, and the error count.

To view monitoring data for your Event Hub, follow these steps:

  1. Open your Event Hub in the Azure portal.

  2. Click on the “Metrics” tab to view the metrics dashboard for your Event Hub.

  3. You can select different metrics to view, such as “Incoming Messages,” “Outgoing Messages,” and “Request Errors.”

  4. Click on the “Logs” tab to view the logs for your Event Hub.

  5. You can use Kusto Query Language (KQL) to query and analyze your logs. For example, you can query for specific error messages or trends in your data.

Conclusion

In this tutorial, we showed you how to use Azure Event Hubs for real-time data ingestion. We covered the basics of creating an Event Hubs namespace, configuring Event Hubs and its components, and creating Azure Functions to send and consume data. We also provided guidance on monitoring and troubleshooting your Event Hubs.

With Azure Event Hubs, you have a scalable and reliable platform for collecting and analyzing your streaming data. You can use it to build real-time applications that can process millions of events per second.

Related Post