This course teaches you how to manage application and network services in the Azure ecosystem.
By the end of this course, you'll have gained a firm understanding of the key components that comprise the Azure application and network services ecosystem. Ideally, you will achieve the following learning objectives:
- How to use Active Directory.
- How to understand networking strategies for Azure and communication services.
- How to use Redis cache.
This course is intended for individuals who wish to pursue the Azure 70-532 certification.
You should have work experience with Azure and general cloud computing knowledge.
This Course Includes
- 1 hour and 10 minutes of high-definition video.
- Expert-led instruction and exploration of important concepts surrounding Azure application and network services.
What You Will Learn
- How to utilize Azure Active Directory.
- How to implement Azure communication strategies.
- How to take advantage of Redis caching.
Hello and welcome back. Let's now take a look at Event Hubs which are a major azure service and critical knowledge for this certification path.
Azure Event Hubs are a highly a highly scalable publish subscribe service that can handle millions of events per second and make them available to multiple applications. Event Hubs allow us to process, reprocess and analyse massive amounts of data and respond to variable loads and spikes. Unlike queues, Event Hubs do not have a concept of message time to live, transactions or dead-letter queues. Event Hubs focus on providing a low latency, reliable and durable event storing system with the ability to replay data for processing and analysis.
This diagram is a simple view on Event Hubs. Multiple senders, for example thousands of mobile applications, can publish event messages to the Event Hub. When these messages arrive at the Event Hub, they're stored in one of the partitions which exist to distribute the message load and therefore improve scalability.
The order in which events arrive in the Event Hub are preserved in each partition. Events with the same partition key are delivered in order to the same partition. At the current time, the current minimum partition count being two, with a maximum of 32. When reading data from the Event Hub, receivers connect directly to the individual partitions. Multiple clients can read from the same partitions and a single client can read from multiple partitions by creating connections to multiple partitions.
Let's now go ahead and create an Event Hub. I'm here in the Classic Azure Portal and we wanna get started creating some Event Hubs. Let's create an Event Hub under our existing service bus. So let's click on our service buses, click there, now we click new, and we're gonna click Event Hub, custom create, and we're gonna give our Event Hub a new name. So we'll call this one "test hub." And we'll click next button. And now we have the opportunity to specify the partition count as well as the message retention time. This is the amount of time and days that event messages will be persisted for after arrival, with the maximum being seven. Let's specify four for the partition count and seven, seven days for the event messages.
Okay, this is our new Event Hub ready and waiting to go. Note that we have the familiar connection string item here on the right. And if we go to the configure tab, we can see that we can modify the retention days, but we can not modify the partition count. And below this we see the familiar shared access policy section. This is identical to our previous queue examples, so we won't be repeating the creation exercise here, but we'll need to have our policies and shared access keys prepared before running our client apps. So let's go to consumer groups and have a look at this.
Here, we can specify consumer groups. Conceptually, consumer groups are groups of readers working together so that each group under ideal circumstances receives all the messages in the Event Hub once. Consumer groups are a way to view the data over an entire Hub and this view includes the current state and position or offset for an active reader in the consumer group. We see here that a default consumer group has been created, $default. We can create additional consumer groups if required, and depending on the pricing tier we're on.
You would typically create a consumer group for each distinct application, reading data from the Event Hub. And as part of the partition consumer model, only one reader should be reading a specified partition at one time. For example, we may have a multi-threaded web job with multiple worker threads, each reading one of the partitions in our Event Hub. The web job should have it's own dedicated consumer group. Another application, perhaps one that consumes event messages much more slowly would have it's own consumer group to allow it's readers to read the partitions independently at different positions and speeds compared to other apps consuming data from the Event Hub. Let's head back a level. Lastly, let's look at the Event Hub's scaling option, so if we click on scale. In the service plus scale tab, for our service bus instance that holds the Event Hub, we can increase what are known as throughput units on this slider here all the way up to 20 units in maximum size.
By default, the service bus configuration sets throughput units to one. This provides up to one megabyte per second of ingress or incoming data and about two megabytes per second of egress or outgoing data. Throughput units are pre purchased units of capacity. We buy additional TUs, throughput units, when we need to scale up to handle more data. However, it's more important to remember that a single partition in an Event Hub is always limited to one TU. Therefore, partition is key to scaling and the number of TUs must be less than or equal to the number of partitions. Otherwise we won't be utilizing the TUs that we're paying for.
Here is a simple example of an Event Hub client receiving messages from a specific partition. You'll note that it's very similar to previous examples. The key difference is that we call CreateEventHubClient to get our client object and then we need to make two additional calls to define our consumer group and the partition we want to connect to. Having set this up, we call receive and check whether our client received any data. This is obviously a very simple client, dealing with only one partition. As you can image, there's a lot more to writing an Event Hub consumer. We need to as a minimum read all the partitions and track our current state so that we can resume reading from where we were up to in case we disconnect or restart our client. We won't be covering this in detail here, but we'll shortly discuss a high level approach.
Sending a message is also fairly simple and should be familiar after our previous examples. Having created our Event Hub client, we create an event data instance containing our message and call send on our client to dispatch the message. While sending messages is fairly straightforward, a proper receiver client involves a few other concerns. As a minimum, a client will need to track it's current position when reading a partition. Unlike a queue, received messages are not removed from the partition. Clients can read and reread messages and multiple clients can read the same queue at different speeds and positions.
When a client connections to a partition, it needs to specify an offset. This offset could be zero, if the client wishes to read from the start of the queue or another value if the client needs to start reading from somewhere else in the queue. When a client disconnects or is terminated and needs to start again, it will usually want to resume from a position it was previously at. The Event Hub does not track previous states for disconnected clients and therefore it's the clients' responsibility to persist information about it's current state, or checkpoint as it's commonly known.
Our consumer client also needs to manage reading from all available partitions. All of this requires some development effort, however, there is an existing high level client that already handles these common requirements. The EventProcessorHost is available in the Microsoft Azure Service Bus EventProcessorHost new get package and provides a high level Event Hub reader client that manages partition access and simplifies checkpointing. It's worth exploring this client in detail, and we recommend getting familiar with it as part of further reading.
That's the end of this short session, but stay tuned, because in the next session, we'll start talking about notification hubs.
About the Author
Isaac has been using Microsoft Azure for several years now, working across the various aspects of the service for a variety of customers and systems. He’s a Microsoft MVP and a Microsoft Azure Insider, as well as a proponent of functional programming, in particular F#. As a software developer by trade, he’s a big fan of platform services that allow developers to focus on delivering business value.