Understanding Streaming Data Processing for IoT Hub
Understanding Streaming Data Processing for IoT Hub

Once your data is in IoT Hub, you need to be able to access it; ideally with different processors. Using different processors allows you to break out the responsibilities of each.

Azure provides a few options for processing events and sending feedback to devices. In this course we'll introduce you to Stream Analytics, Azure Functions and raw processing with C#.

If you want to follow along, or just try this out for yourself, you can find the code here.

Processing Azure IoT Hub Events and Data: What You'll Learn

Lecture What you'll learn
Course Intro What to expect from this course
Understanding Streaming Data Processing for IoT Hub  
Writing an Event Processor for IoT Hub with C#  
Handling Device-To-Cloud data with Azure Functions  
Processing IoT Hub data streams with Azure Stream Analytics  
Sending feedback to devices with Azure Functions Reviewing the code for the device
Final Thoughts Wrapping up the course



Welcome back!

In this lesson, we'll talk about the streaming data processing architecture for IoT Hub.

In the context of IoT Hub, when we talk about device-to-cloud communication, we’re talking messages that devices send to IoT Hub. IoT Hub collects these messages to be processed in the cloud. Processing may mean storing the messages, performing analytics, sending commands to the device, or something along these lines.

When we talk about sending messages generically, it seems like we’re talking about queues where we ask the queue for the next message. A queue is a sort of server-side cursor that know which message is up next. With a queue you could implement a competing consumer pattern, where you can multiple processes consuming messages. In this pattern a message is handled by a single process.

Well, IoT Hub is not a queue, it’s a data streaming infrastructure. With a data streaming infrastructure you look at the data starting from point of time, forward.

The data streaming infrastructure of IoT Hub is built upon an important nonfunctional requirement, of being a "low latency, distributed, partitioned stream ingress".

The idea is to be a high-scale infrastructure capable handling large amounts of messages from devices, making it ideal for real-time analytics, logging, and telemetry.

The data streaming infrastructure is basically a time retention buffer. Messages will exist in IoT Hub until a "Time To Live" is reached. You don't need to remove a message, after time-to-live expires, the message will be automatically removed.

Since it’s a streaming infrastructure, and there’s isn’t really a concept of completing a message, you get the benefit of processing the messages multiple times; which is done use consumer groups.
If you have multiple processes interested in the messages, you could create a consumer group for each, and each has it’s own view of the messages. Maybe you would have one consumer group that saves all the data to a persistent data store. And another consumer group used to perform analytics.

Here’s a diagram to help explain how IoT Hub works.

Messages in IoT Hub are distributed over partitions, allowing for a scale-out architecture to handle a massive amount of events. Events that are generated by devices that are sent with different protocols.

The device itself doesn’t know or care about partitions. Devices only send messages, and then IoT Hub distributes them among the partitions.

Then you have the event processors. You can have multiple event processors read messages from all the partitions and keep track of what they have read with consumer groups.

Okay, now that you have an understanding of the IoT Hub messaging infrastructure, we can start talking about what an event processor is and how to write one. And that’s what we’ll do in the next lesson.

About the Author

Marco Parenzan is a Research Lead for Microsoft Azure in Cloud Academy. He has been awarded three times as a Microsoft MVP on Microsoft Azure. He is a speaker in major community events in Italy about Azure and .NET development and he is a community lead for 1nn0va, an official Microsoft community in Pordenone, Italy. He has written a book on Azure in 2016. He loves IoT and retrogaming.