Welcome back!

In this lesson, we'll talk about the streaming data processing architecture for IoT Hub.

In the context of IoT Hub, when we talk about device-to-cloud communication, we’re talking messages that devices send to IoT Hub. IoT Hub collects these messages to be processed in the cloud. Processing may mean storing the messages, performing analytics, sending commands to the device, or something along these lines.

When we talk about sending messages generically, it seems like we’re talking about queues where we ask the queue for the next message. A queue is a sort of server-side cursor that know which message is up next. With a queue you could implement a competing consumer pattern, where you can multiple processes consuming messages. In this pattern a message is handled by a single process.

Well, IoT Hub is not a queue, it’s a data streaming infrastructure. With a data streaming infrastructure you look at the data starting from point of time, forward.

The data streaming infrastructure of IoT Hub is built upon an important nonfunctional requirement, of being a "low latency, distributed, partitioned stream ingress".

The idea is to be a high-scale infrastructure capable handling large amounts of messages from devices, making it ideal for real-time analytics, logging, and telemetry.

The data streaming infrastructure is basically a time retention buffer. Messages will exist in IoT Hub until a "Time To Live" is reached. You don't need to remove a message, after time-to-live expires, the message will be automatically removed.

Since it’s a streaming infrastructure, and there’s isn’t really a concept of completing a message, you get the benefit of processing the messages multiple times; which is done use consumer groups.
If you have multiple processes interested in the messages, you could create a consumer group for each, and each has it’s own view of the messages. Maybe you would have one consumer group that saves all the data to a persistent data store. And another consumer group used to perform analytics.

Here’s a diagram to help explain how IoT Hub works.

Messages in IoT Hub are distributed over partitions, allowing for a scale-out architecture to handle a massive amount of events. Events that are generated by devices that are sent with different protocols.

The device itself doesn’t know or care about partitions. Devices only send messages, and then IoT Hub distributes them among the partitions.

Then you have the event processors. You can have multiple event processors read messages from all the partitions and keep track of what they have read with consumer groups.

Okay, now that you have an understanding of the IoT Hub messaging infrastructure, we can start talking about what an event processor is and how to write one. And that’s what we’ll do in the next lesson.