In module four, we begin to take the first steps towards implementing the architecture developed in the previous module. What we hope to show is the first steps taken towards actually building it, and how do you move towards production, whether this is load testing, server selection, and how to even begin to monitor and evaluate your solution as it's being deployed. So, to dive in and actually explore it, the best perspective to do this from is from that of the event processing engine. If you remember from the architecture, NiFi is the glue that holds the whole thing together. So, by examining our journey of where it starts with data landing in NiFi as the rest server, how it gets posted to the service bus, and then how it gets processed and maybe even some additional interfaces at the end state, we're able to take a holistic end to end solution and examine how the solution actually takes form and what it will look like for a developer to begin to build the solution. This is the NiFi interface. 

As the event bus, it is one of the best ways to see how data actually flows through our solution starting with where it lands to how it gets processed, how it gets placed in the service bus, and finally how it gets pushed out to the user interface. For those of you unfamiliar with NiFi, it has a drag and drop interface ability to add processors and capabilities at the top which are then graphically represented here. You can begin to see data coming in and out of the solution. This demo isn't running as of right now. You can see there's zero bytes moving over the last five minutes. As we start to run the solution however, you'll see these numbers changing to real time statistics which actually helps with some of our monitoring requirements we're also facing. To double-click on to the received data section we can see how the data starts to get landed as per our architectural requirements. If you recall, we have the requirement that we defined of landing the data via http rest or restful interface. 

This is because the automobiles already have this style of interface on board and adding this is minimal rework. We have flow here where we land the request and send it back, but before we get too far into that let's learn how to read NiFi a little more. You have handle http request and you can see a descriptive name for it which in this case is just handle the request and then the processor name below it. In some cases we can give it a descriptive processor name and you could even see some of its uptime statistics over the last five minutes and its checked 140,000 times to listen for new incoming data. This is extremely scalable so it meets that requirement of this can be massively parallelized and this flow right here is also stateless in that events handled on one node can also be handled on another node and ordering doesn't matter because that's handled down the pipe. 

Basically, a request will come in from here and it'll flow through the pipeline so you could see a successful request received goes to an accumulator, or a funnel rather, and then it goes to publish and so on. To start the flow, you can see that we start with handling the http request as per the first step in our architecture. This then flows to the funnel in which requests from future pipelines that we've put placeholders in for for listening from TCP and UDP connections can come in. Below this wasn't a requirement. We just decided to include it to show some future capabilities to the stakeholders. It then goes to publish Kafka. Now, publishing to Kafka is important because this is our service bus. You can see that we very quickly handled landing data and going to the service bus. 

This is actually where the restful interface is part of the journey ends where we've actually landed the data and published it. However, we can go a step further and have some guaranteed delivery capabilities and that's handling the ack back. For those of you familiar with restful APIs, you can use this to send a success code like 200 back or an error code like 403 back if there's some type of authentication error. 

Now, an interesting step of putting the response after publishing to Kafka is that if something happens wrong in this pipeline and we don't publish to Kafka, the response never gets sent or it captures the error message. An alternative way to do this architecture would've been to put the response in an earlier stage such as over here where we can just immediately send the response and now you can see that will just land the data and say hey we got it. This doesn't necessarily capture whether or not Kafka has successfully recorded the message which is why it's important to put it after the publish. Another important part of this architecture is that by publishing it to Kafka after landing the data, it's all going to a common storage location for the service bus. If we have hundreds, or maybe in production, thousands of nodes of NiFi handling these incoming requests, when they go to Kafka they're all going to be riding into the same location and Kafka is more than capable of being able to handle this scale, being able to handle the production loads of even the biggest autos or manufacturing facilities. Let's look at what happens after Kafka and how we begin to start handling the event processing.

 Now, if you remember the event processing is a flow in which we begin to take the messages out of Kafka, parse them, enrich them, and put them into each base for future storage and retrieval. There's a few things here that are interesting. We can expand this at later dates to include even more enrichment. These pipelines are very flexible and can even support multiple users. What we're showing here is very basic consume from Kafka, evaluate the JSON message, and storing it in HBase. In the future we may wish to show some enrichment with weather data or maybe enrich it off another internal database. To demonstrate what's on screen here, we start with the consume from Kafka. This, too, is parallelizable at mass scale because consumers belong to groups and technology's like a patchy Kafka track what the current message for that consumer group is so you can have guaranteed once delivery. From here, the message is passed to a JSON parser at which point we begin to extract the message ID, perhaps the message type it is. Remember, the key to this is that we're allowing the vehicle engineers to send any type of data down the pipe and have it recorded. This empowers us to not fully have to understand what the messages contain as long as we can parse it and send it to the right way. That means after we extract the key we send to route on attribute. 

This is really where NiFi fills its event processing and routing capability. By parsing what's in the message and sending it to where it needs to go, we've created a flexible yet expandable interface to handle lots of future scenarios. From the route on attribute, we can send an alert which goes over any existing ADP notifications system. Note that this wasn't actually part of the strict requirements. It's just once again part of the solution that we can add in and show how it's expandable much like the UDP and TCP reception. The more important part, though, is the alert or GPS goes into the HBase. This is where we store NoSQL, massive big data warehouse where it can further be retrieved by end users. We also, though, in this solution to help with debugging, development, and maybe even production have the events going into queues. 

These queues will time expire over time and in production probably shouldn't exist. They're great for showing how the data is flowing in development, but maybe in production we want to clean this up and not have loose ends. Once the data is in HBase, we've actually fulfilled our requirement of providing an end to end solution. We've landed the data. We then sent it to the service bus. We then did some event routing and stored it in a database and then the database has some built in user functionality to access it. To expand on this just a little bit more and show how we can further improve the user experience using NiFi as it's already a product that is going to production and not having to introduce additional technical products will make this solution easier to support long-term. This on screen right here is showing us adding a very friendly user restful API to retrieve data from HBase. HBase imports APIs. HBase supports lots of ways to access the data, but if we want to customize it we can make it so the user can send and http get request, a restful get. We can then parse what they're asking, perhaps reparsing some headers. Then, we're able to fetch the HBase row. 

I could also see a system in which we don't require the person to send JSONs. We could potentially have them use headers. The point of this demonstration here is to show we can find creative ways to use existing technology to keep overall architecture costs low. Not introducing more technology, not introducing one-offs. We're using common components, really adds flexibility. Then, of course, this can go into handling different responses such as we found a response. The row is good and we're going to send a 200 or maybe it's not found or it's not allowed. We can send a 403, 404, maybe 500 error code. Beyond just showing how the architecture is done, let's talk about how we can begin to size this and take advantage of different technologies to make sure we understand how big our footprint is going to be. 

There are some rule of thumb approaches that you can build such as this much CPU, this much memory, this much IOPS, maybe an SSD versus a hard drive for the server decisions. That's something that individual vendors both software and hardware and cloud can help you understand. Regardless of what the estimates say, we should really start to examine how to look at the solution as it's running and use that as the basis of our future sizing. What are the indicators that tell you that you might have too much or too little processing? Many of the cloud providers have CPU utilization metrics. Many server monitoring solutions and virtualization solutions also can tell you how much memory, CPU, and disc capacity is being used. However, this can be affected by certain programs that will reserve large blocks of memory and not use them. It might not be an accurate representation. A great way for the application level to tell how heavily it's being loaded is to look at the queues.

 If you remember, we included this service bus and one of the reasons was to detach incoming messages from outgoing messages. What we're able to look at is how much data is being queued between landing a request and riding out the service bus or more likely, how much data is in the service bus and not being consumed back out by the back end processing of the program. This will allow us to do things such as monitor the length of queue in Kafka that hasn't been consumed by the application. Also, NiFi shows some graphical ways of showing it such as how much data is stored up in the queues and also if the car manufacturers are not getting meaningful ack backs in time, we can also tell the system is beginning to fail. As you implement this, take steps to understand how the data is actually flowing the application beyond just what the VM or server level metrics are indicating.

 In summary, we've shown a prototype of the solution that was built out to capture automobile events, parse them to a service bus, store them in a NoSQL solution, and serve them to end users allowing people to send messages from the auto in their native format and read them in a reliable format. We've also understood how to use applications to add capabilities such as using existing applications to add new interfaces and how to leverage applications performance data to aid in server sizing so that we don't have to just use a naive approach such as gross CPU utilization metrics at the monitor level.