Agile Integration Technical Overview
Combining It All Together
The course is part of this learning path
In this course, you will learn about the technical platforms that Red Hat offers for integration and messaging purposes. The course begins with a comprehensive look at the OpenShift Container Platform and then dives into Red Hat AMQ, which allows you to asynchronously connect various application platforms and exchange information reliably between them. Moving onto Red Hat Fuse, you will learn how to connect disparate systems through technologies such as Apache Camel. The course also looks at Red Hat's 3Scale Management Platform, a highly versatile system for controlling APIs. Finally, a demonstration shows you how these three technologies can be used through an example that implements a Camel Route to follow a twitter account and then translates the Twitter feed into a particular language.
- Gain an in-depth knowledge of the OpenShift Container Platform
- Learn about Red Hat's technical platforms and how they can be used
This course is intended for:
- System administrators, architects, developers and application administrators
In this section we're going to have a look at Red Hat Fuse which is a distributed integration platform where you have the option of running it either on-premise as a standalone service, running in the cloud, be it platform as a service or any other containerized option, but also have it as an online service where you basically just rent the resources, the compute resources involved in running your integration solutions. Using Fuse, it's actually possible to independently develop solutions and then integrate them using what we call a lightweight ESB which is what Red Hat Fuse basically allows you to implement by using several different technologies such as, for example, cxf for exposing and consuming of web services, AMQ for communicating in an asynchronous way with your external applications but also internally, should that be necessary.
At the bottom of it all, the driving horse for Red Hat Fuse though is Apache Camel. Apache Camel is an implementation of what we call enterprise integration patterns whereby it's actually fairly easy to consume information from several different sources. Just as an example, expose it to your application through various different technologies such as FTP, NFS, REST Database and so on and Camel then consumes this information from what we call an endpoint by means of a route. The route in Camel is a sequence of instructions basically that explain to Camel where to get the information that it's processing from and then explains what steps need to be taken to convert this incoming data into something useful to the other components in the information system.
So as we see in the diagram over here, it's possible to consume information in different formats from using different technologies from various different sources, process that information into a unified form using several Camel routes, and then temporarily store it, let's say in a relational database, where a back-end system can immediately consume it or, even further, process it using several other routes and then send it to other back-end systems for further processing. For example, again using JMS messaging.
This is obviously just an example of an application using Camel routes for integration. Essentially though, Camel can be used to implement integrations by using building blocks that are called Enterprise Integration Patterns which were first described in a book by Gregor Hohpe and Bobby Woolf called exactly the same, Enterprise Integration Patterns. That book being about 65 different integration patterns, so basic integration building blocks they can be used to establish a technology-independent vocabulary and can be used to assemble those blocks or those integration patterns into an arbitrary solution addressing an integration problem. Among others, for example, Camel implements integration patterns such as splitting messages into smaller fragments, implementing content-based routing, so looking at the message that it's currently processing and deciding which of the available consumers of this message it should be delivered to. Based on certain properties or the content of the message, it can be used to implement splitters, wiretaps, aggregation of messages on the other side of the route, various sorts of transformations, content filtering and so on and so on.
So a similar example to what we were previously looking at is a simple route which can receive JSON files. JSON requests from a Web Store through a REST endpoint, so the Web Store is sending the JSON orders to our Camel application. Those orders may contain several items, obviously, which may be supplied by different vendors. So these orders are then split using the splitter EIP into several submessages, each of which is then processed by the content-based router, looking at the product ID, establishing the vendor that particular order should be forwarded to and then again, for example, each of those vendors could be stored in an asynchronous message queue that can, in turn, feed the information to either an external vendor's application where they can review the open orders that we have for them or can simply be sent forward to an internal order fulfillment application.
Camel works by, as we said already, by means of messages, so it is a request response or request-driven approach and the fundamental building block of Camel route is what we call a processor. Processors pass in a sequence that the route is defined in. Processors pass between each other objects which are called message exchanges and the message exchanges contain various different properties such as, for example, the ID of the exchange which uniquely identifies each request to receive by our Camel application, but also the messages themselves, whereby the original message received by the application is called the in message and each processor that processes the message then places it into the out message as a result of the processing.
If we have a look at what a route normally would look, like let's say we're reading files from an FTP server and delivering them to an asynchronous message queue using JMS protocol. What we would have in the simplest possible route is a consumer component which is, in our case, an FTP client. They can be configured with the authentication data for an FTP server, for example, the endpoint information about where to connect to in order to read files and so on. And that consumer would, as we start the route, connect to the server and for each file found on that particular server it would initiate a new message exchange. Each message exchange, as we said, contains the message itself and some additional properties and is passed by the consumer to what we call a Pipeline Processor. Because an easy way to think of Camel routes is just them being pipelines that define a certain order sequence of events that has to be performed or they have to happen in order for a message to be successfully processed. So the Consumer component creates a message exchange, delivers it to the Pipeline Processor which then forwards it to one or more processors in the pipeline which can do various transformations, can be EIPs.
So integration patterns which, for example, may split the incoming message into several additional exchanges which may do filtering and so on. And each processor does its own job and passes the message exchange back to the Pipeline Processor which, in turn, forwards it to the next one and the next one and so on until it finally reaches the end of the pipeline whereby a producer component delivers that transformed processed message to its final destination, in our case, a JMS message queue. This is a very simple example though because it uses what we call unidirectional processing.
The FTP server simply delivers the file to the consumer component and does not require any sort of response from the pipeline itself when the processing continues. Camel also supports what we call in/out exchanges were normally an HTTP client rather than an FTP server would initiate the exchange by sending a REST request to the consumer. The pipeline would then finish processing as per design, but then obviously the HTTP client will not go away just without getting a response from the webserver that it sent a request to. Which means that some sort of a response has to be provided to the client and in that case we can explain to Camel that our route is actually not unidirectional or must we call it, in Camel terms, in only but it can also be in/out which means that upon the conclusion of processing when the producer component finally places the result into the ending endpoint of the pipeline, a special message can be returned back to the original client, such as an HTTP response, so that the client can reason about the outcome of our processing.
Let's take a look at a simple Camel application that we'll use to showcase the simplicity and power of Camel routes. What we have here is a fairly trivial application but among other things we'll be using it to demonstrate the richness of components that we can use in Camel programming to integrate with various different services such as, for example, in addition to the Enterprise Integration Patterns that Camel implements as components as we talked about, for example, splitters and aggregators, but also content-based routing, message translation, Publish/Subscribe Channels, dead letter channels and so on and so on.
So there's a large library of components that Camel comes up with for the integration patterns. What we also have is an enormous library of ready-to-use protocol components from various messaging ones to actually online services such as, for example, Cassandra DB or even, for example, Salesforce, Facebook, Twitter, GitHub, and all sorts of other components, even to generic RSS ones, but also components that they are actually tailored to a particular service that we may already have and want to integrate with.
So in our example application what we'll be doing is we'll be using the Twitter component to follow a news feed of a certain Twitter user. Let's say we're interested in sports results, so we want to follow an online sports account and be notified about tweets the moment that they're published or with a slight delay maybe. For that, we'll be using a Camel Route that is basically that can be executed in two ways. Initially, it was written as a standalone application, so what we'll just run it as is your usual run-of-the-mill public static void main whereby we simply create a new DefaultCamelContext, instantiate a new route which is a Twitter, a TweetFollower component, a TweetFollower route, set some properties for that route to be able to perform its job. Twitter has an online API but for it to be used we have to apply for a developer account and provide some authentication data to that component and obviously we need to decide on certain accounts that we want to follow. I'm defaulting here to BBCSport, for example.
What we're also doing is we're periodically checking because Twitter API comes with a rate limit. We may only be able to use the Twitter API x number of times in a given period, let's say an hour or three hours. So what we want to do is control the consumption of a rate limit by actually increasing the poll interval between two subsequent checks for new tweets to something like 30 seconds. But we may also want to leave this configurable in case we're running our application in a certain different way. So, essentially, apart from creating a
DefaultCamelContext and instantiating our route, the rest of our application is simply about making sure that we're able to use Java system properties to set some parameters that the component needs to do its job, add the route to the component, to the Camel Context and just simply start the Route, right?
An alternative way that we'll also be looking at a bit later is to run the same Route as a SpringBootApplication. So we have wrapped that same Route in a slightly different way as a SpringBootApplication. SpringBoot can scan for components in a list of configurable packages. Since our Camel route is in the following package, we're just instructing SpringBoot, upon the startup of the application, to actually place any SpringBoot components that it finds in the package into the context and simply run the application. So that's the framework we'll be deploying our Camel route on. As we said, either instantiate the default context, add the route to it, and start it, or let Spring discover it and automatically start it as well.
If you look at the route itself, it's called a TweetFollower route. Again, the majority of the route's source code actually deals with simply setting the parameters the route needs to execute to connect to Twitter API and to execute this job such as variables that determine what account to follow and how frequently to poll for new tweets. Apart from that, as you can see, the route content is extremely simple. What we need to tell it is simply where to read, what kind of reading, what kind of scanning to do with the Twitter API. We'll be using the polling approach, we'll be telling the Twitter component to authenticate against the Twitter API, and just simply selecting an account to follow.
What we'll also be using here is one simple EIP which is called a converter component. So we have a simple converter class that we use to convert the object type returned by the Twitter component which is called twitter.status into a simple much-simplified version of the class called Tweet that we use in the rest of our application. So it's not a feature-rich Twitter status. It's actually a simple Tweet class which simply contains the text of the tweet and the language that it was written in and to convert between twitter4j.status object in the tweet object what we're using is a simple converter component. There we go, we just annotate the class with an @Converter, provide a method that matches the source type and produces the target type, and register that with Camel in a special file called TypeConverter.
We just tell Camel to scan for any converters in a particular source package of our application and it will automatically be able to convert the incoming message type into a target type that we want to use in the rest of our application. What we also have here in the route is we're forwarding the converted type to a simple bean, which we call printer, and the printer is actually just called an StdoutBean which essentially just prints the received message to standard output before we forward it to an internal in-memory destination where it can be consumed by other routes that maybe are waiting for incoming tweets and are waiting to process them further.
So again, a simple three-step route will be consuming messages from a Twitter feed, would be converting them to an internal simplified representation and just printing what we received on standard output. So these are literally three lines of code. Yes, the Twitter components configuration is a little bit lengthy because we have to pass the authentication data but literally three lines of text to connect our application to Twitter and then receive some information from it. If you're interested in playing around with this, Twitter, as we said, has a developer site where you can apply to register for an account, sign in, apply for an application, and receive the credentials you need to use the API. I've already obtained those beforehand so I'll be using those in the demonstration.
So let's look at how to run this application in a standalone mode. I'll just be leaving the initialization properties along the side display so that we can see what java system properties we need to set to configure our application to work properly. So we're going to change the working directory to where our application is. We're going to display the credentials, obviously them being sensitive they're being grayed out. And I'll be starting the application using the exec Maven plugin again by the mainClass of our application is com.redhat.training.agile.main, and we need to specify the authentication properties so agile.camel.consumer.key. Another one is agile.camel.consumer.secret. Another system property we have to specify is agile.camel.access.token and finally, agile.camel.access.secret.
Since this is a standalone application, we'll be setting the poll interval for a little bit shorter than 30 seconds because when running in standalone mode, Camel will actually exit when it's idle for more than 10 seconds, so I'll also be using agile.camel.poll.interval and set it to 3 seconds between polls. So this was Camel starting up and, as we can see over here, we already have a display of a couple of tweets that have been published in the recent time, for example, whereby England surrendered on the final day of the first Ashes Test cricket match. Another one would be where Wayne Rooney is going to be perhaps a player coach and so on and so on so.
As you can see the Twitter component is actually doing its job really nicely. Apparently, it was idle for 10 seconds or the application exited automatically, but as a proof of concept, we have just been able to literally connect the Twitter and obtain some feed information from a source. Alternatively, as we said, we can also use the same application in SpringBoot. For that, I've already, we've already compiled the application but we'll also need to repackage that as a Spring Boot uber jar.
So there we have a jar file which incorporates all the SpringBoot components needed to run this application, so all we need to do is literally, instead of using the Maven exit Java plug-in, just go and run the Java application as an executable jar. We're going to use the default poll interval of 30 seconds. Actually, I have to use environment variables here so instead of setting Java properties, we'll be using environment variables. So Twitter consumer key, Twitter consumer secret, Twitter access token, and Twitter access secret. You see in the last section that the environment variables are actually which I've configured the Spring Boot context with we're going to help us in deploying this application to OpenShift as well.
So let's just run it and see how it works in SpringBoot. There we go, again, we have just found out that British number three Cameron Norrie came from a set down to beat Hungarian Marton Fucsovics in the opening round of the Rogers Cup. So very, very useful information. You can see that, due to our annotations, the SpringBoot context is actually looking, scanning for the components and found our Route, starting it and made it consume from the Twitter timeline of BBC sports.
That is just obviously a very simple example of how to use Camel to integrate applications. We'll be revisiting that in the final section where we combine all the technologies we talked about together. For now, though, that's it about Camel. Join us in the next section.
Jeremy is the DevOps Content Lead at Cloud Academy where he specializes in developing technical training documentation for DevOps.
He has a strong background in software engineering, and has been coding with various languages, frameworks, and systems for the past 20+ years. In recent times, Jeremy has been focused on DevOps, Cloud, Security, and Machine Learning.
Jeremy holds professional certifications for both the AWS and GCP cloud platforms.