This course covers the Design advanced applications part of the 70-534 exam, which is worth 20–25% of the exam. The intent of the course is to help fill in an knowledge gaps that you might have, and help to prepare you for the exam.
Welcome back. In this lesson, I'll talk about background processing.
Background processing is about running tasks that don't require any human intervention. They can be scheduled tasks or some sort of asynchronous process, for example, a scheduled task to parse through some log files and generate data to display on some sort of dashboard.
Background processing tasks are common, and so as Azure has some options to help. The first option is the platforms and service option called Azure Batch. This allows you to run applications in parallel, and it will scale to meet the demand.
It's basically a managed, high-performance computing service. Batch has APIs for .NET, Java, Python, Node JS, among others. And these APIs are how you programmatically manage pools of nodes as well as scheduling jobs.
Okay, let's check out how to set up a sample Azure Batch application. So we're gonna start out in the portal, and as most of these things do, we'll start with the New button. And then we'll click on Virtual Machines, and we're gonna scroll down just a bit to Batch Service.
So we have a form to fill out on this blade, we need to provide an account name, okay, and we also need a resource group, and we'll use one of our existing resource groups, and we're gonna need a storage account, and there we go, perfect.
And now we'll click create, and it's gonna take just a moment to complete, and then we're gonna have our batch account. Okay. Now that that's done, the first thing we're going to need to do is create an application. Under the Features section, we'll click on Application, and we're gonna click on the Add button.
So we have a sample application here, and this is what we're gonna use for this demo. If you're familiar with C#, you're gonna recognize this as a very simple console application. All it does is write out a string to standard out, and that string contains the date and time it was run.
So, let's go back into Azure and set the application ID, and we're gonna call it Hello Batch, and we'll need a version, we can call this one 1.0, and now we need to upload our actual application package, and all that means is that it's expecting a zip file with all of the files required to run our application. Now, we click on the app, and we set the default version here. Okay great, let's save that.
And next up, we need some servers to actually run our batch job. So, we select Pools under Features, and click on the Add button. Okay, we'll need to name our pool, so let's give it a name, and this is where we can change things like the type of OS, the version it is, and things like that.
We'll leave it here by default, but we do need to select a pricing tier for our nodes. Let's go with A1 for this demo. And okay. And now we need to determine how many dedicated machines we'll use. So this is just a demo, so one is gonna work fine.
Okay. Notice that we have some parameters here, these are gonna allow for additional things such as scaling. We'll leave these by default. And it's gonna take some time to spool up this server, so while that's happening, let's go create a job. And so we'll click Add, and let's name it Hello Batch Now. We're naming it that because when you create a job, it's executed immediately.
If you want to schedule one, you can do that, but you're gonna have to use the job scheduler. So it's an option if you need to schedule it, we're just gonna run it right after we've created it. Okay, we need to select the pool that this job will execute in, and we only have the one, so this is going to be an easy choice, and we click okay.
Alright, our job is created, however it needs to actually know what to do. So we need to give it a task, and for this we'll give that a name, and as you know, naming things is one of the hardest things in computer science. Give it a name there. And let's paste in a command for this task. We're gonna have it run our application, and we'll need to use this environment variable for our directory that the application lives in.
Okay, so now let's scroll down to the application package, and this is how our task knows which application and version to use. So let's give it just a moment to complete. We'll click on Refresh, and there it is, it's complete.
Let's check out the output of this. Remember, we're expecting a string, and at the end of that string, we'll see a date-time appended. So we're gonna click on Files on Node, and select the standard out .txt file, and we're gonna select that, and there's the output that we were expecting. So this is a very basic overview of how to use Azure Batch.
Okay, the next service that will help with background tasks is the Azure Scheduler. Azure Scheduler is a service that as the name suggests, allows you to schedule tasks. The ability to run scheduled tasks in a cloud-native way saves the headache of setting up virtual machines and using local schedulers.
With this, you can do things such as run a task of your creation that maybe pulls comments from your product's social media feeds and then puts them in a queue for processing, and these sorts of long-running, periodic tasks are commonplace, and they're made all the easier if you don't need to handle the management of that scheduling infrastructure.
Okay, let's move on to WebJobs. A WebJob enables us to perform background processing tasks in an appropriate context. It allows us to run .NET console apps, scripts, such as PowerShell or Windows batch scripts, PHP, Python, and Node, among others. If you need to deploy more files than a single executable or script, you can also zip up all of the content and upload that.
And we can deploy it directly from the portal or via FTP as well as through Visual Studio. Since you may want to run a job continuously, you can enable the always on setting, which will avoid disposing of the WebJob while it's running.
And while you can use the option to run it continuously, which will kick off the job, say, every 60 seconds or so, if you want the app to truly always run, you're going to want to manage that with a loop inside of code and then have the thread sleep as needed.
WebJobs scale the same way Web API does and run on all the instances deployed from Web API by default, and should you need more processing power, again you can scale those instances up. You should consider WebJobs when you have a task that needs to run on a schedule, or a task that will run continuously and will monitor the response to some trigger, maybe something like a filesystem watcher.
Also, any long-running tasks that aren't suitable for short-lived requests, maybe something like response to a life cycle event in a Web App handler. Also, if you currently run a task via scheduled task on Windows or a cron job on Linux, then WebJobs are probably the analogous option.
A classic example is a WebJob that retrieves and prepares data for a Web App. For example, let's say you have a long-running SQL query that creates a large data set, and instead of having the query being kicked off by your Web API, you could have this WebJob run ahead of time, grab that information and cache it, and then your Web API could pick that up.
Okay, let's create a basic WebJob and then deploy it. We're here in Visual Studio and we have this console app, and what it does is outputs a string with the current date and time appended to the end. And that's it, it's nothing fancy. So if we run this locally, and we set this breakpoint, you can see that we have a string.
Alright, let's deploy this as a WebJob, and we can do that by right-clicking on the project and selecting Publish as WebJob. Okay, perfect. And now we have options here for how it's going to run, and that's going to be continuously on a schedule or on demand. We'll use a continuous option, and we're going to need to import our profile.
If we jump into the Azure Portal, we can refresh and in a moment it's going to show up. Alright, we have it here, and if we click on the logs we can see what it's been up to. This is gonna take a moment to load, but when it does, we're going to see a log with our output. Now what I'm gonna do is jump ahead to a few minutes from now, after it's run a couple of times so you can see what the log looks like.
Okay, welcome back. Here we have our log, and we have our strings with a date-time appended to it, and it kicks off every 60 seconds. When it comes to background processing, sometimes you need to have the background process executed in response to some sort of event, and this is what Azure Functions can help you solve.
Azure Functions fall under the generic industry term of serverless computing, which basically means that you don't need to manage the servers and instead you can focus on the business logic of the functions. Azure functions allow you to execute code developed in C#, F#, Node JS, Python, PHP, Batch, Bash, and just about any executable.
And the code can be triggered by different types of events. Here's a partial list of some of the different triggers that will help with background processing tasks. The first is the event hub trigger, which responds to events delivered to an Azure event hub.
Next is the generic webhook trigger, which processes webhook HTTP requests from any service that supports webhooks. There's the HTTP trigger, which allows for the execution of code based on an HTTP request, which is pretty cool.
There's the queue trigger, which will respond to messages as they arrive in an Azure storage queue. The service bus queue trigger will execute code by listening to message queues. The service bus topic trigger will execute code by subscribing to topics, and then the timer trigger will execute code on some set schedule.
Functions as a service is a growing trend in the industry due to the lower overall cost, managed platform, and flexibility. As you're architecting solutions, you're going to want to consider using Azure Functions for tasks such as image processing, long-running tasks that you want to run in the background, or for scheduled tasks.
Okay, let's wrap up here. When it comes to background processing, Azure provides options such as Azure Batch, WebJobs, and Azure Functions. In the next lesson, I'm going to cover connectivity for hybrid applications. So if you're ready to keep learning, then let's get started in the next lesson.
About the Author
Ben Lambert is the Director of Engineering and was previously the lead author for DevOps and Microsoft Azure training content at Cloud Academy. His courses and learning paths covered Cloud Ecosystem technologies such as DC/OS, configuration management tools, and containers. As a software engineer, Ben’s experience includes building highly available web and mobile apps.
When he’s not building the first platform to run and measure enterprise transformation initiatives at Cloud Academy, he’s hiking, camping, or creating video games.