Getting the Most From Azure Storage
The course is part of these learning pathsSee 3 more
The Azure Storage suite of services form the core foundation of much of the rest of the Azure services ecosystem. Blobs are low-level data primitives that can store any data type and size. Tables provide inexpensive, scalable NoSQL storage of key/value data pairs. Azure queues provide a messaging substrate to asynchronously and reliably connect distinct elements of a distributed system. Azure files provide an SMB-compatible file system for enabling lift-and-shift scenarios of legacy applications that use file shares. Azure disks provide consistent, high-performance storage for virtual machines running in the cloud.
In this Introduction to Azure Storage course you'll learn about the features of these core services, and see demonstrations of their use. Specifically, you will:
- Define the major components of Azure Storage
- Understand the different types of blobs and their intended use
- Learn basic programming APIs for table storage
- Discover how queues are used to pipeline cloud compute node together
- Learn to integrate Azure files with multiple applications
- Understand the tradeoffs between standard/premium storage and unmanaged/managed disks
Okay, so let's talk a little bit about how Azure queues can be used to facilitate a decoupled architecture for an application running in the cloud. So, what I'd like to show you is a small application or a small service that I've already built running in my Azure subscription. So, this is my, yeah, here it is.
This application was built using a feature in Azure called logic apps. Logic apps they're based on the web application or the web app infrastructure in Azure, so if you're familiar with that, the idea is that, as you can see on the screen logic apps provide sort of a declarative, little bit of a dragon drop kind of visual canvas for stitching together logic, and it's sort of an alternative to writing your own code by hand to perform kind of basic workflows.
You know, the idea is that this application would be triggered by some event that occurs in the cloud, would do some processing on the data that's associated with that event, and then, you know, potentially write data some place else or again, ultimately surface that data that processed data in some other way.
So, in this case we're going to be triggering our logic app off of messages arriving in an Azure queue since that's of course our focus here is just to demonstrate the use of Azure queues. So, basically what's happening is this application is used to do what's called sentiment analysis. Azure has a number of machine learning-based features that are surfaced via APIs, so you can of course write your own machine learning algorithms and applications if you want to but there are some that are already kind of pre-built and pre-packaged which is kind of handy, and so one of them is this sentiment analysis feature, and basically what that does is it allows you to pass in text, just random text, and then get back a numeric score, kind of a quantitative score that tells you whether the text expresses a positive or a negative sentiment, you know, somebody really likes something or they really dislike something.
And so, you can use that numeric value to sort of drive additional behavior and additional conditions and logic, and that's just kind of a useful thing. It's often used in marketing and market research for things like Twitter posts or perhaps blog comments, product reviews, that kind of thing and those are kind of common scenarios.
So, if I just walk through this real quick and just show you what's happening, you can kind of get a sense of it even without drilling into the details. Basically we're waiting for messages to arrive on a particular Azure queue. Those messages will contain the payload of those messages will just contain some text that we want to run this sentiment analysis algorithm on.
So we do that. We grab the text, and pass it into, again, I've already pre-configured this but we're passing it into our sentiment analysis API, that text, and then getting back a numeric response. Once we get that response and that's represented by this score value here, I'm passing it into an Azure function because ultimately this score, it's a numeric value, but what I really wanna do is I wanna turn the numeric value into something that says like excellent or good or fair or poor or something like that, I wanna give it some sort of like English description.
So, I just wrote a little small Azure function. We won't even step into it. It's not super interesting but it just basically looks at that score. The score goes from zero, meaning, you know, extremely poor sentiment, to one which means it's a perfectly positive sentiment, and so we're basically just kind of breaking that up, chunking that up and assigning it to buckets, and then I'm returning an English description of the sentiment.
So, once I get that back from the functions that comes back as JSON. I just need to do a little parsing so that I can grab the exact text that I want out of it but then I'm going to post that to a Slack channel, and so I have a private Slack account and a channel in there that I'm just using talking to that channel from this logic app and just posting that analyzed text and the ultimate sentiment that's derived from that API, and then ultimately just delete the message when I'm done from the queue.
So, I don't wanna spend too much more time focusing on the logic app itself. I mean, that's certainly very interesting, logic apps are an interesting capability in the sentiment analysis and some of the cognitive services in Azure are very interesting but the big picture here, or the thing that I want to drive home is the use of an Azure queue and a message appearing in the Azure queue to drive this application.
The goal here is that we wanna have this service, this sentiment analysis service, kind of stand on its own and we may have one or more front end applications that want to pass text to this service and receive some sort of result or you know, do this processing and then we see it show up in Slack saying hey, this text, you know, that we may have a problem here, we need to follow up with this customer.
You know, that sort of processing. And so, we don't want the service to be aware of all of the potential applications that may want to use the service and pass text into it. Instead, what we really wanna do is we wanna have this nice decoupling or separation where we can add as many applications as we want to far into the future potentially if we want to and the service doesn't have to be aware of those because the only thing the service really knows about is the queue itself which is triggering it.
So, that's really where you get the decoupling. That use of queues in a cloud architecture to provide that decoupling, that's a very common pattern in Azure queues. There are some other options certainly in Azure but Azure queues are one of the best options within Azure for providing that kind of sense of decoupling.
Okay, so we've got our application. Let's close out of that, and let's go ahead and wire up. I'm gonna be using an application called Postman which is basically a rest API testing tool. What I'm gonna do is I'm going to post messages to the queue using the rest API, so just sending HTTP posts with the properly formatted message, will just get text into the queue, and that's just a very, you know, that's obviously a very simple manual demonstration of passing messages into the queue but ultimately, again we could have any number of applications maybe posting thousands of messages a second to the queue, and again, the nice thing is because we're not sending messages directly to the service but instead sending them to the queue, which is then triggering the service, we get this nice decoupling, and this nice buffering where the service can scale out or contract as needed to sort of handle incoming request frequency.
So, if the service figures out or notices that there are a lot of messages buffering in the queue, maybe hundreds or thousands, the service can use that as a trigger to start to scale itself out and then ultimately do faster processing of the messages in the queue. Similarly, it can also contract itself when the text that it's processed most of the messages, and the message count in the queue is starting to decrease in size.
And ultimately that saves you money when you're running in the cloud of course. Okay, so let's go to our storage account, and I'll show you real quick. I've already go a queue in there. Click on queues. And you can see here this is my queue here. It's just called sentiment requests. There it is. So, that's the queue that we're gonna be targeting.
Again, that's already kind of wired up and the logic app is listening to it already. So, what we need is we need, recall, we talked about shared access signatures as kind of a discreet, fine-grained way to provide access to storage services and storage resources, so what I wanna do is I wanna generate a shared access signature token that I can use to post messages to this queue.
So, I'm going to specify that I only am interested in processing against a queue. We'll pick all the resource types for now, and I don't need to read anything, and the only thing I really need to do is I need the ability to add messages to the queue, so I'm just going to keep the add permission. I have a timeframe kind of already predefined for me.
I'm actually going to use, I'm UTC minus four, and so let's see. That window should work correctly. And I think the rest of this should be okay, so let's generate our SAS token. Okay, here we are, so we have this nice long SAS token, and I will click copy, and then I'm going to switch to Postman. So, this is the Postman tool.
And again, this just allows you to issue HTTP requests to any URL that you choose, so I'm going to switch to a post, and I'm gonna paste in my URL. Now, the URL that I've got for my SAS token, this is an account level SAS token. It's not yet pointed at a specific queue that I need, so what I need to do is I need to amend this just slightly and I need to add in the name of my queue, sentiment requests, and then also say messages because that's specifically what I'm doing, is I'm adding again, via the rest API, I'm adding a new message to this particular queue, so that's kind of just the format that we need.
Alright. And the last thing I need to do is I need to define the body of the message. So, Azure queues require a specific kind of XML syntax. Oops. Queue message. Of course, if you used things like the . NET SDK or the Node SDK you don't have to type all this sort of stuff yourself and kind of manually construct these messages, that'll all be done for you by the SDK code, and it gives you nice high-level APIs for sending messages but I'm just, you know, I'm kinda just doing this by hand.
The nice thing is of course if you're running on some other platform where you don't have an SDK available or maybe even if you just prefer, you know, using raw rest calls instead of using an SDK then quite literally this is all you have to do to kind of put messages together, so it's actually fairly straightforward.
Okay. So, let's pick like a positive message like, you know, we'll say something like wow, I really love this new thing. Again, we're just kind of trying to get a sense of what the sentiment is of this thing. So, if I click send, if I've configured everything correctly, then yes, we actually see a 201 down here which indicates success, my message was successfully posted to the queue.
And there's some details down here as well that are sent back from the queue they're just kind of more or less it's just metadata that confirms that yes, the message was added. So, if I go back to my logic app. The logic app will give me some sense of how it's running and whether it's been successful or not.
Basically, I've configured it to look at the queue every 20 seconds or so and so it's not, you know, sometimes it's a little bit more than 20 seconds but it's approximately that amount, so we just need to make sure that we've hit that window. Let's see. I don't know if it's quite picked it up just yet.
Alright, looks like it's running now. So, we'll give it a second. It usually only takes a second, or a couple of seconds to process. If I switch over to Slack we should see yeah, okay, here we go, here it is. So, let me zoom in here so you can see it. I've got a couple previous ones in here which you can see incoming message, wow, I really love this new thing.
Sentiment is excellent, so yes, we've converted the numeric sentiment score into again, some sort of English phrase that, you know, gives us something that we can post in Slack and maybe be a little bit more meaningful so that's one example. So, let's pick another one. We'll just say boy, this thing is the most awful thing ever.
Alright. Post it again. Looks like it was successful. Again, and it'll take a couple seconds for the logic app to pick it up. Alright, there we go. Looks like we got one now, another one processed. Boy, this thing is the most awful thing ever. Sentiment is indeed poor, which of course is the case. Okay, so hopefully that gives you a bit of a sense of how to build a distributed architecture like this in the cloud and use Azure queues to provide a good decoupling between pieces of your application.
Again, very common pattern, something that you should become more and more familiar with as you build applications to run in the cloud. It's a pattern that you'll see and use over and over again.
About the Author
Josh Lane is a Microsoft Azure MVP and Azure Trainer and Researcher at Cloud Academy. He’s spent almost twenty years architecting and building enterprise software for companies around the world, in industries as diverse as financial services, insurance, energy, education, and telecom. He loves the challenges that come with designing, building, and running software at scale. Away from the keyboard you'll find him crashing his mountain bike, drumming quasi-rythmically, spending time outdoors with his wife and daughters, or drinking good beer with good friends.