1. Home
  2. Training Library
  3. Google Cloud Platform
  4. Courses
  5. Developing Solutions for Google Cloud Platform with App Engine

Task Queues


Course Intro
Getting Started
PREVIEW12m 23s
App Engine
15m 53s
11m 19s
9m 28s
Wrap Up
8m 35s

The course is part of this learning path

Google Cloud Platform for Developers
Start course
2h 1m

Building Solutions for Google Cloud Platform with App Engine

As developers, the learning never ends. Just when we get used to a certain technology, it goes and changes. We’re always needing to learn new languages, frameworks, APIs, and platforms.

And if you’re also in charge of deploying your code, then you need to understand how to setup and configure web servers, deal with scaling issues and manage databases. Frankly, it can be exhausting!

So, why invest the time in learning yet another set of technologies? That’s the question I find myself asking whenever some new trend comes along. I want to know how it’ll make my job, and the jobs of my peers better or easier.

Throughout my career I’ve been responsible for deploying my own code. And I think that’s why the Google Cloud Platform resonates with me so well. It’s a platform that understands developers.

So, why take the time to learn something like App Engine? I think the answer is simple. Because you can take all your development experience and apply it to a platform as a service that removes most of the obstacles to getting code running in production.

The value of having Google ensure your app is highly available is worth a lot to me. We get a rich set of tools for developing locally, and simple deployments, with application versioning. All while using the same programming languages and frameworks that we’re used to.

If you’re looking for a native cloud platform for building out highly scalable applications, or mobile back-ends, then you’ve come to the right place. App Engine provides all that and more.

This course focuses on teaching you about the tools App Engine provides to build out highly scalable systems.

We’ll be building out Python web applications using Flask, and using Cloud Datastore as our database. There is a bit of a learning curve to getting started. And that’s what this course is for, to minimize the amount of time spent learning the platform, so you can get back to writing code.

The source code lives on: Github so feel free to download it, and follow along.

We’ll cover a lot of data in this 2 hour course. And by the end of it, you should feel comfortable getting started building out App Engine applications of your own.

And if you’re looking to get your Qualified Solutions Developer certification, this is going help you with that too.

So, if this sounds useful to you, let’s get started!

Course Objectives

In this course

  • We’ll create an App Engine application, and deploy it
  • We’ll be Developing a REST API with Cloud Endpoints
  • We’ll learn about the different Authentication and Authorization methods available on App Engine.
  • We’ll learn about the monitoring and management tools available.
  • We’ll cover the different storage options available.
  • We’ll review Cloud Datastore more in depth.
  • We’ll look into ways to improve application performance.
  • And We’ll learn about Task Queues.

Intended Audience

This is a intermediate level course because it assumes:

  • You have development experience

What You'll Learn

Lecture What you'll learn
Intro What will be covered in this course
Getting Started Creating our first App Engine application
Cloud Endpoints How to create RESTful APIs with Endpoints
Services How to break our applications down into separate services
Authentication How can we authenticate users?
Managing Applications How do we manage App Engine apps?
Storage How do we use the different storage options?
Datastore A more in depth look at datastore
App Performance How can we make our apps more responsive?
Task Queues How can we run tasks outside of a user request?
What's next? Where do we go from here?



Welcome back to Developing Solutions for Google Cloud Platform. I'm Ben Lambert and I'll be your instructor for this lesson. In this lesson, we'll talk about task queues. We'll cover push and pull queues and scheduled tasks.

Task queues provide us with a mechanism to perform work outside of the scope of a user request. Why is this useful? Sometimes, we have tasks that are long-running or things that need to be done on some sort of schedule. So things like batch data store imports and exports or sending out emails. With App Engine, we have two types of queues, we have push and pull queues.

By default, every app gets a push queue. Push queues allow us to create tasks, which is what we call a unit of work. So, we can create a task, add it to the queue, and then by default, App Engine will run the task for us and if it fails, it's gonna keep trying and it's gonna progressively back off on each failure so that it doesn't bog down the system. If we don't want a task to keep running forever and ever until it's successful, we can also configure the queue allowing us to set a maximum number of retries. When our tasks are complete, or it hits the maximum number of retries, App Engine will delete that task for us.

Let's see how we're using tasks in our demo application. If we look at the upload.py file, we can see that after we create our image, we're creating a task that's gonna post some data to the /tasks/apply-tags route. It's sending the name of files stored in our bucket and then it's sending over the url-safe key and if we look in the tasks.py file, we can see this just a flask application, there's nothing special about it, and that we have a url listening and it decodes the date in the body to a JSON object and then it pulls the bucket name and key out of that and it uses that info to upload the image to the Vision API and then it's gonna get back the labels and it will save those as tags. This allows us to offload the external call to Vision API as a task and that way it happens asynchronously. So, should it run slowly, it's not gonna hinder the user and make for a poor user experience.

Let's watch our task flow through the app. First, we're gonna pause the task cue because usually this runs so fast that you just won't be able to see it happen. And then, let's upload an image. And we'll just grab one here. Okay, now that it's complete, let's look in the queue. And there it is, we have a new item. So, we can inspect it, retry it, and delete it.

Let's look in data store for our image. And if we filter Okay, there it is. So, it has no tags. Now, let's start the queue. And the task is gone. Remember, push queues will remove the item when they've successfully completed the task or if it's hit the retry-limit that we've set. And if we look back at the data store object and refresh this… okay, we have our tags.

When working with queues, we can specify settings in the queue.xml if we're using Java or queue.yaml for everything else. The yaml version looks like this. We can set a max storage size, the queue name, the bucket size, and a refill rate. Queues work by assigning each task a token. And then the tokens are stored in a bucket And our app uses those tokens to know which tasks need to be done. And when a task is complete, the token is removed from the bucket.

So, we can set the size of the bucket and how fast it refills allowing us to control the flow of tasks. And we can also set caps on the age of a task and the number of retries. To secure our tasks, we want to make sure that we set the login property to admin in the app config. You can see here that ours is set to admin. The task queue will be considered an admin. So, we don't to worry about that. And if we wanna insure that our tasks are only triggered by App Engine, we can also check the header of each request and make sure that we look at the queue name and the task name. Okay, these headers can only be applied by App Engine so users aren't gonna be able to fake these.

Now, we saw push queues, now, what about pull queues? Pull queues are intended to be used when we need to work with tasks outside of App Engine. Push queues will work only inside of App Engine, but pull queues will work anywhere, in or our of App Engine. So, if we wanna have code running elsewhere, maybe on a compute engine instance, and we want it to interact with our App Engine apps we could use pull queues. They work a bit differently than the push.

First, we need to lease a certain number of tasks and then for a specific duration and then we can process the tasks and explicitly delete it when we're done. We have to delete it ourselves. For the push queues, it's deleted for us, but we have to do it ourselves for pull queues. We can also set access control for Google user email addresses. We can grant a user the permission of user or writer. User permissions mean that they can list, get, lease, delete, and update tasks. And the difference between them and writers are, writers can also create tasks.

We're not going to dive into pull queues anymore, though I'll show a quick snippet of code that's provided by Google to help solidify the concept. So, in this example, we can see that they're defining a queue and then appending some tasks into a list and adding that list of tasks to the pull queue. And then in some other method, we define the queue and loop over the tasks returned from our lease and then we would delete them when they're done. So, it's more of a manual process than the push queue. And while this really does work just fine for sharing work across services, projects, and even platforms, I recommend using Cloud Pub/Sub in place of pull queues. It works across Cloud, Language, etc. And it very well documented with a lot more sample code available.

So, we checked out tasks queues, which are fantastic for doing work outside of the scope of a user request. But sometimes, we need to do things like send out scheduled emails maybe do some weekly data import or export or send monthly billing notices. App Engine has us covered. We can use these scheduled tasks with cron. We can use the cron.xml files for Java or the cron.yaml files for everything else to specify what we want to run and when.

Check out our cron example. It specifies that we call the tasks/email user every five minutes. All right, that seems a bit aggressive, wouldn't you say? Why don't we edit this? We're gonna say once a week and we can do that by changing it to every Monday at 9am. There, that's better Now, let's deploy it and let's see it. Once we deploy it We can see it in the task queues under the cron jobs and there it is, it's set to run every Monday. Just like with tasks, we'll want to set the handler to allow admins only to insure that no one calls this code from a web browser. So, between push queues, pull queues, and cron tasks, we have a lot of flexibility over how we run tasks.

In our next lesson, we'll review what we covered so far and we're gonna talk about where we go from here. So, if you're ready to wrap up our exploration of App Engine then, let's get started.

About the Author
Ben Lambert
Software Engineer
Learning Paths

Ben Lambert is a software engineer and was previously the lead author for DevOps and Microsoft Azure training content at Cloud Academy. His courses and learning paths covered Cloud Ecosystem technologies such as DC/OS, configuration management tools, and containers. As a software engineer, Ben’s experience includes building highly available web and mobile apps. When he’s not building software, he’s hiking, camping, or creating video games.