Zappa! or How to go Serverless with Django and Never Look Back

Maybe you’ve heard all the buzz about serverless technology? It offers the promise of never again spending time on web server configuration, meddling with permissions, just the pure joy of developing your web app and leaving all the heavy lifting to your cloud vendor. Pretty cool, eh?

However,  you’re not ready to forsake the blissful ergonomy of Django, which allows you to write up the DB model and turn it into a working web page in a matter of minutes, for developing, maintaining, and coordinating a fleet of microservices.

This presents us with a difficult choice between a) simple deployment, management, and limitless scalability or b) being a happy production developer.

Fortunately, thanks to Zappa, we can have the best of both worlds.

What is Zappa?

Zappa is a nifty open source software that allows you to quickly deploy an existing Django project (actually any WSGI-compatible project) on AWS Lambda and AWS API Gateway. How quickly? Amazingly so: Just one command: Zappa deploy! Ok, I’m cheating here. You also need to configure it, running Zappa in it and set a couple of params through a CLI wizard. Still, it’s a whole different magnitude compared to the classic webserver + WSGI interface.

One command, deploy. That’s a delivered cloud promise.

Serverless on Amazon Web Services

Zappa allows us to deploy a WSGI-compatible web app on the Amazon Web Services platform using two AWS products: Lambda and API Gateway, and S3.

You’re probably familiar with the latter, S3, an inexpensive storage service that allows you to create ‘buckets’ where you can upload your files. Zappa uses S3 to upload your application, wrapped so that AWS Lambda can spin it up. (Our course, Storage Fundamentals for AWS, is a great resource for learning more about S3.)

Lambda is the core of the AWS serverless offer. It allows you to run functions without having to manage the servers yourself. It supports many programming languages and has been recently updated to support Python 3.6 (before the latest update, you *could* use Python 3, but only through some unofficial shenanigans).

One of the advantages of Lambda is its billing model: You pay only for the compute time you consume; you’re not charged when your code isn’t running.

The other great selling point of AWS Lambda is scalability. Because it doesn’t have a permanent infrastructure, we can handle almost any load. There is no limit to the number of Lambda functions that you can run (apart from your bank account balance) and so there is no load that is too high to handle.

If you’d like to dive deeper in AWS Lambda, I would recommend Cloud Academy’s course, Compute Fundamentals for AWS, and the Introduction to AWS Lambda Hands-on Lab.

API Gateway is a fully managed service that acts as a “front door” to your application. It allows you to create, publish, maintain, monitor, and secure APIs at any scale by routing traffic to backend services running on Amazon EC2 or—and this is our case—to code running on AWS Lambda. It’s cost-effective, with no upfront costs: You only pay for the requests you receive and for the bytes transferred. You can learn more about AWS API Gateway in our Introduction to Amazon API Gateway course.

How Zappa Works

Zappa makes it really easy and straightforward to deploy Django using these tools.

Here’s how it works. When a request comes into API Gateway, the Gateway fires up a server inside a Lambda function. The server is then fed the request, which is handled by the Django app through the WSGI layer. When the response has been calculated, the server sends it to the API Gateway just before it gets destroyed. Then, the API Gateway returns a response to the client.
All of these steps happens in less than 30 minutes and add little to no overhead to your application.

Advantages of Zappa

Ok, so what are the advantages of this approach? Why should I deploy my Django app with Zappa instead of my usual combination of Nginx/uWsgi (or similar)?

First of all, by using AWS Lambda we can make our application globally available. This leads to several major advantages. The first is speed. If my data has to go all the way from, say, Rome to Los Angeles, it will take longer for the request/response round trip to finish. By deploying our app closer to our users, they can call an instance that is close by, allowing for faster round-trips. The second advantage is scalability. By deploying in multiple regions, we can handle a much larger load. Moreover, using AWS Lambda with Zappa, we make it so that there is one server for every single request coming in through API Gateway. Our app can scale in a tremendous way!

By going global, we also gain redundancy. While it’s true that AWS outages are rare, they still happen. Since outages are usually limited to a region or two (and not the whole system), having our app deploy across multiple zones grants us increased protection against any potential disruptions.

Walkthrough

Now that we’ve clarified what is Zappa, on which technologies it builds upon and why it’s a great choice for deploying a Django app, let’s put it to work.

Going forward, we’ll make some assumptions: that you’re working on a *NIX machine like Linux or OSX, and that you have Python 3.6 installed on your machine.

We will use Django 1.11 and the latest version of Zappa. (At the time of this post, it is the 0.42.0, but the release frequency is pretty high, so your version may be different. Please let me know if anything stops working with the newest versions of Zappa).
Finally, this walkthrough is extremely opinionated. I’ve put together the best practices that have worked for me. They may or may not fit you as well, so feel free to bend it if you are uncomfortable with any of these steps.

AWS Credentials

First of all, we need an AWS account. Register on AWS to get the Free Tier for a year.

After you’ve completed the registration, including the account activation, we need to save the AWS credentials on your machine.
Verify that you don’t have an AWS credentials profile file already in your system; it’s usually located here:

~/.aws/credentials

The file should contain your AWS Access Key ID and Access Key Secret and should be formatted like this:

[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key

Here you can find the AWS docs for creating the credentials through IAM.

Create the virtualenv

Zappa needs a working virtualenv to perform its magic. (I assume that you already know what a virtualenv is, but in any case, you can read more about it here).

mkdir zappatest
cd zappatest
virtualenv env
source env/bin/activate
pip install django zappa

Now that we’ve set up the virtualenv, let’s move on with the Django project.

Init the Django project

We will start from a very basic Django project. Just lunch

django-admin startproject zappatest .

and then, to test that everything works

python manage.py runserver

Settings

I adamantly adhere to the 12-factor guidelines. In particular, I want to talk about the third factor, Config. According to the guidelines, an app config—that is, everything that is likely to vary between deploys—must be strictly separated from the code. This includes, in our case, the Django settings that handle the connection to the DB or the cache, the secret key, the debug bool, and the allowed hosts.

To grant the separation, I usually use Django-environ. Django environ gives you tools to more easily read and parse env variables, also by allowing the env vars to be written in an .env file. This way, we can write the DATABASES settings as

DATABASES = {'default': env.db()}

and add a connection string like this in our .env file:

DATABASE_URL=psql://user:password@127.0.0.1:5432/database-name

Just remember not to add the .env to your vcs system (put it in the .gitignore or similar).

Django environ allows you to handle all of the Django settings though its API. It’s a great app, one that I suggest that you check out, even outside the scope of this tutorial.

Zappa init

We can start with the first deploy through Zappa, even if our app doesn’t offer much at the moment.

Just type

zappa init

to start the Zappa init wizard.

███████╗ █████╗ ██████╗ ██████╗  █████╗
╚══███╔╝██╔══██╗██╔══██╗██╔══██╗██╔══██╗
  ███╔╝ ███████║██████╔╝██████╔╝███████║
 ███╔╝  ██╔══██║██╔═══╝ ██╔═══╝ ██╔══██║
███████╗██║  ██║██║     ██║     ██║  ██║
╚══════╝╚═╝  ╚═╝╚═╝     ╚═╝     ╚═╝  ╚═╝
Welcome to Zappa!
Zappa is a system for running server-less Python web applications on AWS Lambda and AWS API Gateway.
This `init` command will help you create and configure your new Zappa deployment.
Let's get started!
Your Zappa configuration can support multiple production stages, like 'dev', 'staging', and 'production'.
What do you want to call this environment (default 'dev'):

First, Zappa asks you to define which env you want to configure, as it’s able to handle multiple envs. Let’s stick with the default ‘dev’.

AWS Lambda and API Gateway are only available in certain regions. Let's check to make sure you have a profile set up in one that will work.
We found the following profiles: default. Which would you like us to use? (default 'default'):

After that, it will read the AWS Credentials file (that we created before) and read all the available profiles. We should have only ‘default’ at the moment, so just press enter to confirm. In case you have multiple AWS Credentials profiles on your machine, choose the appropriate one. Beware that AWS Lambda and Gateway, as noted, are not available in a few regions.

Your Zappa deployments will need to be uploaded to a private S3 bucket.
If you don't have a bucket yet, we'll create one for you too.
What do you want call your bucket? (default 'zappa-honv26p9p'):

Zappa needs an S3 bucket to push your app. You can use an existing bucket or let Zappa generate one for you. For the purpose of this tutorial, let’s stick with the default. Press enter to have Zappa generate the bucket for you.

It looks like this is a Django application!
What is the module path to your projects's Django settings?
(This will likely be something like 'your_project.settings')
Where are your project's settings?:

As you can see, Zappa is clever enough to recognize that our app is a Django project, quickening the init process. Just type in the setting module:

zappatest.settings
You can optionally deploy to all available regions in order to provide fast global service.
If you are using Zappa for the first time, you probably don't want to do this!
Would you like to deploy this application globally? (default 'n') [y/n/(p)rimary]:

The possibility of a global deployment was one of the selling points of a serverless architecture that we mentioned in the introduction. However, for our first experiment, we should stick with a simpler configuration. Press enter to select the default ‘No’.

Okay, here's your zappa_settings.js:
{
    "dev": {
        "aws_region": "us-west-2",
        "django_settings": "zappatest.settings",
        "profile_name": "default",
        "s3_bucket": "zappa-honv26p9p"
    }
}
Does this look okay? (default 'y') [y/n]:

We are now prompted with the configuration generated for us by the wizard, which, as you can see, is pretty darn simple. Confirm it to be prompted with a recap of all the available deployment commands:

Done! Now you can deploy your Zappa application by executing:
	$ zappa deploy dev
After that, you can update your application code with:
	$ zappa update dev
To learn more, check out our project page on GitHub here: https://github.com/Miserlou/Zappa
and stop by our Slack channel here: https://slack.zappa.io
Enjoy!,
 ~ Team Zappa!

And we’re done! Zappa is configured, ready for the deploy!

First deploy

Time for our first deploy! Just type

zappa deploy dev

and watch the magic happen!

Calling deploy for stage dev..
Creating zappatest-dev-ZappaLambdaExecutionRole IAM Role..
Creating zappa-permissions policy on zappatest-dev-ZappaLambdaExecutionRole IAM Role.
Downloading and installing dependencies..
Packaging project as zip..
Uploading zappatest-dev-1496245095.zip (11.0MiB)..
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.5M/11.5M [00:15<00:00, 751KB/s]
Scheduling..
Scheduled zappatest-dev-zappa-keep-warm-handler.keep_warm_callback!
Uploading zappatest-dev-template-1496245132.json (1.6KiB)..
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.61K/1.61K [00:01<00:00, 1.23KB/s]
Waiting for stack zappatest-dev to create (this can take a bit)..
 75%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                                      | 3/4 [00:10<00:05,  5.48s/res]
Deploying API Gateway..
Deployment complete!: https://1kfcd79nh5.execute-api.us-west-2.amazonaws.com/dev

Visiting the returned URL, however, raises an exception:

"{u'message': u'An uncaught exception happened while servicing this request. You can investigate this with the `zappa tail` command.', u'traceback': ['Traceback (most recent call last):\\n', '  File \"/var/task/handler.py\", line 427, in handler\\n    response = Response.from_app(self.wsgi_app, environ)\\n', '  File \"/private/var/folders/ng/z5x6b_t94qg13yx2gh3rzd_h0000gn/T/pip-build-XVZTup/Werkzeug/werkzeug/wrappers.py\", line 903, in from_app\\n', '  File \"/private/var/folders/ng/z5x6b_t94qg13yx2gh3rzd_h0000gn/T/pip-build-XVZTup/Werkzeug/werkzeug/test.py\", line 884, in run_wsgi_app\\n', \"TypeError: 'NoneType' object is not callable\\n\"]}"

This is because we didn’t set the ALLOWED_HOSTS in the Django setting. We couldn’t, as we did not know the actual URL. Add the returned URL to the ALLOWED_HOSTS and run

zappa update dev

to have a working Django site, deployed on a serverless architecture.
Next, we need a database.

Configure the database

One of the few parts that Zappa does not handle for us is the DB configuration. The best way to do it is to leverage the AWS RDB service. As the creation of a DB is outside the scope of this tutorial, I urge you to take Cloud Academy’s course on AWS RDB and the hands-on lab.

After setting up the DB, you need to add the VPC configuration inside the zappa_settings.json, to allow your lambda functions to connect with the DB. Annotate the subnets and security group that you have set for the DB, then add a ‘vpc_setting’ key to your configuration, like this:

{
    "dev": {
        "aws_region": "us-west-2",
        "django_settings": "zappatest.settings",
        "profile_name": "default",
        "s3_bucket": "zappa-honv26p9p"
        "vpc_config" : {
            "SubnetIds": [ "subnet-xxxxxxxx", "subnet-yyyyyyyy" ],
            "SecurityGroupIds": [ "sg-zzzzzzzz" ]
        },
    }
}

Finally, update the DB settings of Django. If you’re using Django-Environ, add an env variable to the zappa_settings.json, like this:

{
    "dev": {
        "aws_region": "us-west-2",
        "django_settings": "zappatest.settings",
        "profile_name": "default",
        "s3_bucket": "zappa-honv26p9p"
        "vpc_config" : {
            "SubnetIds": [ "subnet-xxxxxxxx", "subnet-yyyyyyyy" ],
            "SecurityGroupIds": [ "sg-zzzzzzzz" ]
        },
        "environment_variables": {
            "DATABASE_URL": "psql://user:pass@rbd_url:5432/db_names"
        }
    }
}

Conclusion

I hope you’ve learned something from our quick walkthrough. There is much to build upon this, of course, but by now you should have all the means necessary to use and enjoy serverless Django, thanks to Zappa.

Cloud Academy