One of the best ways to learn new programming languages and concepts is to build something. Learning the syntax is always just the first step. After learning the syntax the question that arises tends to be: what should I build? Finding a project to build can be challenging if you don’t already have some problems in mind to solve.
Throughout this course, we’re going to learn more about Python 3 by building a data ingestion process. We’re going to go from setting up a development VM through to deploying the app to a 16 core, cloud VM where we can test. The application is going to allow us to submit articles to a front-end HTTP endpoint where they’ll be enqueued onto a multiprocessing queue. On the back-end, a set of worker processes will dequeue the articles, extract named entities from the article, and enqueue the results to be saved. A set of saver processes will dequeue the results and save the records to Cloud Firestore. Both the front and back-ends will be run together using supervisord. And we’ll use setuptools to create a software distribution used to deploy the app.
This course is broken up into sprints to give you a real-world development experience, and guide you through each step of building an application with Python.
If you have any feedback relating to this course, feel free to contact us at firstname.lastname@example.org.
- Configure a local development environment for an app using a VM
- Implement a data processor that can accept text, extract named entities, and return the results
- Implement a multi-process aware message queue and use Pytest
- Create data models to use as messages to pass on the message queue
- Create the backend for the application
- Create a web endpoint that we can use to enqueue our post models
- Implement a method for running the frontend and backend together
- Run the application using a dataset to see how the system performs under actual server load
This course is intended for software developers or anyone who already has some experience of building simple apps in Python and wants to move on to something more complex.
To get the most out of this course, you should be familiar with Python, ideally Python 3, and have some knowledge Linux and how to use Git.
Hello, and welcome to Sprint 1. Our goal for this sprint is to get a local development environment configured. My preference is to develop inside virtual machines, rather than using my host OS. Using a VM allows for a layer of isolation that keeps all of the clutter out of my host. Since we're not going to be run with Docker in production, I'm not gonna use it for development. Instead, I'm going to use Vagrant to create an Ubuntu VM. Vagrant is a virtual machine manager from Hashicorp. It uses a Ruby DSL that lives inside a file named Vagrantfile. This one specifies that we want an Ubuntu Bionic64, and that it should bind both the host and guest on port 8000. This is going to allow us to use localhost on port 8000 on our host and have access to our guest. I'm using Vagrant 2.2.9, with no plugins. And to get the VM, we can call vagrant up from the directory where our vagrantfile resides.
Right off, notice it requires that we update this box and I'm seeing this because I downloaded this box earlier in testing. Running vagrant box update will get the latest version, and this is going to take a while.
Okay, with that done, if we run vagrant up again it tells us to destroy the current VM so let's do that. Vagrant destroy, and we'll say yes to the prompt, great. Now, I'm going to run vagrant up and then skip to when it's done, and here we are.
Next we can connect into the VM using vagrant ssh and here we are, great. Let's check the existing version of Python, which seems is not installed. However, it does mention Python 3 is. Checking that version, we have 3.6.9. We're gonna use a newer version of Python. We're gonna use 3.8.3, and we're going to download and compile it for ourselves.
First, we need to install some development dependencies. Let's update apt. And now we need to install some libraries that will be used to compile Python on Ubuntu. Again, this is going to take a while. Okay, with this done I'll use curl to download Python 3.8.3. Let's extract it with tar.
Okay, notice we now have our source directory. This configure file here is what we need next. We're going to run it and pass the enable optimizations flag. This is going to take a while, so I'll see you when it's done. Okay, next we're going to use Make to actually compile Python.
One of the flags we're going to use allows us to specify the number of cores to use. To make it as fast as possible I'm going to have it use all of cores. And using nproc we can determine how many cores we have. So it says we have two. Using htop we can verify. You can see we do have just the two cores for this VM. Let's call make with -j and pass in the results of nproc. Okay, here it is.
We finished compiling Python 3.8. Notice that if we were to check the version of Python 3 that's installed, it still says it's Python 3.6. This is good, we don't wanna overwrite the default version. Our version of Python is accessible through its full name Python 3.8. So we have Python 3.8 installed. However, we're going to be using a virtual environment. So let's set that up now.
I'm going to paste in some commands, starting with this one. This directory here is where the code will live in production, so I'm going to create this directory to help me work out the deployment. To create the virtual environment we need to use the venv module and specify the path where we want our new virtual environment to reside. I'm going to store this in my home directory under venv.
To use a virtual environment we either have to be activate that environment or use the fully qualified path to the binaries. I'm going to edit the bashrc file so that our environment is activated whenever we connect in over ssh. If I tail the bashrc file now, we can see that the environment is activated by sourcing this activate file. Let's log out and then back in, so that we can actually see how this will work.
Okay, so we're connected back in. Notice that our prompt has changed. We see this venv prepended to it. That's our first cue. It's a visual cue that tells us something is different. If we check the version of Python now, notice that it's Python 3.8.3. And we accessed it by just calling Python, rather than specifying the full name Python 3.8.
With this environment active, running pip installs will install them into this virtual environment. I find it valuable, even inside a virtual machine to use virtual environments. It's going to prevent future headaches because different apps can use different versions of the same library without conflicting.
Okay, with this done, we now have an Ubuntu VM with Python 3.8 installed. We have a virtual environment configured and it lives in our home directory and it's activated whenever we log in. Some advice, virtual machines are going to drain a laptop battery very quickly, so make sure to shut them down. You can do that with vagrant halt and you can destroy a VM when you're done with it using vagrant destroy.
All right, that's going to wraps up our first sprint. We were successfully able to get our development environment configured. In our next sprint, we're gonna work on creating the entity extraction process. So whenever you're ready, I will see you in the next sprint.
Ben Lambert is a software engineer and was previously the lead author for DevOps and Microsoft Azure training content at Cloud Academy. His courses and learning paths covered Cloud Ecosystem technologies such as DC/OS, configuration management tools, and containers. As a software engineer, Ben’s experience includes building highly available web and mobile apps. When he’s not building software, he’s hiking, camping, or creating video games.