Post Mortem

One of the best ways to learn new programming languages and concepts is to build something. Learning the syntax is always just the first step. After learning the syntax the question that arises tends to be: what should I build? Finding a project to build can be challenging if you don’t already have some problems in mind to solve.

Throughout this lesson, we’re going to learn more about Python 3 by building a data ingestion process. We’re going to go from setting up a development VM through to deploying the app to a 16 core, cloud VM where we can test. The application is going to allow us to submit articles to a front-end HTTP endpoint where they’ll be enqueued onto a multiprocessing queue. On the back-end, a set of worker processes will dequeue the articles, extract named entities from the article, and enqueue the results to be saved. A set of saver processes will dequeue the results and save the records to Cloud Firestore. Both the front and back-ends will be run together using supervisord. And we’ll use setuptools to create a software distribution used to deploy the app.

This lesson is broken up into sprints to give you a real-world development experience, and guide you through each step of building an application with Python.

The source code for the lesson is available on GitHub.

If you have any feedback relating to this lesson, feel free to contact us at support@cloudacademy.com.

Learning Objectives

• Configure a local development environment for an app using a VM
• Implement a data processor that can accept text, extract named entities, and return the results
• Implement a multi-process aware message queue and use Pytest
• Create data models to use as messages to pass on the message queue
• Create the backend for the application
• Create a web endpoint that we can use to enqueue our post models
• Implement a method for running the frontend and backend together
• Run the application using a dataset to see how the system performs under actual server load

Intended Audience

This lesson is intended for software developers or anyone who already has some experience in building simple apps in Python and wants to move on to something more complex.

Prerequisites

To get the most out of this lesson, you should be familiar with Python, ideally Python 3, and have some knowledge of Linux and how to use Git.