Hello and welcome! My name is Ben Lambert and I'll be your instructor for this course. This course is intended to help you gain a better understanding of Python by building out an application that's a bit more complex than your standard hello world application.

When thinking about what topics to cover, and how to cover them, I did some searching online to see which topics developers were labeling as being advanced. I distilled those results and included some of my own and came up a rough set of topics including multi-processing, magic methods, unit testing, debugging, context managers, et cetera. I wanted to come up with an application that was complex enough to cover these topics. However, I didn't want it to become so contrived that I'm just showing each of these features in their own context. I wanted this to be more like an actual development project.

The past few projects that I've worked on have been rather different from each other, however, they all had the same shape. By that, I mean they were all some form of data ingestion where data was ingested, processed, persisted, and consumed. I searched for public datasets that we could use to make something interesting here and I found a dataset from a Kaggle competition. It's an eight gigabyte CSV file containing articles from different publications over the last several years. It made me curious to know what information we could learn by analyzing the topics that were mentioned in each publication. I decided that I want to extract any named entities from these articles and count them for each publication. And I wanted to see with these counts if any patterns emerged.

Now, when I say named entities, what I mean in this context is just specific people, places, and things. Here's why I found this interesting. Extracting named entities from text is a non-trivial task. It's a CPU bound problem. Meaning, the CPU is always going to be our bottleneck. And in Python, using multiple CPUs is not a straightforward thing. However, because it will allow us to process more text at the same time, in this case, it's worth doing. I started sketching out the basic process. I envisioned the holistic system as being capable of accepting articles, processing them to extract the named entities, and saving the results to a database. And then, that data could be consumed with a web interface.

This course is going to focus on the ingestion and persistence. We'll cover the web consumer in another course. Once I had a rough sense for the system, I started doing some proof-of-concept work. I used different libraries to perform text extraction, and I used a library called Faker to generate some fake data to process so that I could start doing some basic performance benchmarking. Nothing too precise. Just rough numbers on how long entity extraction takes. The extraction process was pretty quick. I was processing fake articles that were around 40 to 50 words. So really, rather more of a Tweet than an article. However, the extraction process took less than half of a second usually.

The problem remained that this is still running on a single CPU, so that CPU is going to be our bottleneck. So I started testing with the multiprocessing module. Running the extraction in multiple processes allowed the code to utilize multiple cores, and that sped up the process, by roughly, the number of cores.

Now, multiprocessing doesn't give us all that power for free. Because we're running our code in multiple processes, sharing data is not that straightforward. We don't have easy access to shared memory. This required the addition of a message queue for passing data between processes.

After a week or so of testing, I landed on this. We have a web frontend protected by an API Key which allows us to submit an article. The frontend takes the article, it makes sure that the fields are not empty, and it adds it to a message queue, and we call this message queue the input queue. The backend creates a set of worker processes which are responsible for taking messages off of the queue and extracting the named entities. They'll cache the results until some number of articles has been processed, and then they'll transform the results and add them to a separate message queue.

We call this message queue the output queue. These worker processes are responsible for performing the CPU heavy work. The backend also creates a set of saver processes which are responsible for taking message off of the output queue and saving them to Cloud Firestore. There's not much CPU time required for saving the data, so by having it in its own process, we can separate out the CPU heavy work of entity extraction from the IO heavy work of persistence. During this course, we're going to build out this functionality.

I'm going to walk you through the way that I build apps. I'm not going to use terms such as the right way or the best way. Because, honestly, everything is contextual. So as we go, I'll show you how I build things, as well as my reasoning behind it. When I'm learning something new, I find that simply learning a concept exists is sometimes enough to fill in some knowledge gaps, and give me specific search terms for when I'm ready to learn more. So, I'm not going to explain every line of code. Hopefully, if you see something new that I don't actually cover in depth, it'll serve as a future search term.

The syllabus for this course is broken up into sprints because that mirrors the rough approach I follow when developing. Here's our breakdown. Sprint One will cover setting up a development VM so that it has Python 3.8. In Sprint Two, we'll be implementing the DataProcessor. In Sprint Three, we'll implement and test the Message Queue. Sprint Four will cover creating the Data Models, as well as creating a Shutdown manager. Sprint Five will cover creating the worker and saver processes in the backend. Sprint Six is going to cover creating the frontend API, and Sprint Seven will configure a Linux daemon to run the front and backend. Sprint Eight will cover deploying the application to a 16 core cloud virtual machine. In Sprint Nine, we're going to test the code and see how it performs under load, and we'll wrap all of this up with a Post Mortem where we'll talk about what worked, what didn't, and some of the lessons that were learned as I've built this out.

Here's what I hope that you'll get out of this course. By the end of the course, you'll be familiar with several useful standard library modules. You'll be familiar with several popular third-party libraries. You'll be familiar with some of the challenges revolving around multiprocessing. You'll be familiar with some Linux tools such as htop and strace, and you'll be able to add several new search terms to your educational road map.

Before starting, you'll want to be familiar with the programming language. Ideally, that will be Python 3. Knowing a bit about Linux will be helpful as well because we're going to spend a lot time on the command line. Knowing how to use Git will be helpful because that's where the completed code for this course resides. And other than that, as long as you're willing to fill in any knowledge gaps as you go along, you should be ready.

All of the source code for this course is available to download. You can check the course description for the links. I had so much fun creating this course! I hope that it is even a fraction as much fun to watch as it was to create. So, if you're interested in getting our development virtual machine configured, then I will see you in the first sprint!

Lectures

Sprint 1 - Sprint 2 - Sprint 3 - Sprint 4 - Sprint 5 - Part One - Sprint 5 - Part Two - Sprint 6 - Sprint 7 - Sprint 8 - Sprint 9 - Post Mortem