This course is the second course in two-part series on how to build an application in Python. In the first course, we built a data ingestion process that extracted named entities from articles across a few different publications. We extracted named entities from around 100,000 articles and we saved the results into Cloud Firestore. In this second course, we'll explore the codebase for a web application used to visualize those results.
We'll kick off the course by checking out some quality of life changes implemented while developing this app. That includes a custom bash theme, a replacement debugger, a debugger command for starting an IPython shell, and pytest plugins. After that, we're going to review the data access layer and its accompanying tests. That's going to include multiple implementations of each data access service. Then we'll check out Python's web application standard.
Next, we'll review the web application layer and its accompanying tests. That's going to include a fast web application framework, custom middleware, request hooks, and application configuration. After that, we're going to review the presentation layer, including a Vue.js app and materialize CSS. Finally, we're going to run the app locally and trace some requests through the application using the debugger.
If you have any feedback relating to this course, feel free to contact us at firstname.lastname@example.org≥
- Implement a few developer quality of life changes
- Implement a testable data access layer
- Understand how a Python web app operates
- Understand how to build and test a more complex web app
- Understand how to use ipdb and IPython
- Enhance your knowledge of the Python programming language
This course is intended for software developers or anyone who wants to learn more about building apps with Python.
- Before taking this course, please make sure you have taken the first course in this two-part series: Building a Python Application: Course One
- You should also have an understanding of Python 3, Linux CLI, HTML/JS, and Git
The source code for the course is available on GitHub.
Hello and welcome! We're going to use this lesson to take time to get the application running so that we can use it locally. I have two terminals here, the top is going to be where we start up the development web server that will serve up our HTML. The bottom is where we'll run the WSGI app with Gunicorn.
or the web server I'm in the web/assets directory, which contains the index.html file. To start the web server, we're use Python's built-in simple web server. This will serve up whatever is in the current directory. If I load this page in a browser, we can see that we get our navigation loading, though, we don't have any page data and that's because we need to start up our WSGI application notice the use of these environment variables here. These are how our create_app function is able to configure the app to use Cloud Firestore and Cloud Storage.
Since these are currently set to use the real implementation the app is going to create a Firestore and Cloud storage client. These client needs credentials in order to connect. I have a Google Cloud service account key saved locally, and I've enabled the VM to access it. Using this environment variable here we can tell the SDK where to find the credentials. We're binding to port 8080 inside the development VM. To make sure that it's accessible to the host I've set up a mapping in the Vagrantfile, so that it maps 8080 to the host and guest.
The application callable that we want Gunicorn to use is inside the web package, its inside the main module, and we want Gunicorn to call create_app. Recall that create_app is going to return a callable, and that's a falcon WSGI application.
If we run this, and then head over to the browser to reload it we get actual data. The reason we see this images here is that I generated the images prior to this by using CURL to submit a post request to the Word Cloud resource, and passed in the authorization header token.
Clicking on preview will open the sidebar and we can page through the results. All of the publications basically look the same. Opening up DevTools we can see the first API call is to publications. The response is a list of publications consisting of object with name, count, and image URI.
Vuejs uses these image_uris to fetch the images from Cloud Storage. Currently the base URL is hard coded in the front end, and this image URI is appended to it. Let's close and reopen DevTools, and then we can click preview to see the network call. This calls our frequencies resource, and passes the publication name in the URL. And we can see that the results are a list of word counts.
Let's click to see the next page, notice it includes the word and count in the URL as params. These are our checkpoint that we use to page the data. Clicking it again, we can see that the front end is taking the last word and count, and including them as parameters.
If we scroll down so we can actually see the bottom of this list. The last results are the word German with a count of 68. When we click next, these are values that will be in the URL and there they are! Recall that the frequencies method uses these values to fetch the next 10 entities following the entity with a word German and count of 68.
Let's debug this to better understand. Let's add some breakpoints to the code and follow a request through the application. I'm going to add breakpoints at the start of each Falcon resource's here. Once these are in place we can run the code again and inspect the state of the app. And I will rerun this with the same configuration as last time.
By reloading the browser we can see the navigation load and it's called publications, and that's going to trigger the debugger, so notice we're inside an IPDB debugging session. Typing "ll" is going to display a few more lines than just the default three. We could also set the default to something other than three though this is fine.
Notice, that we're inside the on_get method, and line 92 will be called next. Inspecting locals we can see that self is our publications resource. We have request and response objects, so these first three locals here are our method's arguments. Typing n will move to the next line in this method.
So, the next line to run is going to be line 94, and it's going to convert the named tuples to dictionaries, which means the publication call actually return successfully and returned our generator. And we can see that based on its type, that we do indeed have a generator.
Check it out if I keep trying to move through this app with next, we just keep doing next, next we going end up being here while, so typing c will continue until another breakpoint is hit. So, our call to the publications resource worked and returned the results. If we check the browser, we can see it here.
Let's click on one of the preview buttons to start a request for the frequencies resource. So, we're here inside the frequency resource's this is on_get method. We can see that self is a frequencies resource. Let's move forward using next. And if we inspect the checkpoint variable, we'll see that it set to a Tuple of none and none. This what we would expect, since this is the first page of these results and doesn't have a check point.
Let's continue this and let it complete and over in the browser we have our data. By clicking next we can request the next page. Now, we're back in the frequencies resource. I'm just going to move forward until the checkpoint is set and notice it's now Amazon and 128. If we inspect the request object's representation notice that we have this full URL here and it shows our URL parameters.
Let's use our customized interactive shell by typing interact I. And calling locals here shows us a pretty-printed dictionary. This interactive shell uses tab completion which I find does make debugging easier. Notice if I type tap now we get this list that we can navigate with a keyboard. It even shows some data type info. However, I've found that the first time I use it, it doesn't fully load. I find if I just clear out whatever property am on try it one more time it does works.
Notice it shows us some of these are instances some are functions, we even get this nice function signature. I find this really useful for debugging it's also useful for me to learning new API because it cuts down the amount of time that I have to spend searching online. Also, having pretty-print enabled by default does make it a bit easier to inspect objects.
Okay, to close out of the interactive shell we use Ctrl + D. Let's see where in our debugging session we are right now. Right, we just populated the checkpoint. Okay, let's continue so that this code can complete. In the browser, we can see the results of the completed request.
All right, let's use our debugger to further inspect the data storage functionality. Let's click next to call frequencies route. And if we start up an interactive shell and we type self._storage. and then hit TAB. Notice we have the DB property, and our three functions.
If we select DB, we can further see the methods from the Firestore client we don't need to spend any time drilling into that. So, let's check out the methods that we implemented. Let's look at the publications. Notice it returns a generator. Let's consume it by wrapping it in a list and we get our list of named tuples. Calling it again and passing in a bucket name is going to change the image URL and here we have the image with a new URL path named path.
Okay, I wanna set up a different configuration. I want to run this app again, only I wanna use our NoOp storage implementations. All right, I've removed all of the environment variables, and we're left with having Gunicorn call our create_app function, which will use the default NoOp versions. I'm also using a timeout of an hour so that we can debug without Gunicorn killing our process because it thinks that it's unresponsive.
If we reload the page to call the publications resource we end up in our on_get method. And notice the type of the data storage object is our NoOp data storage. Let's go interactive. I like to check locals whenever I debug it's just a habit for me.
If we call the storage object's publications method and wrap it in a list, we can see that we have our fake version which behaves the same as the real one except it's just using generated result. And just like the real one, if we add a bucket name we can see that it prepends that to the image URL.
Okay, let's detach the interactive shell, and continue the debugger. Back on the web page, notice we have our fake publications, and counts. We don't have any images that's because we didn't generate any fake images. However, we could generate them if they seem like it would be a good addiction for local development. All right, let's wrap up here.
In the next lesson we're going to summarize what we've covered during the course so far and will talk about next steps. So, whenever you're ready, I'll see you in the next lesson!
Course Introduction - Quality of Life for Developers - What Is It That We're Building? - Exploring the Data Access Layer - The Web Server Gateway Interface - Exploring the Web Application Layer - Exploring the Front End Code - Summary / Next Steps
Ben Lambert is a software engineer and was previously the lead author for DevOps and Microsoft Azure training content at Cloud Academy. His courses and learning paths covered Cloud Ecosystem technologies such as DC/OS, configuration management tools, and containers. As a software engineer, Ben’s experience includes building highly available web and mobile apps. When he’s not building software, he’s hiking, camping, or creating video games.