Hello, and welcome to Sprint Eight. We've spent a lot of effort so far getting the system to this point. Our next step is to get this running on a more production-ready server. We're going to use Google Cloud's Compute Engine to start up a virtual machine, so let's do a little preparation. First, I wanna clear out the existing test data from Firestore. And I'm leaving this here because I want you to see what that test data is going to look like. Here's a quick look at the data structure for the final data. We have documents for each publication. We're tracking the number of articles with account property, and we're storing the entities in a collection called "Ents". Each entity has its own document. This ID here is the result of using Python's hash function to hash the entity text. The reasoning for this was that I wanted a consistent method for getting IDs for each entity, but I didn't wanna use the text itself and run into any weird issues with specific character sets. Each of these Entity documents has a word property that stores the entity, and it has a count of how many times it was used for the given publication. Let's delete this collection, which will delete all of the test data. Okay, perfect. Let's start up a Virtual Machine. I'm going to name this "Ingest-0", and I'm going to run this in east4. I'll select a 16 core n1 standard. And I want to use Ubuntu 18.04 with a 50 gigabyte SSD. I'm going to use a service account that I created previously. This service account has project editor access. Now, it's far more permissive than it needs to be, however, I didn't want to invest time into locking this down for the demo, when it's really not the focus of the course. Let's jump forward once this is complete, and we'll go from there. Okay, our Virtual Machine is up and running, so we now have some place to deploy too. However, we still need a deployment plan. I'm going to create a few files here, and then we'll populate them in a moment. First will be bootstrap.sh. Next will be build.sh. And finally, deploy.sh. Okay, let's start with bootstrap. When we set up our vagrant Virtual Machine, we compiled Python 3.8 and we created our virtual environment. We need to do that for our newly-created Cloud Virtual Machine. Now this script does just that. It's exactly what we ran previously, with a little bit of extra to make sure that it doesn't run again. So, once it's complete it creates this file here that signals that the Virtual Machine has been bootstrapped. Let's check out our build script, which is responsible for creating a distribution. It starts by installing the dependencies, which includes our development dependencies, and that's gonna allow us to run pytests in development, make sure everything's working, before we run our build. Once it runs a test and everything is successful, it then creates a distribution. Let's fire these off and see how they work. We've seen the tests run, but calling this will collect the tests and it's going to run them. And, in our case, they're all passing. I wanna show what happens when we run this setup. To do that, you'll wanna watch the Explore here on the side. When we run this setup and we specify the sdist option, Setup Tools is going to create a distribution file. Okay, notice this dist directory got created here, and inside we have our application that has been tarred and gzipped. The deploy.sh file is going to use the Google Cloud SDK to configure the Virtual Machine. It starts by making a directory that we can use to store our code. Then it uses Secure Copy to copy that new distribution to the directory we just created. It's also going to copy over bootstrap and the supervisor configuration file. Next, it runs bootstrap. Then it installs our distribution, using the version of pip from our virtual environment. And, finally, it's going to install the spacy model. The complete process for building and deploying is to run our build script, followed by running our deploy script. When we run build, we run it from inside of our development VM, because that's where the version of Python that we've set up is configured. The deployment, however, uses G-Cloud. Now, my personal preference is to avoid configuring virtual machines to have any sort of credentials, and that includes authenticating to the G-Cloud SDK. So, while we run the build script from the Virtual Machine, I'm going to run the deployment from my host machine where I've already pre-authenticated G-Cloud. In order to deploy, we need to know the name of the virtual machine, which is ingest-0. This is going to bootstrap our Virtual Machine and install our distribution. And here it is. This is going to allow us to connect in via ssh. And we're going to set the name. And it'll take a second to connect, but here we are. Notice, we have our visual indicator that we're inside of our virtual environment. If we use htop we can get a sense for what this system is doing by default. We got 16 cores, roughly 60 gigs of RAM, and there are 32 processes running. None of them are Python 3.8, and that's what we'd expect, because we just installed it. The Python version for our activated environment is indeed Python 3.8, meaning that the bootstrap process did work successfully. If we attempt to call the back end with the defaults, we can see it did load up our spacy models, which means our workers are running. So, our deployment process works. Now, with this ready, we can actually start to test things out, and that's what we're going to do in the next sprint. So, whenever you're ready to start sending some data through the system, I'll see you in the next lesson.

Lectures

Course Introduction - Sprint 1 - Sprint 2 - Sprint 3 - Sprint 4 - Sprint 5 - Part One - Sprint 5 - Part Two - Sprint 6 - Sprint 7 - Sprint 9 - Post Mortem