The course is part of this learning path
In this lesson, we will finish up the application deployment process in Chef. The end goal is to deploy this entire application on any node that is managed by Chef.
We will verify that we have Apache web server installed, and then install MySQL. Next, we will create a database and database user for the application code. Finally, we will install a few packages: Git, a Python package manager called pip, and mod-WSGI, which is an Apache module to run a Python app.
You will use git to download the app source code from Github, and install its dependencies. We will then copy the files to a directory where Apache expects them.
You will view the recipes directory including ones for installing apache, for installing MySQL, and for installing the application.
We will introduce you to the Chef Supermarket, a community that creates and shares their cookbooks. We will work with Berkshelf to download cookbooks to make sure they are the most recent versions.
After all this, we will execute the kitchen destroy command to clean up VMs that are not of use to us at this time.
Finally, we will perform a recap of the lesson.
Welcome back! I hope by now you’re starting to feel comfortable with the way that Chef operates, at least conceptually. In the previous lesson I made the default recipe a bit more generic, so that it would run on multiple Linux platforms.
In this lesson there’s quite a bit of work to do in order to finish up the application deployment process.
Here’s a look at the LAMP application that I’m trying to install. While it’s not beautiful, it’s a basic web application, with a MySQL database. When I add records, they’re stored in the database, and the displayed in a list when the page is loaded. So this is the end objective. I want to be able to deploy this entire application on any node that is managed by Chef.
In order get this LAMP application installed, I have a few requirements. The first is to have the Apache web server installed.
Next I need to install MySQL. Then I need to create a database and database user for the application code. The application expects a database named “appdata” and a MySQL user named “appuser.”
Then I need to install a few packages. The first is Git, so that I can download the application code, the second is a Python package manager called pip, and the last is an Apache module that allows Apache to run a Python app, and that’s called mod-WSGI.
With those packages installed I can use git to download the source code for the app from Github. Once the code is downloaded it needs to have its dependencies installed. It’s worth mentioning, that I don’t recommend building your application on the server where it’ll reside. It’s a better practice to have it built as a part of the continuous delivery process. If you’re already familiar with continuous delivery as a concept, then you know what I’m talking about. If you’re not familiar with it you might want to check out the Introduction to Continuous Deliver course that’s a part of the Intro to DevOps learning path.
After the application has it’s dependencies installed, I can then copy the files over to a directory where Apache expects them. And Apache knows where to expect them based on the configuration file that I’ll deploy as a template.
So that’s a high level overview of what needs to happen. In the previous lessons I typed out the recipes in real time. However there’s a lot more code to write for this, so I’ve written it offline, and I’ll go through it with you step by step.
The first thing to show you is the recipes directory, where I have a few new recipes. I’ve created a recipe for installing apache, one for installing MySQL, and another for installing the application.
Notice there’s still a default.rb file. I want the default behavior of this cookbook to be that it installs the complete application. So to do that I removed the code that was in the default.rb, and I’ve replaced it with this section here to update the apt cache, followed by these three “include_recipe” method calls.
Hopefully you recall this method from earlier in the course. This method allows you to include other recipes as a part of your own. Notice how it works, I’m passing in the cookbook name, and recipe, and they’re joined with two colons. So if anyone runs this default recipe, it’s the same as running the apache, dbsetup and app recipes. And the order matters. These recipes will be executed from top to bottom.
So, since the apache recipe is first, let’s check it out. In place of the code I wrote in the previous lesson, I’m now using these two lines here to include the default recipe for a cookbook named Apache2, and a recipe named mod_wsgi from the same cookbook. I’ll explain where these cookbooks come from shortly. For now, let’s keep looking at the code I’ve written.
Looking back at the default.rb file, you can see the next recipe included the dbsetup recipe.
So, let’s check that out.
There’s a lot happening in this recipe. It starts off by installing the mysql client with the mysql_client resource.
Then I set the initial database root password in a variable. In production you wouldn’t do this, because storing passwords in code is not very secure. You’d use either an encrypted data bag item or Chef Vault, which is an easy to use wrapper around data bags for managing secrets. However, since there’s a lot to learn with Chef, I want to introduce this stuff slowly.
This next section makes sure the MySQL service is installed and running using the mysql_service resource. There are quite a few attributes set here.
The first is the bind address, and in this case it’s set so that connections can only be established from the localhost, and not the outside world. Next is the port, followed by the data directory. And then the initial root password, which is set to the password variable. After that is the socket file which will be different depending on the platform, followed by the create and start actions.
The next resource is the mysql2_chef_gem resource, which makes sure the mysql2 gem is installed. The next two resources require that gem, and if it’s not there they’ll fail. If you’re not familiar with it, it’s a Ruby gem for interacting with mysql databases from Ruby.
Now, this next portion of the code just creates a hash containing the connection information used to connect to MySQL. And I’m adding it here so that I don’t need to duplicate it in the two resources below.
So, at this point, the MySQL client is installed, the server is installed, and its service is running. And now, I need to create a database named “appdata” and that’s what this mysql_database resources here does. It takes the name and uses that for the database name. The connection property is set from the conn variable, and then the action is set to create.
The last resource in this recipe creates a MySQL user. It uses the mysql_database_user resources, and passes the name of 'appuser.’ The connection uses the conn variable, and then there are two actions, create, and grant. Create makes sure the user exists. Grant makes sure that the user has permissions. And by default grant applies full permissions to the user.
The last recipe included in the default recipe is the app recipe, so let’s check that out next.
If you recall, I said that in order to get this application running I’d need to install Git, pip and an Apache module called mod-wsgi. Because git uses the same package name on Debian and Redhat based systems, I can install it with the package resource by passing in the name of ‘git.’
With git installed, I can use the git resource to download the application code from Github. Notice the name isn’t a hardcoded string, it’s an attribute. The attributes are pulling from the default.rb file in the attributes directory. These are automatically fetched when Chef runs, so if you set an attribute, you can access it with the node.
I’m using the value_for_platform_family to conditionally set the apache_settings key to be a hash that I can look up information in later.
And then I’m also setting a couple attributes that’ll point to the git repo, and the directory to download the app to.
Back in the app.rb, you can see that the git resource is using that “app_repo_url” attribute to set the repository property. And then the action here is to sync. There are a few possible actions, and the reason I chose this one is that sync will either clone or checkout depending on if the repo exists locally, and it will also reset the HEAD which means it’ll discard an local changes. That tends to work well if you’ve made local changes that you don’t really want to keep, such as performing a local build.
The next resource is the python_package, which makes sure Python 2.7 is installed, and it’s also installing pip, which is one of the requirements I mentioned for this cookbook at the start of the lesson.
Then the pip_requirements resource will install the python dependencies that are listed in the requirements.txt file. Understanding the inner workings of things such as pip, and the requirements file isn’t required. So if you’re new to it, don’t worry. The only thing you need to know is that this pip_requirements will install any of the downloaded application’s requirements.
The next step in this is to use the bash resource to copy the app over to the /var/www/webapp folder using rsync.
Next up I need to configure Apache to be able to run my web application, and that’s what this web_app resource does. This template here is the configuration file that I want Apache to use, and it’s stored in the templates directory.
Here’s the contents of the config file, and all that’s happening is it tells Apache where to expect the application code. And some information about the Python app. The interesting part is that it’s dynamically pulling the user to run the web app under, since CentOS and Ubuntu have different users for that by default. These pull from the defaults.rb file in the attributes directory.
And then the final bit here is to make sure the owner of /var/www directory and its sub-directories is the same user from the template file I just showed you.
Okay, so there a lot going on there. And you might think it’s time to test this out. However there’s still one piece that’s missing.
Remember the two include_recipe methods being used in the apache recipe?
See these two resources here in the app recipe, the python_runtime and pip_requirements? …
What about all of these resources in the the dbsetup recipe? Do you recall seeing any of those in the list of resources I showed you earlier in the course? Probably not. And that’s because these aren’t built-in resources. These are functionality from Cookbooks developed by the Chef community, and shared online.
One of the great things about Chef is that there’s a community that creates and shares the cookbooks they create. And they post them here, on the Chef Supermarket. Having access to all of these pre-created cookbooks saves a lot time, and gives you instant access to all kinds of functionality that you don’t need to write for yourself.
Let’s take a look at the community cookbooks I’m using in my cookbook. The first is the “mysql” cookbook, which is responsible for providing the mysql_client and mysql_service resources. Skimming through, you can see that the attributes and actions are well documented, and there are examples too.
The next one I’m using is the “mysql2_chef_gem” cookbook, which is able to install the mysql2 gem on quite a few distros of Linux. Again, there are examples of how to use it, making it easy to get these cookbooks incorporated into your own cookbook. So this installs the mysql2 gem that the database cookbook uses, and that’s the next cookbook I’m using.
The database cookbook is cool to me, because it’s able to create databases and users for MySQL, Postgres and SQL Server. Which makes it rather versatile. This is the cookbook that provides the mysql_database and mysql_user resources that I’m using in the dbsetup cookbook to create the appdata database schema, and the appuser database user.
The next Cookbook that I’m using is the Poise Python cookbook, which can install Python and manage Python dependencies with pip.
And the final Cookbook is the Apache2 cookbook. The reason I’m using this, and not the one I created in the previous lesson is that this is much more feature rich. It’s able to install Apache for a variety of different platforms as well as being able to configure it, and install modules. And the best part -- which admittedly at times is also the worst part -- is that these are all managed by the community. Which means by and large it’s less code for you and your team to maintain.
So how do you get these cookbooks? Remember, at their core, cookbooks are simply files and folders. And so you could copy the cookbooks to your Chef workstation manually by using this button on the side here labeled “Download Cookbook.”
However, by manually downloading cookbooks, you end up taking on the responsibility for keep them up to date. And who needs that kind of headache? Fortunately there options for managing cookbook dependencies. And the option I’m going to show is a tool that is now a part of the Chef Developer Kit. The tool is called Berkshelf, and it’s a dependency manager. And it works by allowing you to specify all of the Cookbooks that you want, as well as their versions, and then install them.
The way Berkshelf does this is through the use of a file called the Berksfile.
Looking at the Berksfile for my cookbook you can see at the top there’s this line here that sets the source to Supermarket. That tells Berkshelf where to look to download cookbooks.
And then this line here tells Berkshelf to look inside the metadata.rb file for dependent cookbooks. And so I’ve specified those here in metadata.rb.
You don’t have to add the dependencies to the metadata file. You could add them solely to the Berksfile, however by adding dependencies to the metadata file you can tell Chef that if a cookbook you require isn’t found on the node, that it shouldn’t run. So I like to do that for clarity. It makes it more clear to others if they try and execute your Cookbook and don’t have the dependencies, because Chef will show an error about the missing cookbooks.
You might be wondering where the cookbook names and versions come from, and the answer is from the Supermarket.
If you look at the Berkshelf tab on the top of the page you can see that it has the code to add to the Berksfile. However, since I’m specifying my dependencies in the metadata.rb file, I just need to change the word cookbook to depends, and that’s it. If you want to understand more about the versioning syntax, check out Chef’s documentation shown here. https://docs.chef.io/config_rb_metadata.html
The best part about using Berkshelf to manage dependent cookbooks is that Kitchen knows all about Berkshelf. All you need to do is add your dependencies either in the metadata.rb or Berksfile, and run kitchen converge just like before. Kitchen will manage Berkshelf for you by running the berks install command at the start of the converge process.
So, I still have two platforms defined in the Kitchen configuration file, let’s see if the code works. Remember the ugly web application I showed you at the start? If all goes well that’s what I’ll see on ports 8081 and 8082.
Before running anything I’m going to issue the kitchen destroy command. And what this will do is remove those Vagrant VMs. This provides a clean slate to start over with, and it’s one of the aspects that makes testing with virtual environments so appealing.
Okay, with that complete I can issue the “kitchen converge” command. I’m going to fast forward because this could take a while.
Okay, when all was said and done these took about 6 minutes each. So this is the moment of truth. If I view them in the browser, will they work?
It looks like the Ubuntu version loads, which is a good sign. And it seems to be saving the values to the database. So that’s, awesome!
Next up is the CentOS version on port 8082, and it also loads...which looks promising. And it also seems to be working! This is great!
All of that work paid off, because I now have a cookbook that works cross distribution, to install my application with all of it’s components.
I’ve covered a lot in this lesson, let’s how much you remember.
Half of the resources in my cookbook are from the community cookbooks. What value do you see in using cookbooks from the Supermarket?
There are a lot of potential great answers here. The value I see in this approach is that you can incorporate well documented, well tested functionality into your cookbooks faster than if you develop it all for yourself.
The next question is: Which tool did I use for managing the Cookbook dependencies?
The answer is Berkshelf. While there are multiple options out there, Berkshelf is a widely used and well documented option.
The final question is: What do you see as the main value of Kitchen?
To me the value comes from being able to test my code on multiple VMs, and remove them when I’m done. It helps to build confidence in your code.
So far all of my work has been done either on my workstation, or the Vagrant VM. However now that the code works, it’s time to see the Chef Server in action. So in the next lesson I’ll setup a Chef Server, and if you’re ready to keep learning, then let’s get started with the next lesson!
About the Author
Ben Lambert is the Director of Engineering and was previously the lead author for DevOps and Microsoft Azure training content at Cloud Academy. His courses and learning paths covered Cloud Ecosystem technologies such as DC/OS, configuration management tools, and containers. As a software engineer, Ben’s experience includes building highly available web and mobile apps.
When he’s not building the first platform to run and measure enterprise transformation initiatives at Cloud Academy, he’s hiking, camping, or creating video games.