Container Technologies


Getting Started
Wrap Up
4m 43s
Container Technologies

In this lesson, you will be introduced to containers as used in Docker, walk through several use cases, and learn about the Docker toolchain. By the end of this lesson, you will understand container technology and Docker's role in the bigger picture.

You will begin with an overview of container history. You will start by learn about the chroot command, and its importance in isolating and managing applications.

Next, we will get into BSD jails and the use of FreeBSD jails.

We will delve into the world of containers, where they were first created, the changes that have happened to them in their short history, and how they were merged into the Linux kernel.

We will define the sysadmin, and how they use the power of cgroups to allocate and prioritize systems. This was eventually merged with namespacing to create the modern container tools.

The modern container, now known as Docker, arrived in 2014, and includes the cgroup, namespacing, and filesystem to create portable application containers. This is the key to the lesson.

After discussing its history, we will visit the use cases of Docker; from development environment automation to deployment.

And finally, we will then look at Docker toolchain and understand all the tools included: Engine, Compose, Machine, Host, Swarm, Registry, Data Center, and Cloud.


Hello and welcome back to the Introduction to Docker course from Cloud Academy. I'm Adam Hawkins, and I'll be your instructor for this lesson.

Here's our agenda. First, our topics, container technologies and their history, an introduction to Docker itself, Docker's use cases, and finally, the Docker toolchain. So given these topics, what can we expect to learn by the end of this lesson? By the end of this lesson, you'll be able to understand past and current container technologies and most importantly, understand Docker's role in the bigger picture. The overall objective is to familiarize you with container technologies and how Docker fits into the bigger picture.

So let's start out with a bit of history. Containerization technologies are not new by any means. Engineers have always desired better tools to isolate and manage applications on the underlying system. This effort dates back almost 40 years and continues today with small progressive changes. Container technologies started with locking down a process' file system access and have evolved complex management systems for things like CPU, memory, network, and other compute resources.

Let's see where it all began. The chroot command or change root, as I call it, was added to BSD in 1979. The change root command allowed a user to change the root directory for a given process. From that point on, the process could not access files outside the new root directory.

This may sound a bit confusing in text, so let's explain with an image. So starting on the left, we have a directory listing. Then when we invoke the change root command, given a directory, the process now assumes a new root directory. So running the same command now produces a different directory listing. The change root command was the first of its kind. Iteration Zero is never perfect, so of course, there were drawbacks. The process running in the change root still had full access to the system resources, think, CPU, memory, network, et cetera and also the users on the system.

Plus people eventually learned how to break out. Then BSD jails came along. FreeBSD jails were introduced in the year 2000. The FreeBSD manual describes jails as quote, "a type of operating system level virtualization." Operating system level virtualization is where the kernel of an operating system allows the existence of multiple isolated user space instances instead of just one. So in a nutshell, operating system virtualization is about processes unlike traditional virtualization, which is dealing with kernels.

Anywhos, back to jails. Jails went a step further than a change root by isolating the process from the host. Let's look at a jail's features. Jails virtualized access to the file system. They also isolated the set of users and isolated the networking stack. Jails were much more secure than a change root. However, a jail did not virtualize access to a CPU, memory, or IO.

That's where our next set of technologies fill in the gap. Google started development on process containers in the year 2006. They were eventually renamed to control groups or cgroups for short. Cgroups were merged to the Linux kernel in the year 2007. This was a major leap forward. Control groups provided a way to isolate and manage compute resources for process groups. A sysadmin could use this cgroup to allocate and prioritize CPU, allocate and prioritize memory, allocate and prioritize IO, allocate and prioritize network, do the accounting on all the above, and finally, checkpoint and restart the process.

So what does this mean in practice? Well, a sysadmin could allocate 30% of RAM to the web server, turn down the server's IO, and dedicate as much IO as possible to a database process. This is really powerful stuff, especially for shared servers. Well, what's after cgroups? Control groups were combined with other kernel namespacing features to create a full-featured container solution. Modern container tools focus on operating system level virtualization. Again, this just boils down to each container having its own unique namespace, users, filesystem, and managed resources.

This is where Docker comes in. Here's a quote from Docker Inc. describing Docker itself. "Docker containers wrap a piece of software in a complete filesystem that contains everything needed to run: code, runtime, system tools, system libraries, anything that can be installed on a server. This guarantees that the software will always run the same, regardless of its environment."

Now, we've arrived at the star of our show. Technically, Docker wraps up cgroups, namespacing, and a union filesystem to build portable application containers. Docker hit the scenes back in 2014. Since then, it has radically changed the industry by making container technologies accessible to engineers of all backgrounds. The technology is so ridiculously powerful that it's being used for many things. These days, it seems Docker and microservices go hand in hand.

Okay, so we have this tool, but what can we actually use it for? Let's look into some of the use cases. Building software is one of the most complex human endeavors. It's always changing and full of obligations. Complexity multiplies when engineers use multiple languages and even multiple data stores. Workflows become more complicated, and bootstrapping new team members never goes as expected. Containers may be applied to this software development phase for drastic productivity increases, especially in polyglot teams.

Let's examine a few Docker uses during the built or development phase. First up, development environment automation, say you're building a C program. You have a bunch of libraries and other things installed on the system. This can be packaged up as a Docker file, more on this later in the course, and committed to source control.

Now, every member will have the same environment, independent of their own system. Managing data stores, say one project depends on database version A. Another project depends on version B. Running both versions may not be impossible with your package manager. However, it's trivial to start a container for version A and version B and then point the application to talk to the appropriate container. Cross operating system development, consider a team using Linux, OS X, and Windows. Building the application on each platform will create many problems. Instead, package the application up as a Docker file, and then each team member will always run the same thing.

Development and production parity, build and use an image in development. Then use it for staging and production. You can be certain the same code is always running in the same way. I bet you can think of even more use cases. I know I could spend the entire day just thinking of this stuff because of how useful and cool it's been. However, we're on a tight schedule, so we've got to keep moving.

Building software is only half the battle. What do we do with it after we've created it? Well, we've got to deploy it. This is where containers really shine. I'm a bit production-biased these days, so I'm gonna put the most important and my favorite point first. Infrastructure standardization, this one is massive. DevOps and traditional teams can build standardized infrastructure. to run and scale any application. Some brand spanking new language comes out. Well, it's no problem. Deploy with Docker, and it doesn't matter what's actually inside the container.

Running and orchestrating containers in production is the hottest topic right now. Watch this space for future developments. CI build isolation, CI systems can be fickle. Each project may change the machine in some way or need to install some random software or drop artifacts wherever they want. Don't even get me started on how complicated each project's dependencies can be. All of these problems are a thing of the past with containers. Run each build in an ephemeral container and throw it away afterwards. No fuss, no muss. Testing new software, it's a happy day.

The newest version of language X was just released, and now it's time to upgrade. You just want to test it out, so you set up a virtual machine to not break your existing setup. This is a resource-heavy and time-consuming process. Docker makes this easy. Simply bump the image tag from language X to language Y. Build, run, and test. Distributing new software, you've just finished your tool in language Y. Unfortunately, your tool has a ton of dependencies that your users may not be knowledgeable enough to install. Instead, build a Docker image and push it to the Docker registry. Now, anyone can pull down your image and run your software. This may be especially nice for handing over builds to your QA team.

I've just covered four points here, but I could really talk about this stuff all day. It's hard to convey how useful and how much this has changed the way that I and my team have worked in the past two years. It really feels like we're living in the future with this stuff. This is just a small taste of what Docker can do. Docker Inc., the official company behind Docker and the other first-party tools, provides tools suited for various other use cases. These tools form the larger Docker toolchain. Let's take a look at that now.

The use cases we've covered so far have mainly focused on using the Docker Engine. The Docker Engine is the thing that actually runs the containers. However, as I mentioned before, there are also a bunch of other tools to look into as well. Let's look at the first and most popular one.

Docker Compose, Docker Compose is a CLI tool to manage multi-container applications via a YAML file. It's a popular option for development environments. We'll cover Docker Compose later into this course, so we won't go into it much detail here. Docker Machine is another CLI tool for creating Docker hosts.

So what's a Docker host anyways? Well, a Docker host is a machine running the Docker engine. Docker Machine can bootstrap Docker Host using a virtual machines or even launch machines into various cloud providers, such as Amazon or Azure. We'll use Docker Machine later in this course for some sample deployments.

Docker Swarm is a tool to build a cluster from multiple Docker hosts. It's Docker's entry into the orchestration space. This is another very hot space with furious competition from open source and other paid providers. We're not gonna cover Swarm in this course. However, I do recommend you check it out as soon as possible after completing this course.

The Docker Registry is a place to push Docker images to. You can think of it sort of like GitHub for Docker images. You can run your own registry or use the official hosted service. Paid plans are available if you require private images. We'll be interacting with the official Docker Registry throughout this course, so we won't go into much detail here.

Docker Data Center is a combination of a few other Docker products. First, it includes Universal Control Plane or UCP for short. UCP is a GUI for managing a Docker swarm cluster and the users allowed to access the cluster. It also includes Docker Trusted Registry, which is a self-hosted Docker registry aimed at larger teams. We will not cover Docker Data Center in this course, so please check it out after completing my course.

Docker Cloud is sort of like a hosted Docker swarm. You provide a machine in a public cloud, and it connects to the Docker Cloud. Then you can use this service to deploy and scale your own Docker applications on your own machines. This product is aimed at individuals or smaller teams who don't require something like Docker Data Center. We will not cover Docker Cloud in this course. However, it is a good deploy target for the things we'll build in this course. I recommend you check this one out after completing this course. These first-party offerings make up a large part of the Docker ecosystem.

Docker follows the philosophy of batteries included but removable. Thus, many first-party offerings may be replaced by open source solutions or other paid offerings. Here's an example. Don't want to pay for Docker Registry and your deployment AWS, then use AWS' Elastic Container Registry. Not too keen on swarm, use Mesos, Kubernetes, Rancher, or a host of other tools. It's a very exciting time for the ecosystem as third-party vendors compete against Docker. The end result is better tools for all of us. Together, all these tools provide many choices to build, ship, and run Docker containers.

All right, I think I've kept my mouth running for long enough. Now, it's time to wrap up what we've covered so far. So we started off this lesson with a in-depth dive into the history of container technologies. We saw that all started back in 1979 and took a huge leap forward with cgroups.

We also covered how modern container technologies such as Docker and LXC are built on top of previous work like cgroups and namespace isolation. Containers focus on operating system level virtualization instead of traditional virtualization. Remember, this is about process isolation and resource management instead of kernel isolation.

We also covered Docker's application to different stages in the software development process. And finally, we wrapped up with an overview of the entire toolchain to build, ship, and run Docker containers. Now we're ready to dive into the meat of the course.

The next series of lessons focus on the down and dirty bits of working with Docker containers, building Docker images, and using the Docker toolchain. So that's what you're here for, right? So hit next, and let's run our first Docker container.

About the Author

Adam is backend/service engineer turned deployment and infrastructure engineer. His passion is building rock solid services and equally powerful deployment pipelines. He has been working with Docker for years and leads the SRE team at Saltside. Outside of work he's a traveller, beach bum, and trance addict.