It’s time to start digging into the details of Docker Compose. We’ll start by taking a close look at Compose files in this Lesson.

Agenda
I will start by giving you a brief introduction to the file format used for Compose files: YAML. If you haven’t used YAML before, you’ll learn enough to understand the Compose file examples used in this course.

Next, I will teach you about the root elements in a Compose file document. These are the top-level element of a Compose file and include: Compose file version, services in the application, volumes used by the services, and networks to be created. There is a lot of similarity between these sections of a Compose file and Docker commands that you are familiar with.

I will finish the lesson with a couple special topics in Compose files.
Let’s get started with YAML.

YAML

YAML is a data serialization language. Data serialization languages can be employed for a broad variety of programming scenarios including internet messaging, object persistence, or, in the case of compose, configuration files. Some of the design principles of YAML are that it should be human-friendly, and that it should work with any programming language. When YAML is stored in a file, the file can have a .yaml or .yml extension. Both are recognized as YAML. The capabilities outlined in the YAML specification are quite extensive. We’ll only really scratch the surface of what you can do with YAML.

Another, perhaps more common, data serialization language is JavaScript Object Notation (JSON). JSON formatted files are supported by Docker Compose. However, it is rare to see JSON Docker Compose files in practice. Comparing a YAML file to its JSON equivalent shows the cleanliness and fewer characters needed to represent the contents. In this example, Part of the reason why it can cut down on the line count is because it is whitespace sensitive. That means if you insert an extra space at the wrong place, the file will be corrupted. JSON on the other hand isn’t whitespace sensitive, meaning that you could squash all the whitespaces out and not harm the integrity of the file. However, readability would not fare so well. For that reason, JSON files tend to be formatted with abundant whitespace resulting in a less compact representation than YAML. It can take some getting used to working with a whitespace sensitive language, but IDEs tend to have support for formatting YAML files making it quite painless to work with. Some features of YAML, which is actually a superset of JSON, further enhance the compact representation of Compose files. You will see an example of this later in this lesson.

Data Types
Let’s take a look at a few basic data types in YAML. This list isn’t comprehensive but is enough to understand what usually goes into Compose files.

The first YAML data type we’ll consider are integers. Integers are whole numbers like zero or 1. You can also include a leading plus or minus sign to indicate positive or negative integers.

Strings are a sequence of characters that aren’t interpreted as a different data type. Strings can include spaces and the use of quotes to indicate the start and end of a string. Quotes are optional unless you use symbols that have a special meaning. Use single quotes around strings that use YAML syntactic characters like the pound symbol or colon. Use double quotes if you want to escape control characters like backslash n for newlines. If you want an integer to be interpreted as a string, you need to enclose it in quotes because it will be interpreted as an integer otherwise.

The null type is used to represent the absence of a value, or no value. It isn’t very common to see in Compose files but is a recognized data type in Compose. You use a tilde or the word null to represent it.

Booleans
Booleans are the last data type we’ll discuss. They indicate one of two values: true or false. Booleans can be represented with true, false, yes, no, on, off as well as the same words with the first letter capitalized or all the letters capitalized. In YAML any matches of a pattern including yes, and on get converted to true. This can cause unintended consequences if you have conditions testing Boolean values. For example, if you had a condition that was checking if a variable has a Boolean value of yes but it automatically got converted to true. Compose simply disallows the use of Booleans in contexts where such issues can arise to be on the defensive side.

You’ll see an error message similar to this if you use a Boolean value. In this particular case, I tried to use a Boolean as an environment variable. As the error message hints, only strings, number, or a null can be used. If you want to use true or false, yes or no, on or off as values, you need to use the string representation by wrapping them in quotes. If you ever encounter an error message involving true or false, this is probably what it relates to.

Collections
Collections are data structures that allow you to collect basic data type values in an organized manner. The first YAML collection we’ll consider is the mapping. Mappings are also known as dictionaries or hashes in different programming languages. Maps consist of keys mapped to values. The syntax for a mapping is a key followed by a colon, a space and then the value. The space is important. A mapping can have multiple key value pairs.

Mappings can also have mappings as values. Using a mapping as the value for a mapping is referred to as nested mappings. In Compose files the inner mapping is usually located on a new line with indentation.

There is also an inline syntax that lets you write a nested mapping on a single line. You use braces to wrap the inner mapping in this case. You may see this from time to time but I think it tends to hurt readability and should be avoided.

Sequences
The other kind of collection is a sequence. Sequences are also called lists or arrays in other languages. Sequences are simply lists of values.

You use dashes to indicate items in a sequence. Each item goes on its own line at the same level of indentation.

Sequences can also be nested. To represent an inner sequence, you use an indented dash on a new line below the outer sequence.

As with mappings, there is an inline syntax to represent a sequence on a single line. You use brackets to wrap a comma-separated list of items to use inline syntax. You see this inline syntax used for command options in Compose files.

Combos
You can also combine the sequence and mapping collections. For example, you can have a sequence as the value of a mapping. The dash in a sequence as a mapping value also counts as indentation, so you don’t have to use spaces for indentation when including a sequence in a mapping. That means both of these examples are valid YAML. You might prefer to indent for consistency, but they are not required and you will see both styles used in practice.

Similarly, you can use mappings in a sequence. There is again two ways to represent a mapping in a sequence. You can use a line with only a dash followed by the indented mapping, or you can include the first line of the mapping on the same line as the dash. Both styles get used and you should be aware of each.

The last thing I want to mention about YAML is that is supports inline comments. This makes it much more useful in terms of documenting Compose files than JSON which doesn’t support commenting. In YAML, a comment starts when a pound character is encountered and continues to the end of the line. The exception to a pound character starting a comment is if it’s within a quoted string.

That is enough YAML to get through the example Compose files used in this course. It’s also enough to understand most examples you might find online and enough for you to write your own Compose files. As I mentioned before, using an IDE that can automatically format YAML can save you some headaches. Compose also includes a command for verifying configuration files in case you need to verify your YAML syntax and configuration declared in a Compose file.

Compose files
Now that we’ve built up a foundation in YAML, we can focus on Compose file specifics. As mentioned earlier, YAML Compose files are the focus and make up the vast majority of Compose files in use. We will only consider version 3 Compose files.

From this compatibility table, you can see that version 3 Compose files require a Docker Engine of version 1.13 or higher. The Docker Compose command-line interface has a different release schedule than the docker engine, and requires version 1.10 or high for version 3 Compose files. Docker recommends using version 3 Compose files. There are multiple minor version numbers for version 3, for example 3.0 and 3.4. Unless otherwise noted, examples used in this course follow the 3.0 Compose file format. This covers versions of Docker released since the beginning of 2017.

https://docs.docker.com/compose/compose-file The Compose file reference is a great reference for understanding all the configuration options available to you in Compose files. This course covers many frequently used configuration options, but leaves out many that might be useful in certain situations. Let’s take a quick look at it now.

Here we are at the Compose file reference page. It defaults to showing reference material for the latest major version which is 3 at this time. There is a handy navigation bar on the right to see all the available configuration options

Just note that it can be confusing at times because some configuration options only apply to Docker swarm mode (deploy), some only apply when not running in swarm mode (security_opt), or specific Compose file minor versions (Extension fields), and sometimes different options are available for Docker running on Linux or windows based systems (isolation).

Version
A YAML Compose file is a mapping with several keys at the root or top level. The first one we’ll discuss is the version. The value for the version must be a string. The string can specify a major version number only, for example 3, or a major and minor version number, such as 3.1. If a only major version is included, the minor version is implied to be 0, or the earliest release. So if you want to use a feature that came out in the latest minor version release, you need to include it in the version string. The version string tells Compose how the contents of the file should be parsed.

Services
The services mapping is where you configure the containers created for services in your application. Each service is configured in a nested mapping under the services key. You can assign an arbitrary name for each service. In the image, the service names are web and redis. Each service has a nested mapping that declares the configuration for containers started for the service. The configuration is the main piece of declaring services in a Compose file, so let’s focus in on that.

Configure the container for the service.
Inside of the service configuration mapping, you declare the configuration options for service containers in a way that is similar to how you would configure containers using docker run command parameters.

This table shows how you would configure a container using the docker run command and the corresponding configuration key in a Compose file. The only required argument for docker run is the image name, which you specify using the image key in a Compose file. There are a few ways to configure a volume with docker run, but they all map to the volumes key in Compose files. There are different syntaxes in Compose to support the different volume configurations that you use different parameters for with docker run. The -p parameter to publish ports on the host corresponds to the ports key in Compose files. The -e parameter for setting environment variables in a container maps to the environment key in Compose files. With docker run you have two parameters for setting up logging and both parameters go into a nested mapping under the logging key. The last one that I’ll mention is security-opt for setting security options which only differs in the use of an underscore in Compose files. There are many more, but aside from a using YAML and slightly different names your experience with docker run will make writing Compose files easy.

Caveats
There are some points to be aware of for the correspondence between docker run and Compose file service configuration. Some docker run parameters that you might expect to be able to use in Compose only work in swarm mode. This is the case for setting runtime constraints such as -m for memory limit, or --cpus for number of cpus. It’s possible to run Docker in swarm mode with a single machine so it isn’t a significant barrier. However, Swarm mode is outside of the scope of this course so that’s all I will say about it.

Other docker run parameters such as -d to run in detached mode or --rm to clean up the container when it exits are specified through the command-line interface and not in the Compose file configuration.

Dependencies
Because Compose supports multi-container applications, there are additional options for configuring services that don’t exist with docker run.

The depends_on key provides a way to list the services a service depends on. Docker Compose can use the dependency relationships to determine the order to start services. If you tell Compose to start a specific service instead of the entire application, Compose can also use the information to automatically start any dependencies of the service. However, it’s important to note that Compose won’t wait for the dependencies to be ready before starting a service. For example, Compose can start a database before a web service but it can’t account for the time it takes the database process to be ready to handle connections. Because of this, it’s best practice to write your applications in a way that can tolerate connection failures. If that isn’t an option, you can use scripts that poll the dependencies to wait until they are ready. One such script is called wait-for-it.sh.

The other key that expresses dependencies is links. Links correspond to the link parameter of docker run allowing you to grant access to a container to access an exposed port on a private interface and to provide aliases to reach containers. Links in Compose carry the same meaning but additionally determine startup order of services, the same way depends_on does. Generally, networks are a better way to express communication relationships. We’ll discuss more about networks in a bit.

Examples
To get a taste of service configuration in Docker Compose, let’s look at some examples. I’ll arbitrarily use the redis image for the examples. Starting off simple, this docker run command will start a container named app-cache using the redis image.

The equivalent service configuration in a Compose file would like this. In the first line we need to specify that we are using version 3 of compose files. The services mapping is a pretty simple conversion of the docker run command parameters. The container name used as the service key, and the redis image argument used as the image key-value pair.

Now, if you specify a tag to pull a specific version of the image,
you add the same tag to the image value. Note that although the colon is a special character in YAML, you don’t need quotes around the string because colon only takes on special meaning when followed by a space.

If you want to the redis server port of 6379 available on the Docker host, you include the -p argument like so.

The corresponding Compose file includes a ports key which has a sequence of port strings. Just like with Docker run you can specify host and container port, or just the container port to allow Docker to choose an available host port. When specifying host and container port like in the example, it’s a good idea to put quotes around the string because YAML will parse numbers separated by a colon as sexagesimal or base 60 numbers if the numbers are less than 60.

In this last example, a command with arguments is added to override the default command.

The same string can be used as the value of the command mapping in a Compose file.

Or you can use the same syntax you would use in a Dockerfile for setting the default command of an image. In this case you do need quotes around the last argument “yes” otherwise it gets treated as a Boolean value. It’s best to always quote as you would in a Dockerfile. You might also recognize that syntax as the inline syntax of a sequence.

That means you can also express the command in the normal sequence syntax. This form can make long commands more readable. You get the idea of how to work with service configuration in Compose files through these examples. You might need to consult the Compose file reference to get the correct key names but it’s usually a fairly straightforward exercise to write the configuration.

Volumes
The next root key in the Compose file mapping is volumes. It is an optional key. You use the volume mapping in a way that is similar to how you use docker volume create. Services can reference volumes in each service’s volumes configuration key.

It’s a good time to point out that you can use volumes in the service’s configuration even if you don’t have a volumes key in your Compose file. The use case for the root volumes key is to use named volumes and to share volumes across services.

You can also declare external volumes that have been created outside of the context of the Compose file. For example, a volume created by docker volume create, or a different compose file. In a volume’s nested configuration mapping, you can set the external key to true to declare an external volume. If the external volume doesn’t exist, an error will be reported.

Take a look at this example Compose file using the root volumes key. There are two named volumes declared on lines 13 and 14. The first, called named-volume, doesn’t have any nested configuration. This will create a volume using the default local volume driver. YAML sees the absence of any value and represents it as a null. You could equivalently write a tilde or the word null for the value on line 13. The other named volume is called external-volume and is configured as an external volume.

In the app-cache service’s volumes configuration starting on line 5, you can see a few ways to declare volumes in Compose. The first is using a named volume in the root volumes key and will be mounted at /data in the container. The next example on line 9 uses a relative path to set the source of the mount. Relative paths are relative to the location of the Compose file. The last example on line 11 will have the Docker Engine create a volume automatically to mount at /tmp/stuff in the container.

Lastly for volumes, you can configure a custom volume driver using the driver and driver_opts keys. The named volume in the example called ebs-volume uses the convoy docker volume plugin by rancher. If you want to specify any driver-specific options, you can do so under the driver_opts key.

Networks
Networks are declared under the top-level networks key. By now it will come as no surprise, that network configuration in Compose files aligns closely with the docker network create command. However, there are a few new concepts to networking in Compose.

By default, Compose will automatically create a new network using the default bridge driver for an application in a Compose file. The name of the network is based on the name of the directory the Compose file is in with default appended on the end. All containers created for services in the Compose file join the default network and can be reached and discovered by the corresponding service name. This is slightly different from running containers with docker run and not specifying a network. In that case, the containers get added to the default network named bridge.

To review how you can use a bridge network, consider this example Compose file. There are two services, web, and cache. The Compose file doesn’t declare any networks so all service containers will join the default network created for the app declared in the Compose file.

In the default network, the cache container can reach the web container by using web as the hostname for the container.

Similarly, web can reach cache by resolving the cache hostname. Because web is inside the network, it uses the container port of 6379 to connect.

From the Docker host machine, cache can be reached at the host port of 36379. What about web? How can the host reach web?

It cannot reach web, because no ports are published making it accessible to the host.

Besides the default network, you can declare custom networks under the root networks key. This gives you more control and allows you to create more complex network topologies. Custom networks can be external to the application, similar to external volumes worked. You add the external: true mapping to tell Compose to verify the network already exists and join services to that external network.

This example Compose file illustrates how custom networks are used. The file declares two networks under the top-level networks key beginning at line 18. The frontend network uses the default network configuration, while the backend network refers to an external network created outside of the Compose file. The networks mapping for each service shows the proxy and app service are part of the frontend network, and the app and db service are in the backend network. With this configuration, the db and proxy services are isolated from one another. This approach of limiting communication between services to allow only what is necessary is a best practice. The db service further configures the alias database for itself on the backend network. The app container could resolve the hostname db or database as a result of the alias. The mapping syntax must be used for specifying aliases.

Special Topics: Variable Substitution
I will finish of the lesson by discussing a couple special topics in Compose files, starting with variable substitution. Variable substitution allows you to generalize your Compose files to create different environments without having to modify the Compose file. Docker Compose will substitute shell environment variables in place of variable placeholders in a Compose file. You indicate a variable by using a dollar sign before the variable name, and optionally surrounding the variable name in braces. If the variable is not defined in the environment where Docker Compose is running, an empty string is substituted for the variable. The example on the bottom half of the slide shows a snippet of a Compose file that uses a variable named REDIS_TAG for the image tag. In the shell environment, the REDIS_TAG environment variable is set to 4.0.5. When Docker Compose creates the application, the environment variable is substituted into the image value string.

Special Topics: Extension Fields
The other special topic I wanted to mention is extension fields. Extension fields let you reuse configuration blocks and move important configuration fragments to the root of the Compose file. Extension fields only work with version 3.4 or higher Compose files, so you will need to include a minor version number to get this to work in version 3. To use extension fields, add a root key that begins with x- and add the configuration to reuse under it. To insert the configuration somewhere else in the file, you use YAML anchors. Anchors allow you to create an alias for the configuration that effectively inserts the configuration fragment wherever you reference the anchor. The example on the right shows how an extension field called x-logging is used for configuring logging in two services. The anchor is indicated with an ampersand and is named default-logging. In the service definitions, the asterisk precedes the anchor name to indicate that an anchor is being used as the logging value. The extension field mapping including both options and driver are inserted at the proper indentation level where the default-logging anchor is referenced with an asterisk. The example illustrates how extension fields are useful for removing configuration clones.

Recap
This lesson began with a crash course in YAML. You learned about the string, integer, null, and Boolean data types. You also learned about the two collections in YAML: mappings, and sequences. This was just enough YAML to understand and write Compose files.

Next you understood the anatomy of a Compose file. The configuration in compose files falls under the top-level mapping keys of version, services, volumes, and networks. The configuration for services, volumes, and networks are similar to parameters you pass for docker run, volume create, and network create commands, but formatted in YAML syntax.

We finished the lesson by covering a couple special topics in Compose files. Variable substitutions allow you to generalize a Compose file using environment variables. Extension fields let you reuse configuration fragments to cut down on clones.

Up until now, you’ve only seen how to declare what’s in your multi-container applications. In the next lesson, you will see how to use the docker-compose command-line interface to run and manage the applications. When you are ready to learn how to start running your Compose file applications, continue on to the next lesson.

 

 

https://docs.docker.com/compose/compose-file