Hello and welcome to this lecture where I shall be looking at some of the points that you need to be aware of when it comes to container security.

Typically existing security controls and processes that you might already be running for your infrastructure and applications, may not be able to integrate with container development and at the scale of which your container infrastructure can quickly grow.  You need to have a definitive set of policies for your container architecture.  Let me run through some of the elements of container security you should focus on.

Build a Secure Base Image.

Your containers start with your images, and typically your base images.  These are used to compile your containers and variants of containers based off of the image.  So from the very start, implementing security when building this base image will help your drive security into the origins of your container pipeline.   When you have this base image secure, building off of and developing from this base image adds reassurance from a security perspective across your environment.  Failure to have a secure base image will inevitably feed weaknesses and vulnerabilities throughout your container architecture more and more as and when it’s used. 

So how do you build a secure base image? You should strive to use trusted and recognized sources to build your image.  Pulling code and libraries from unknown and untrusted sources are a sure way to introduce vulnerabilities and risks.  You should look to maintain a level of authenticity within your image where it can be safely regarded as a secure and trusted baseline. 

There are public repositories where you can download an image to use as your base image, and these should be used with caution as you do not know where the components of these images were sourced from.  The image itself is created from layers upon layers of files, and it’s important to know the origins of these files and components.

The rate of which applications, technology, operating systems are updated, you need to revisit your base images regularly to ensure they are also updated with the latest OS patches and bug fixes.  If a vulnerability is found within OS files or library, you must also ensure that your base image is updated accordingly.  An out-of-date base image is a security threat to your container architecture.

Recording and recognizing vulnerabilities is a good practice to implement for all of your images within your repositories, your development teams should be able to recognize and understand the impact it will have if a particular image is used within your environments, even for a very short time if it has vulnerabilities associated with it.  For each vulnerability detected you should define the impact and severity of the vulnerability.

In the larger picture, some key image-related security considerations are:

• Is every image used in the CI/CD pipeline scanned?

• Do you scan all image registries to ensure that images that either skipped the CI/CD process or images that have gone stale are secure?

• How can you prevent users from running images from outside the pipeline?

• Does the image include sensitive data? Embedded secrets (e.g. API key, SSH key, password, etc.)?

• Does the image include only executables required for its ongoing operation?

• Are there any vulnerabilities at the base image itself? Are there any vulnerabilities in added open-source components?

• Is the image configured with root/admin account by default?

• Can you stop images from being used based on CVEs severity and risk score?

• Can you easily search for a specific vulnerability or a software component in a large number of running containers?

Hard-coded, unmanaged secrets in images.

Managing secrets such as passwords, API keys, and security tokens is particularly challenging in container environments due to the dynamic and ephemeral nature of containers.

Storing secrets inside a container image is unfortunately quite common. It’s an easy way for developers to run tests, and they often forget to remove those secrets before moving to production. This risk exposing secrets to anyone with access to that container, to the registry, CI/CD pipeline, and potential intruders. In addition, if secrets are embedded into the image, it means that the image lifecycle would be coupled with the secrets’ lifecycle (e.g. rotation/revocation). In this case, if you want to rotate a secret, you’ll need to build a new image.

Moreover, providing secrets as an environment variable when running a container is also poor security practice. This is because it’s particularly easy for secrets to be accidentally leaked

through logging, since it is common practice for software to log its entire environment. The number of employees who have access to logs is often much greater than the employees who

should and can access production secrets.

Therefore, it’s recommended to mount secrets as tmpfs volumes, where they’re accessible to the application as a virtual “file” resident in memory. This secrets delivery method to containerized applications is used by most orchestrators.

On a broader scale, some key secrets-related security considerations are:

• Where are secrets stored today? Hard-coded into images?

• Can you map secrets to relevant containers?

• Can you ensure that only certified, designated running containers can retrieve secrets?

• Can you ensure that secrets are delivered and rotated based on your organization’s security policies?

• Can routine rotation be done with no downtime to the running container? No restarts?

• If a secret is rotated/revoked, do you have an automated way to propagate the update or revocation to all relevant containers that currently use the secret?

• Can you ensure that secrets do not persist on the host once the container is spun down?

• Once a secret is revoked, can you ensure it’s deleted from volumes and that access permissions to resources are also revoked?

• Can secrets activity be logged (e.g. secrets delivery, rotation/revocation)?

• Is there a roll-back procedure?

Access Management

Access management can often be one of the weakest points of security within any architecture, and that also applies to Container security as well.  You must devise a clear process and implement procedures to help you control how your containers are managed and who can access them to perform which functions against them.  For example, who can upload images to your own private registry, or pull images from them, or update a base image.  Much of this will depend on the role of the user and what their job entails, as a result, implementing role-based access control (RBAC) is a great method for managing access.  By assigning permissions based on roles you can implement a defined access structure with clear boundaries as opposed to permissions given by root.     

If you have a loose access control method whereby users could implement actions using root credentials then this makes it very difficult to determine who actually carried out the request from an auditing perspective.  Not forgetting, if someone was to compromise these credentials then they would have access to carry out privileged actions against your containers.

You should also look at the file level permissions on your host, for example by ensuring that critical directories containing important information, such as ‘etc/docker/ is only writable by root.

Key access-related security considerations are:

• Can you assign and enforce user access permissions to container resources?

• Can you assign Docker sub-commands to users on a specific host?

• Can you assign Kubernetes master node operations to users by cluster, deployment, node?

• How do you alert on unauthorized user activity on a host?

• How do you intend to ensure role-based access control to container runtime environment?

Hardening your Host

It is best practice to harden every host, but more importantly so for when running infrastructure on top of your hosts such as virtual machines (VMs) or Container infrastructure.  By hardening your host it helps to reduce the attack surface in addition to reducing the risk put upon your container runtime.  

By restricting and removing services and functions from your host through the hardening process, you are able to direct requests from your containers through the container daemon more effectively.

To help you pinpoint how to harden your host, the Center for Internet Security, known as CIS has some benchmarks on how to do this when working with Docker or Kubernetes, which can be found here: https://www.cisecurity.org/cis-benchmarks/  These benchmarks are continuously refined and used as global standards for best practices for protecting data and IT resources.  As a result, it is best practice to review and work along these benchmarks to harden your container hosts. When adopting a hardening process like this, you need to ensure that only hardened hosts are used for your container infrastructure.

Key host-related security considerations are:

• Can you ensure that the OS, as well as the container engine versions, are up to date and that they are fully patched?

• Can you ensure image packages and libraries are authorized (e.g. patched and up-to-date)?

• Can you ensure that your Kubernetes environment is properly secured? That access control, networking, and authentication are all in check?

Lack of real-time visibility and control of the container runtime environment

It’s a relatively trivial task to generate a list of running containers in production. It is, however, difficult to extract any meaningful security or operational activity data from this list of containers, or correlate the (short-lived) activity of containers running across multiple hosts. The reason for this is the way the host environment is built, whereby containers create a layer of obscurity that reduces legacy network-based and host-based security solution visibility.

The host environment is architected so that the shared OS runs the container engine and the engine runs the containers themselves. The OS is ‘aware’ of the container engine but is not ‘aware’ of which containers are running. The container engine on the other hand, ‘knows’ which containers are running but is not aware of container activity. And the containers themselves, conversely, are not aware of each other but access OS resources directly. So, if you’re running application firewalls or host-based intrusion prevention systems (HIPS) to monitor the OS, you’d be lacking both the visibility into container activity and the ability to monitor and control the same host containers’ traffic.

Tracking container activity is mandatory for real-time attack detection and disruption. There are numerous ways in which containers can slip through the cracks through misconfiguration – or explicit actions by attackers through namespace manipulations. By conducting ongoing container behavioral analysis, controlling network traffic, and creating container whitelisting policy, organizations can automatically detect unauthorized activity and block these attacks.

Key security considerations are:

• How do you plan to protect your runtime application against attacks that are not based on known vulnerabilities, such as zero-days, attacks based on host configuration errors, privileged user error or insider threat?

• Do you have the means today to detect an attack in the container’s environment?

• Is every image deployed into production authorized? Can you ensure the latest, authorized image is being instantiated? Do you know if any rogue containers are running?

• Can you ensure that the container image was not changed unintentionally or maliciously altered once on the host?

• Can you ensure that containerized apps are granted the minimum necessary permissions and use secrets securely?

• Do you know if the app is the correct patched version?

• Do you have visibility of what’s running inside a container? What executables are used for its intended purpose?

• Can you detect and alert on unauthorized network traffic?

• Can you block unauthorized inbound/outbound communication from/to containers? For example, can you block outbound connection to a database container?

• Can you block container (e.g. PCI-related) access to a specific IP address?

• Can you stop a container’s unauthorized processes with no downtime to the running container itself?

• Can you perform instant impact analysis and list all containers that have specific vulnerability?

• When considering a shift in app stacks, can you look for the impact or the use of specific components across all running containers?

• Can you immediately locate all containers that have a certain package? e.g. python?

Containers Complicate Compliance

In the event of a breach, demonstrating compliance can be a useful means of avoiding and/or reducing fines. Demonstrating compliance is best defined, for example, in the container

environment as blocking vulnerable images from being used. In parallel, these images are withheld and kept as compliance evidence, serving to prove that the remediation process was

performed. Compliance evidence can also contain additional logged events, including replaced high-risk containers, blocked image executables, blocked user access, etc.

Key compliance-related evidence:

• Can you measure the remediation trends based on KPIs (e.g. ‘open’ vs. ‘remediated’)?

• Do you log user and container activity?

• Do you alert on any behavior deviation in container activity?

• Do you keep a history of security configuration changes?

• Do you keep secrets rotation history?

That has now brought me to the end of this lecture. Coming up in the next lecture I shall be looking at the importance of implementing container security at scale.