Testing Container Images

More and more software is delivered as a container image for container runtimes like Docker, Podman, and Containerd. Even simple container images require substantial logic for their creation and configuration (for example, the official nginx image). Hence, container images should be automatically tested like all other software deliverables to prevent bugs and security issues from creeping in.

In this article, I present multiple tools that help to examine container images and their configuration.

The basis forms a hypothetical container image for the webserver nginx called myimage:

FROM debian:buster

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
        tini nginx procps

ENTRYPOINT ["/usr/bin/tini", "--", "nginx"]
EXPOSE 80
STOPSIGNAL SIGQUIT
CMD ["-g", "daemon off;"]

With each tool, I want to test the following aspects of the resulting container image:

• Is nginx installed?
• Do the nginx workers run as an unprivileged user?
• Is nginx properly exposed to the host and serves a welcome page when accessed from the host?
• Does the container image only expose port 80 by default?

While these are very simple tests, they illustrate whether and how it is possible to assess the internal state of a container, the external state as it can be observed from the host, and the configuration of the container image.

The complete source code can be found in my sample repository on GitHub.

Bash Automated Testing System

Bash Automated Testing System, Bats for short, is a test framework for Bash. Being able to write the tests in Bash is the biggest advantage of Bats, but also its biggest drawback.

What is great about writing tests in Bash is that there is no additional API or framework to learn. You can write the tests using the same commands you type into your terminal. Furthermore, anything that you can do in the terminal can be tested. That is not the case with the other tools presented here that provide dedicated but often limited built-in inspections to assess the state of a system.

What is bad about writing tests in Bash is that you have to use Bash. Not because Bash were a bad choice, but because you suddenly have to deal with unexpected carriage returns returned from Docker that are also invisible¹, lots of quoting, and workarounds required by Bats to capture output. Furthermore, the lack of an API means that some of the tests cannot be portable², and you need to figure out how to use tools like ps.

Apart from that, you get what you would expect from a basic test framework, like setup functions, assertions, and test reports that your build server can process, thanks to TAP support.

The test to check whether all nginx workers run as unprivileged user shows the advantages and problems of using Bats pretty well:

@test "nginx workers do not run as root" {
    MASTER_PID=$(docker exec -i $CONTAINER_ID ps -ouser=,pid= -C nginx | awk '($1=="root"){print $2}')
    RESULT=$(docker exec -e MASTER_PID=$MASTER_PID -i $CONTAINER_ID ps -ouser= --ppid $MASTER_PID | uniq)
    assert_equal "$RESULT" "www-data"
}

Bats in a Nutshell

• All tests could be implemented: ✅
• Generates test reports: ✅
• Supported container runtimes: any with a CLI
• Pros: Bash, easy to adopt
• Cons: You need to deal with Bash’s quirks, installation requires submodules, no built-in inspections
• Complete source code

Goss

Goss is similar to Serverspec and Testinfra in the sense that it is not a test framework but a tool to inspect the state of a system. But it is still very different.

First, Goss runs on the target system. It “sees” the container only from the inside. Consequently, it is not possible to test whether nginx is properly exposed to the host or whether the image is correctly configured. You might not need that functionality, but it is something to be aware of.

Then, Goss is declarative. Instead of writing tests, you describe the desired system state in a YAML file. Goss takes that YAML file and checks whether the system matches the expectations outlined in it. Fortunately, Goss can assist you in creating the YAML file. For example, run

goss add package nginx

and Goss will add an entry like

package:
  nginx:
    installed: true
    versions:
    - 1.17.8

to your YAML file if nginx is already installed on the system. If you do not care about the exact version being installed, you can remove the versions attribute by hand.

Like many declarative tools, Goss provides some means to escape the confines of being entirely declarative. Some attributes accept regular expressions, and there is basic support for some programming constructs like loops:

file:
{{- range mkSlice "/etc/passwd" "/etc/group"}}
  {{.}}:
    exists: true
    mode: "0644"
    owner: root
    group: root
    filetype: file
{{end}}

Unfortunately, the syntax is proprietary and incompatible with goss add (as of Goss 0.3.20).

Like Testinfra and Serverspec, Goss has a decent but still limited number of inspections built-in. If something is not supported by Goss, you can fall back to the command line, like in this test that checks whether all nginx workers run as an unprivileged user:

command:
  ps -ouser= --ppid $(ps -ouser=,pid= -C nginx | awk '($1=="root"){print $2}') | uniq:
    exit-status: 0
    stdout:
    - www-data
    stderr: []
    timeout: 10000

Goss in a Nutshell

• All tests could be implemented: ❌
• Generates test reports: ✅
• Supported container runtimes: Docker, Docker Compose, Kubernetes
• Pros: No additional dependencies required, tests can be generated automatically, fast
• Cons: Can only assess the container internally, limited expressiveness due to YAML
• Complete source code

Serverspec

Serverspec is an extension for RSpec, a BDD framework for Ruby. That means that you write your tests in Ruby as RSpec specifications.

Serverspec provides many built-in inspections, called resource types, that simplify writing tests and give some portability. As a result, checking whether nginx is installed is as simple as

describe package('nginx') do
  it { should be_installed }
end

regardless of the Linux distribution you are using. If the built-in inspections do not cut it, you can still fall back to the command line as I had to check whether all nginx workers run as an unprivileged user:

describe command('ps -ouser= --ppid $(ps -ouser=,pid= -C nginx | awk \'($1=="root"){print $2}\') | uniq') do
  its(:stdout) { should eq("www-data\n") }
end

Thanks to Serverspec being a Rspec extension, you get all the features you can expect from a mature testing framework.

The development of Serverspec has been slow in the past years, and the author only accepts pull requests, but no issues.

Serverspec in a Nutshell

• All tests could be implemented: ✅
• Generates test reports: ✅
• Supported container runtimes: Docker
• Pros: Many built-in inspections, based on a mature test framework
• Cons: none
• Complete source code

Testcontainers

Testcontainers is actually a tool that helps you write integration tests for your applications. But you can still misuse it for testing container images. In that regard, it is most similar to Bats. But instead of writing your tests in Bash, you can use Java.

Testcontainers excels at simplifying the interaction with Docker. Starting, stopping, and inspecting containers is a breeze. On the other hand, it does not provide any built-in inspections, so you have to write them yourself. For example, the test to check whether all nginx workers run as an unprivileged user looks like that:

@Test
void testNginxWorkersDoNotRunAsRoot() throws IOException, InterruptedException {
	var masterPidCmd = container.execInContainer("/bin/bash", "-c",
			"ps -ouser=,pid= -C nginx | awk '($1==\"root\"){print $2}'");
	var masterPid = Integer.parseInt(masterPidCmd.getStdout().trim());
	var workersUserCmd = container.execInContainer("/bin/bash", "-c",
			String.format("ps -ouser= --ppid %d | uniq", masterPid));

	assertThat(workersUserCmd.getStdout().trim()).isEqualTo("www-data");
}

As we are using Java, we are free to choose between any of Java’s build tools, testing frameworks, and assertion libraries. I picked JUnit 5 and AssertJ.

Testcontainers probably makes the most sense if the container tests are part of a larger Java project because you are probably using some of the tools already and can leverage Gradle or Maven to drive the build. But thanks to JBang, even a standalone usage is possible, as can be seen in the sample source code.

Testcontainers in a Nutshell

• All tests could be implemented: ✅
• Generates test reports: ✅
• Supported container runtimes: Docker
• Pros: Simplifies interaction with Docker
• Cons: no built-in inspections
• Complete source code

Testinfra

Testinfra is a plug-in for pytest, a testing framework for Python. As a result, you write your tests in Python.

Testinfra includes many built-in inspections, called modules, that simplify writing tests and provide some portability. As a result, checking whether nginx is installed is as simple as

def test_nginx_is_installed(host):
    nginx = host.package("nginx")
    assert nginx.is_installed

regardless of the Linux distribution you are using. If any of the built-in inspections is not sufficient, you can fall back to the command line. But I did not have to do this a single time. Consequently, testing whether all nginx workers run as an unprivileged user is surprisingly readable:

def test_nginx_workers_do_not_run_as_root(host):
    master = host.process.get(user="root", comm="nginx")
    workers = host.process.filter(ppid=master.pid)
    users = set([worker.ruser for worker in workers])
    assert set(['www-data']) == users

Another advantage of Testinfra is that it supports more container runtimes than any other tool listed here.

Thanks to Testinfra being a pytest plug-in, you get all the features you can expect from a mature testing framework.

Testinfra in a Nutshell

• All tests could be implemented: ✅
• Generates test reports: ✅
• Supported container runtimes: Docker, Kubernetes, LXC, OpenShift, Podman
• Pros: Large number of built-in inspections, based on mature test framework
• Cons: none
• Complete source code