aahlenst.dev

Autoscaling Forgejo Runner

Sun, 12 Apr 2026 12:00:00 +0200

Over the last months, the Forgejo community equipped Forgejo and Forgejo Runner with the necessary building blocks for autoscaling Forgejo Runner. These building blocks should enable developers of workload managers and autoscalers to build reliable and secure integrations for Forgejo Actions. This post aims to provide an overview of the new capabilities, explain how they fit together, and highlight some peculiarities to watch out for.

Forgejo 15.0 and Forgejo Runner 12.8 or newer are required unless noted otherwise.

Autoscaling in a Nutshell

To scale Forgejo Runner instances up and down, an external system has to perform the following tasks:

Monitor Forgejo’s job queue for jobs that are ready to run.
Create runners in response to jobs that are ready to run.
Launch Forgejo Runner to run a job.
Remove runners that are no longer needed.

Start by asking Forgejo for jobs that are ready to run:

$ curl http://localhost:3000/api/v1/repos/andreas/test/actions/runners/jobs
[{"id":982,"attempt":1,"handle":"33ba7d51-59c6-44f8-9d2b-1b94f4033973","repo_id":1,"owner_id":1,"name":"build","needs":null,"runs_on":["debian"],"task_id":0,"status":"waiting"}]

Then, create a runner to run that job:

$ curl -X POST \
	-H "Content-Type: application/json" \
	-d '{"name":"0CEY3JWJ6A4ZK","ephemeral":true}' \
	http://localhost:3000/api/v1/repos/andreas/test/actions/runners
{"id":5,"uuid":"d4637a60-7fea-4075-a81c-7e4ae01bdf0d","token":"e1f3d26e034687a2b6856c4c6c3d87a469ff3bea"}

Run the job:

$ echo -n "e1f3d26e034687a2b6856c4c6c3d87a469ff3bea" > /tmp/runner-token
$ forgejo-runner one-job \
	--url http://localhost:3000/ \
	--uuid d4637a60-7fea-4075-a81c-7e4ae01bdf0d \
	--token-url file:/tmp/runner-token \
	--label debian:docker://node:lts \
	--handle 33ba7d51-59c6-44f8-9d2b-1b94f4033973 \
	--wait
INFO[0000] No configuration file specified; using default settings. 
INFO[2026-04-11T20:18:39+02:00] Starting job
INFO[2026-04-11T20:18:39+02:00] runner: 0CEY3JWJ6A4ZK, with version: v12.8.2, with labels: [debian], ephemeral: true, declared successfully 
INFO[2026-04-11T20:18:39+02:00] single task poller launched
INFO[2026-04-11T20:18:39+02:00] single task poller successfully fetched one task from http://localhost:3000/ 
INFO[2026-04-11T20:18:39+02:00] task 917 repo is andreas/test https://data.forgejo.org http://localhost:3000/ 
INFO[2026-04-11T20:18:47+02:00] Cleaning up network for job build, and network name is: WORKFLOW-891112e47856a708ddac3fdd3d034bc0 
INFO[2026-04-11T20:18:47+02:00] single task poller is shutting down

And that’s it! Because I created an ephemeral runner, Forgejo will remove the runner automatically after the job has completed.

About Scopes

Forgejo Actions has four scopes: repository, user, organization, and instance. For example, a single runner can be bound to a repository, a user, an organization, or the entire instance. While a runner that is bound to a single repository can only run jobs triggered by that repository, a runner that is bound to an entire organization can run jobs triggered by any repository belonging to that organization. Variables and secrets work similarly.

When managing runners using the web UI, you can manage the runners of the current scope. In addition, all runners of superior scopes are visible. The instance scope works slightly different. It shows all runners because it doubles as admin scope.

The HTTP API uses the same scopes. That means a single endpoint like actions/runners/jobs is available in each scope:

Admin: /admin/actions/runners/jobs
Organization: /orgs/{org}/actions/runners/jobs
Repository: /repos/{owner}/{repo}/actions/runners/jobs
User: /user/actions/runners/jobs

By default, the HTTP API provides the same data as the web UI. However, some endpoints like actions/runners can only return the data of the current scope if desired.

The examples in this post usually use a single scope for brevity.

Monitoring Forgejo’s Job Queue

You can monitor Forgejo’s job queue either using the HTTP API or, once the PR has landed, webhooks. This section will use the HTTP API exclusively. However, all concepts discussed here also apply to webhooks.

A list of all active¹ jobs can be obtained by querying the endpoint actions/runners/jobs which is exposed in each scope.

Each scope “sees” all active jobs of all subordinate scopes. For example, /user/actions/runners/jobs returns all active jobs of each of the user’s repositories. Similarly, /orgs/{org}/actions/runners/jobs returns all active jobs of each of the organization’s repositories.

To get all active jobs that belong to the repository andreas/test, run:

$ curl http://localhost:3000/api/v1/repos/andreas/test/actions/runners/jobs

The result:

[
	{
		"id": 301,
		"attempt": 1,
		"handle": "9d52c7d8-aebe-426b-b015-dd453aacaada",
		"repo_id": 1,
		"owner_id": 1,
		"name": "test",
		"needs": null,
		"runs_on": [
			"debian"
		],
		"task_id": 0,
		"status": "waiting"
	}
]

When filtering by label the labels in the query string have to include all labels that appear in runs_on. Otherwise, Forgejo will filter the job out.

Each job can be run multiple times. Forgejo increments the attempt number before each attempt, but the id remains fixed.

If you need to uniquely identify a run attempt, use handle². Do not create an identifier yourself by piecing together multiple fields.

Runner Management

Once a job is ready to run, a new runner has to be created to run the job. That again can be done with the help of Forgejo’s HTTP API.

Forgejo offers the following endpoints for managing runners:

List all runners: GET /repos/{owner}/{repo}/actions/runners
Register a runner: POST /repos/{owner}/{repo}/actions/runners
Get a particular runner: /repos/{owner}/{repo}/actions/runners/{runner_id}
Delete a particular runner: DELETE /repos/{owner}/{repo}/actions/runners/{runner_id}

The same set of endpoints is available in all other scopes.

By default, actions/runners returns all runners that are visible in a particular scope. For example, in the repository scope, all runners of the repository, the user, or organization it belongs to, and the instance will be returned. Append visible=false to the query string to only receive the runners of the respective scope.

To create a new runner, send a POST request to /repos/{owner}/{repo}/actions/runners or one of its equivalents in the other scopes:

$ curl -X POST \
	-H "Content-Type: application/json" \
	-d '{"name":"0CEY3JWJ6A4ZK","ephemeral":true}' \
	http://localhost:3000/api/v1/repos/andreas/test/actions/runners
{"id":5,"uuid":"d4637a60-7fea-4075-a81c-7e4ae01bdf0d","token":"e1f3d26e034687a2b6856c4c6c3d87a469ff3bea"}

You receive the runner’s ID, which is required to use one of the runner-specific endpoints, the runner’s UUID, and its token. You have to supply the UUID and the runner token to forgejo-runner so that it can authenticate itself to Forgejo.

Persistent and Ephemeral Runners

Forgejo distinguishes two types of runners: ephemeral and persistent (the default). Persistent runners can run multiple jobs during their lifetime, whereas ephemeral runners can run at most one job before they get deleted by Forgejo.

If possible, use ephemeral runners in autoscaling scenarios, especially when using Forgejo Runner in host mode. In host mode, the job can get hold of the runner token. With an exfiltrated persistent runner token, an attacker could request additional jobs and, depending on the runner’s scope, access secrets and other sensitive resources of other repositories. With an ephemeral runner token, an attacker can only receive the same job that was already compromised. That means that the damage would be limited to a single repository.

Running Jobs

Forgejo Runner offers two separate commands for running jobs: one-job, and daemon. While daemon runs jobs until it is shut down, one-job does what it says on the tin: it runs exactly one job before it quits.

one-job is purpose-built for autoscaling and should be used whenever possible. It is also the only command that supports ephemeral mode; daemon refuses to work in ephemeral mode.

A typical usage of forgejo-runner one-job looks as follows:

$ forgejo-runner one-job \
	--url http://localhost:3000/ \
	--uuid d4637a60-7fea-4075-a81c-7e4ae01bdf0d \
	--token-url file:/path/to/runner-token \
	--label debian:docker://node:lts \
	--label gpu:docker://node:lts \
	--handle 33ba7d51-59c6-44f8-9d2b-1b94f4033973 \
	--wait

--url, --uuid, and --token-url define the connection to Forgejo, whereas --label defines the labels this runner instance supports. For --handle and --wait, see the next sections.

While it might be handy to use options in many use cases, it is not required. All options except --handle and --wait can also be configured in the runner configuration:

server:
  connections:
    forgejo:
      url: http://localhost:3000/
      uuid: d4637a60-7fea-4075-a81c-7e4ae01bdf0d
      token: e1f3d26e034687a2b6856c4c6c3d87a469ff3bea
      labels:
        - debian:docker://node:lts
        - gpu:docker://node:lts

Together with the runner configuration stored in runner-config.yaml, the invocation above becomes:

$ forgejo-runner -c runner-config.yaml one-job \
	--handle 33ba7d51-59c6-44f8-9d2b-1b94f4033973 \
	--wait

What the Handle is Good For

--handle instructs Forgejo to return the run attempt identified by the handle – and only that run attempt. If the run attempt is not ready to run or already being executed by another runner, Forgejo will not return any other job, even if there are others waiting.

Its purpose is to prevent race conditions, and to simplify the implementation of autoscalers.

As an example for a race condition, consider two waiting jobs: A with the label set [debian], and B with the label set [debian, gpu]. In response to those waiting jobs, two Forgejo Runner instances are started: 1 with the label set [debian] (for A), and 2 with the label set [debian, gpu] (for B). If, for some reason, 2 asks Forgejo first for the next pending job, it will receive A because the label set of 1 is a superset of the label set of A. Once 1 is ready and asks for a waiting job, it cannot run B because its label set does not contain gpu.

If both Forgejo Runner instances were to use --handle, they would both receive the intended job. Crisis averted!

There are more potential concurrency issues that can only be avoided if the workload manager or autoscaler knows which runner is executing a particular job. Without --handle, that would be much harder and require additional APIs in Forgejo.

For --handle to do its magic, all runners operating in a particular scope have to participate and use --handle, too. Otherwise, jobs might still end up on the wrong runner.

Waiting Is Unavoidable (for Now)

--wait causes forgejo-runner one-job to ask Forgejo for a job until it receives one. Unfortunately, --wait must be used for now because not all jobs with the status waiting can actually be run. See the section about Unambiguous Job Statuses for details.

A timeout was omitted from one-job because workload managers and autoscalers usually enforce timeouts by themselves.

Missing Pieces

While the building blocks I have discussed above are a good start, there are still multiple pieces missing.

Webhooks

While polling Forgejo’s HTTP API for jobs that are ready to run is the most reliable mechanism to monitor its job queue, it is not particularly efficient. Operators of larger Forgejo instances like Codeberg would certainly be happy if you would not poll it every few seconds. A well established, more efficient mechanism would be webhooks.

Fortunately, there is a pull request for adding webhooks that are activated by workflow run and job status changes that is close to the finish line.

Runner Removal

If you delete a runner in Forgejo, it will only be marked as deleted, not actually removed from the database. It would make sense if it were used for undo, but it is not. The deleted runners linger forever in Forgejo’s database for no particular reason.

Expect that this soft-deletion will be removed as part of the gradual rollout of foreign keys.

Unambiguous Job Statuses

As of Forgejo 15, it is impossible to determine from the outside whether a job is ready to run. The problem is that a job that is currently waiting might still be blocked by a concurrency group.

There is a proposal to introduce a new status called ready which would signal that a job is ready to run, no exceptions. It would be augmented by a secondary status that would provide additional details.

An Access Token Scope for Managing Runners, Only

In Forgejo, access token scopes are relatively broad. For example, you can select whether a token can only read a repository or have full control over it. There is nothing in between. That means that even if a token is only used to manage runners, it has full administrative privileges over the respective scope. That is unnecessarily risky.

There are ideas swirling around about introducing finer grained scopes, for example, write:repository:runners, but nothing definitive yet.

Organization Tokens

Forgejo only has personal access tokens, but no organization tokens. That, paired with the lack of a token scope for managing runners, means that managing runners using the HTTP API is inconvenient and unnecessarily risky.

Monitoring

There is currently no mechanism to monitor any aspects of Forgejo Actions, like the length of the job queue. There is an open feature request for adding a metrics endpoint. Please tell us what you need.

Rejected Ideas

JIT Configuration

When creating a self-hosted runner, GitHub provides the option to generate a just-in-time configuration, or JIT configuration in short. The JIT configuration is Base64-encoded JSON document that contains the runner configuration and can be passed to GitHub Actions Runner ./run.sh with the option --jitconfig. That avoids the separate configuration step with ./config.sh.

We discussed the idea to create a similar mechanism for Forgejo Runner, but ultimately abandoned it because it would tie Forgejo Runner to Forgejo even more. Furthermore, Forgejo has no knowledge of Forgejo Runner’s label configuration (think debian-latest:docker://node:lts-trixie) which is significantly more involved that GitHub’s.

Kubernetes-native Runners

Forgejo Runner usually runs each job in a separate container that it spawns. Optionally, it creates additional containers to provide services like a PostgreSQL database. That does not work well with Kubernetes, because it requires nesting containers inside containers. Couldn’t Forgejo Runner instead delegate the creation and management of containers to Kubernetes?

While that might indeed be possible, it is unlikely that the capability will be incorporated into Forgejo Runner itself. Forgejo Runner already requires fairly exotic knowledge to work on, and deep knowledge of Kubernetes would narrow the pool of potential maintainers even more. Instead, we are currently investigating the addition of a plug-in interface for back-ends. That would not only enable Kubernetes-native runners, but also using lightweight virtual machines instead of containers for running jobs.

Prototypes of a Kubernetes-native runner and a back-end based on Firecracker already exist and can be tried out.

Recipes

Mixing Persistent and Ephemeral Runners

It is possible to use persistent and ephemeral runners in a single scope without undermining --handle and causing concurrency issues. The key is to use disjoint label sets for ephemeral and persistent runners. For example, debian could activate a persistent runner whereas debian-ephemeral could request an ephemeral runner. It requires that the workload manager or autoscaler ignores debian.

You could also use multiple labels, for example, [debian, ephemeral] (for an ephemeral runner) and [debian] (for a persistent runner). While that might look attractive from the onset, it is tricky to use correctly. The first label that Forgejo Runner encounters defines the execution environment that the job will be executed in. That means that [debian, ephemeral] will be executed in debian, whereas [ephemeral, debian] will be executed in ephemeral. Teaching the workload manager or autoscaler to pass all labels to Forgejo Runner with identical mappings will make it work in any case:

$ forgejo-runner one-job \
	--label debian:docker://node:lts \
	--label ephemeral:docker://node:lts \
	...

Pre-Warmed Runners

Pre-warmed runners are typically virtual machines that are waiting for a job to run. That is supposed to reduce latency. The problem is that without a job to run, there is no handle. The best you can do in this case is to start the virtual machine, but not Forgejo Runner. Only start Forgejo Runner once a job is ready to run.

If that is not an option, see Mixing Persistent and Ephemeral Runners.

Active jobs are those that are either ready to run (unless blocked by a concurrency group) or currently running. ↩︎
It might look like an UUIDv4, but it is not guaranteed to be an UUIDv4. ↩︎

systemd Credentials in Spring Boot

Thu, 10 Jul 2025 12:00:00 +0100

systemd, Linux’s most widely used system and service manager, introduced system and service credentials with version 247. They allow for securely passing passwords and keys to applications, thereby solving a longstanding problem. Let’s find out how we can access them in Spring Boot.

Introducing systemd Credentials

In their simplest form, systemd credentials are straightforward:

[Service]
ExecStart=/path/to/my-program
LoadCredential=my-secret:/etc/my-program/credentials/some-password

In this unit file, we tell systemd to start my-program and to expose an unencrypted credential to it. The credential is stored in the file /etc/my-program/credentials/some-password.

Now, how can my-program access it? Not by opening /etc/my-program/credentials/some-password. Instead, systemd will prepare a special directory¹. There, it will place the credential in a file named my-secret. systemd passes my-program the path to the special directory using the environment variable CREDENTIALS_DIRECTORY. If my-program were a shell script, it could read the secret by opening "$CREDENTIALS_DIRECTORY/my-secret".

There are many more options. It is not only possible to encrypt secrets, but even to bind them to the Trusted Platform Module of a computer. For details, please consult systemd’s documentation.

Now that we know how they work, the only thing left to do is to figure out how we can get Spring Boot to read systemd credentials from files in CREDENTIALS_DIRECTORY.

Externalized Configuration in Spring Boot

In Spring Boot, configuration is declared as properties like spring.datasource.url. Those properties are not only used to configure Spring Boot itself. You can define your own properties to make your application configurable through the very same system. Spring Boot can obtain configuration values from a multitude of sources, for example, .properties, the environment, JNDI or Kubernetes ConfigMaps.

In addition to properties, Spring Boot has dedicated SSL bundles to load keys and certificates from files.

Populating SSL Bundles From systemd Credentials

I start with the simplest case: populating SSL bundles from systemd credentials.

Without systemd credentials, the configuration to define an SSL bundle called mybundle to be used by the embedded web server looks as follows:

spring.ssl.bundle.pem.mybundle.keystore.certificate=file:/path/to/domain.crt
spring.ssl.bundle.pem.mybundle.keystore.private-key=file:/path/to/domain.key

To load the certificate and private key from a systemd credential, specify them in the unit file:

[Service]
ExecStart=/path/to/java -jar /path/to/demo-0.0.1-SNAPSHOT.jar 
LoadCredential=domain.key:/path/to/domain.key
LoadCredential=domain.crt:/path/to/domain.crt

ExecStart= defines the executable JAR to launch. The next two lines starting with LoadCredential= specify the private key (/path/to/domain.key) and certificate (/path/to/domain.crt) that will be made available to the application as domain.key and domain.crt respectively. The only step remaining is to tell Spring Boot to read both files from CREDENTIALS_DIRECTORY:

spring.ssl.bundle.pem.mybundle.keystore.certificate=file:${CREDENTIALS_DIRECTORY}/domain.crt
spring.ssl.bundle.pem.mybundle.keystore.private-key=file:${CREDENTIALS_DIRECTORY}/domain.key

Using systemd Credentials in Any Configuration Property

Now, most Spring Boot applications will probably want to access a database. And that database will most likely require a password, which can be configured with spring.datasource.password. Unfortunately, we cannot leverage our newly learnt knowledge and use the file: prefix to load the database password from a file because file: is only understood by SSL bundles. Instead, we have to repurpose a configuration tree.

A configuration tree turns files into Spring Boot properties by converting their file name into the property’s name and the file’s content into the property’s value:

[Service]
ExecStart=/path/to/java \
    -jar /path/to/demo-0.0.1-SNAPSHOT.jar \
    --spring.config.import=configtree:${CREDENTIALS_DIRECTORY}
LoadCredential=spring.datasource.password:/path/to/database-password

--spring.config.import is the magical argument that makes it all work. If you prefer, you can put spring.config.import=configtree:${CREDENTIALS_DIRECTORY} into your application.properties, instead.

If you find it too cumbersome to specify every secret individually, you can treat your entire configuration as a systemd credential and load it from CREDENTIALS_DIRECTORY:

[Service]
ExecStart=/path/to/java \
    -jar /path/to/demo-0.0.1-SNAPSHOT.jar \
    --spring.config.import=${CREDENTIALS_DIRECTORY}/application.properties
LoadCredential=application.properties:/path/to/application.properties

Credentials Do Not Belong in Environment Variables

While environment variables are handy for configuring an application, they have many drawbacks. Subprocesses inherit them, they are awkward to use with binary data, have size limitations, and can be accessed easily with various tools. ps auxe lists them and systemctl show <service> allows any user on a system to print them if they are defined in a unit file.

To make it only accessible to my-program enable mount namespacing with PrivateMounts=. ↩︎

Finding EFI Firmware That Is Compatible

Sun, 07 Jul 2024 12:00:00 +0200

I run some QEMU/KVM virtual machines as part of one of my GitHub Actions workflows. To reduce the workflow’s runtime and to help make it reproducible, it uses pre-built disk images and pre-defined XML domain configurations for libvirt. That worked well until I changed the workflow to use the new Ubuntu 24.04 images. Afterwards, libvirt rejected the domain configuration with an error I had not encountered before:

libvirt.libvirtError: operation failed: Unable to find ’efi’ firmware that is compatible with the current configuration

Why does a virtual machine image suddenly stop working just because I upgraded Ubuntu? I was stumped, particularly because a quick web search did not turn up much useful information.

The offending domain configuration looked as follows:

<os firmware='efi'>
  <type arch='x86_64' machine='q35'>hvm</type>
  <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
  <nvram template='/usr/share/OVMF/OVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/mymachine_VARS.fd</nvram>
  <boot dev='hd'/>
</os>

<loader/> describes the UEFI firmware to use. <nvram/> defines the path to the firmware’s variable store. It is machine-specific and contains the path to the boot loader, for example. See libvirt’s Domain XML format documentation for further information.

Thankfully, the mailing list message accompanying the patch that introduced the error message into libvirt explains what the problem is:

The [old] message can be misleading, because it seems to suggest that no firmware of the requested type is available on the system.

What actually happens most of the time, however, is that despite having multiple firmwares of the right type to choose from, none of them is suitable because of lacking some specific feature or being incompatible with some setting that the user has explicitly enabled.

In Stephen Finucane’s helpful article UEFI Support in Libvirt, I learnt that we can ask libvirt what firmware it knows about by running:

$ virsh domcapabilities --machine q35 | xmllint --xpath '/domainCapabilities/os' -

And sure enough, /usr/share/OVMF/OVMF_CODE.fd is no longer there because it has been removed¹ from the ovmf package shipped with Ubuntu 24.04:

<os supported="yes">
  <enum name="firmware">
    <value>efi</value>
  </enum>
  <loader supported="yes">
    <value>/usr/share/OVMF/OVMF_CODE_4M.ms.fd</value>
    <value>/usr/share/OVMF/OVMF_CODE_4M.secboot.fd</value>
    <value>/usr/share/OVMF/OVMF_CODE_4M.fd</value>
    <enum name="type">
      <value>rom</value>
      <value>pflash</value>
    </enum>
    <enum name="readonly">
      <value>yes</value>
      <value>no</value>
    </enum>
    <enum name="secure">
      <value>yes</value>
      <value>no</value>
    </enum>
  </loader>
</os>

Solving the Problem of the Missing Firmware

We have multiple options for dealing with the missing firmware, and none is fun. The reason is that the firmware (OVMF_CODE.fd) and its variable store (OVMF_VARS.fd) are inextricably linked. It is impossible to change the firmware only to OVMF_CODE_4M.fd and continue using the old variable store. The virtual machine simply will not boot.

Recreating the Virtual Machine Images

As I use Packer, re-creating the virtual machine with a different firmware image is a matter of minutes. While older versions of Debian and Ubuntu already come with OVMF_CODE_4M.fd, Fedora also switched from a raw to a qcow2 file (OVMF_CODE_4M.qcow2), making sharing the same virtual machine image across distributions impossible. If that is important to you, you might fare better with Adding OVMF_CODE.fd Back In.

Migrating to OVMF_CODE_4M.fd

If it is difficult to re-create the virtual machine, there is an option to migrate to OVMF_CODE_4M.fd. Debian 12 and Ubuntu 24.04 include the program 2M_VARS-to-4M_VARS.sh (direct link) in the ovmf package. According to the HOWTO, converting the old variable store to the 4M variant becomes a matter of running:

$ 2M_VARS-to-4M_VARS.sh -i mymachine_VARS.fd

Adding OVMF_CODE.fd Back In

OVMF is portable and can be copied between systems. Unfortunately, it is insufficient to grab OVMF_CODE.fd and OVMF_VARS.fd from a system that still has them and put them into /usr/share/OVMF. If you do this and run virsh domcapabilities, they will not show up, and libvirt will not accept the domain configuration despite the firmware image and variable store being in the right place.

You also need a matching firmware descriptor in /usr/share/qemu/firmware. These descriptors not only record the known firmware images but also the supported machine types and features like Secure Boot. It is usually sufficient to grab the JSON file that refers to OVMF_CODE.fd and copy it over. The firmware should then appear in virsh domcapabilities.

Resetting the Variable Store

That is the approach for adventurous people. Change the firmware to OVMF_CODE_4M.fd and reset the variable store when starting the domain for the next time:

$ virsh start --reset-nvram domain

Then, use the UEFI shell to manually select the boot loader, start the system and re-install grub-efi. If the guest is a Windows machine, it should boot right away without you ever seeing the UEFI shell (see below for why this works).

Solving the Problem of the Incompatible Firmware

Not only can the firmware be absent. It can also be incompatible with the domain configuration, resulting in the same error message. Usually, that happens when creating a domain for the first time.

The key to finding the problem is to look at the domain configuration and compare it with the output of virsh domcapabilities.

Let’s suppose the domain configuration looks as follows:

<os firmware='efi'>
  <type arch='x86_64' machine='pc'>hvm</type>
  <loader secure='no'/>
  <boot dev='hd'/>
</os>

To list the supported configurations, run:

$ virsh domcapabilities --machine pc | xmllint --xpath '/domainCapabilities/os' -

On my machine, it yields

<os supported="yes">
  <enum name="firmware">
    <value>efi</value>
  </enum>
  <loader supported="yes">
    <value>/usr/share/edk2/ovmf/OVMF_CODE_4M.qcow2</value>
    <value>/usr/share/edk2/ovmf/OVMF_CODE.fd</value>
    <enum name="type">
      <value>rom</value>
      <value>pflash</value>
    </enum>
    <enum name="readonly">
      <value>yes</value>
      <value>no</value>
    </enum>
    <enum name="secure">
      <value>no</value>
    </enum>
  </loader>
</os>

As we gather from the output, the configuration above should work. But if we were to enable Secure Boot, it would no longer work because the only supported value for secure is no.

Generally speaking, there are two ways to fix such a problem:

Change the domain configuration.
Install firmware that supports the domain configuration.

As such problems usually manifest when creating the domain for the first time, changing the domain configuration to match the capabilities is the best option. Furthermore, Linux distributions typically include firmware that covers all supported combinations. If you encounter a combination that is not valid, that is because it is not supported at all.

The only case I can think of where installing firmware makes sense is to test old configurations no longer supported by newer Linux distributions. One example would be using firmware for 2 MB flash devices that are no longer present on Ubuntu 24.04 and newer. In those cases, you can follow the advice in the section Adding OVMF_CODE.fd Back In.

The Easy Way Out

The hands-down easiest way to avoid any problems with UEFI firmware is to forego UEFI boot and rely on the venerable BIOS to boot. You have never seen your problems go away this fast.

If that is not an option, it is time to do it the Windows way. Also known as doing it wrong. In addition to placing the boot loader into the vendor-specific directory (\EFI\Microsoft\boot) in the EFI System Partition (ESP), the Windows installer puts it into the Removable Media Path (\EFI\boot\), too. The reason is that every firmware knows how to start a system from the Removable Media Path because that is how installers are started from removable media (hence the name). By putting the boot loader into the Removable Media Path, Windows avoids problems with buggy boot loaders.

We can do the same and force the installation of grub-efi onto the Removable Media Path. Because the Removable Media Path is known, we no longer need a variable store (OVMF_VARS.fd and friends) that tells the firmware where to look for the boot loader. Consequently, we can get rid of the <nvram/> entry in our domain configuration and tell libvirt to select whatever firmware it finds automatically:

<os firmware='efi'>
  <type arch='x86_64' machine='q35'>hvm</type>
  <loader secure='no'/>
  <boot dev='hd'/>
</os>

That should work everywhere as long as there is some UEFI firmware on the system. Use secure='yes' for UEFI Secure Boot.

This excellent post on the Unix & Linux StackExchange provides a deep dive into the different boot loaders. Force grub-efi installation to the removable media path on the Debian Wiki² provides an explanation of why using the Removable Media Path is wrong, and how to force grub-efi into the Removable Media Path on Debian. Microsoft has instructions on how to do it manually on Ubuntu³.

The firmware image OVMF_CODE.fd (and its siblings with a similar name) is for guests with a 2 MB flash device. 2 MB flash is no longer considered sufficient for use with Secure Boot and does no longer seem to work with recent versions of Windows 11. ↩︎
If you ever need to know more about UEFI, the UEFI page on the Debian Wiki is a great start. ↩︎
Because relying on the Removable Media Path is apparently the only way to boot a Linux guest on a Hyper-V Generation 2 virtual machine. ↩︎

Storing Blobs on the GitHub Container Registry

Sat, 06 Jul 2024 12:00:00 +0200

Recently, I had the need to access some virtual machine images in one of my GitHub Actions workflows. As they came in at more than 1 GB, I was hard-pressed for a place to store them that was also easy to access from a GitHub Actions workflow. If only I could upload a ZIP archive to GitHub Packages!

It turns out that it is not only possible but also much easier than I imagined it to be. Let’s learn how.

Container Images Are Nothing but Fancy Tarballs

The key is to realise that container images (those things that you feed to docker run) are nothing but tarballs and some metadata bundled together. There is no rule¹ that says “Thou shalt only store operating systems in container images.” So, there is nothing that stops us from basically storing anything in a container image. And those can be uploaded to the GitHub Container Registry (GHCR), which is part of the GitHub Packages offering, or any other container registry.

Now, we only need to find a tool that makes putting anything into a container image easy. Enter oras.

Upload Anything with ORAS

oras is a nifty command-line tool made by the ORAS project. ORAS stands for “OCI Registry As Storage”, and OCI is the Open Container Initiative. OCI is the standards body regulating the format of container images, registries, and so on. They ensure that any container image can be run by any runtime (Docker, Podman, containerd, …) and can be uploaded to any registry (GHCR, Docker Hub, …). Thanks to this standardisation, oras is compatible with a dozen different registries.

oras itself is available for all major operating systems and most common platforms. You can either download and run the binary from GitHub Releases or use one of the other installation methods.

I have prepared a directory called data that I want to store on GHCR. It looks like this:

$ tree data
data
├── 50m.bin
└── a-directory
    └── hello.txt

2 directories, 2 files

Uploading it to GHCR is as simple as running:

$ oras push ghcr.io/example/data:1 data

To download the files, create an empty directory and change into it. Then run:

$ oras pull ghcr.io/example/data:1

That’s it! There is now a directory called data in the current working directory that looks exactly like the one I uploaded above:

$ tree data
data
├── 50m.bin
└── a-directory
    └── hello.txt

2 directories, 2 files

While this is not exactly “uploading a ZIP archive”, it is actually much better: You do not have to create the ZIP archive yourself, checksum verification is built-in, and in some circumstances, it is even possible to change the container image without having to re-upload everything.

Saving Space and Bandwith with Layers

In the background, oras takes the directory and turns it into an OCI container image before uploading it. That is the same kind of image that Docker uses. If you have used Docker before, you probably heard about “layers”. In a nutshell, a container image consists of one or more layers. Layer is just a fancy term for a tarball. Having multiple tarballs in an image instead of a single one helps with caching and reducing the amount of storage consumed by all those images in a registry. oras creates layers, too, as we can see when we look at the manifest² of the container image we uploaded:

$ oras manifest fetch ghcr.io/example/data:1 | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "artifactType": "application/vnd.unknown.artifact.v1",
  "config": {
    "mediaType": "application/vnd.oci.empty.v1+json",
    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
    "size": 2,
    "data": "e30="
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:8c9d7307a4263817ad8dd2b845c4bac3a4a59621d98a063d3482df77763e7cee",
      "size": 52445175,
      "annotations": {
        "io.deis.oras.content.digest": "sha256:7520f1358115aa8ffd0ca65b22ba5bf9ef4555e9f9212032f65f8cf91e7ec93a",
        "io.deis.oras.content.unpack": "true",
        "org.opencontainers.image.title": "data"
      }
    }
  ],
  "annotations": {
    "org.opencontainers.image.created": "2024-07-04T15:18:29Z"
  }
}

There it is, a single layer with the SHA-256 checksum 8c9d7307a4263817ad8dd2b845c4bac3a4a59621d98a063d3482df77763e7cee. There is even an annotation called org.opencontainers.image.title with the folder’s name: data. The manifest of a “normal” Docker image does not look much different:

$ oras manifest fetch --platform linux/amd64 docker.io/library/postgres:16.3 | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:f23dc7cd74bd7693fc164fd829b9a7fa1edf8eaaed488c117312aef2a48cafaa",
    "size": 10091
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:f11c1adaa26e078479ccdd45312ea3b88476441b91be0ec898a7e07bfd05badc",
      "size": 29126278
    },
    // Many more layers omitted.
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:95c2c2ef9f02d7666e80992c98c53c9ec7b5e8ccf244d00a5c85e46bbc2820ae",
      "size": 184
    }
  ],
  "annotations": {
    "com.docker.official-images.bashbrew.arch": "amd64",
    "org.opencontainers.image.base.digest": "sha256:39868a6f452462b70cf720a8daff250c63e7342970e749059c105bf7c1e8eeaf",
    "org.opencontainers.image.base.name": "debian:bookworm-slim",
    "org.opencontainers.image.created": "2024-05-09T18:58:11Z",
    "org.opencontainers.image.revision": "d08757ccb56ee047efd76c41dbc148e2e2c4f68f",
    "org.opencontainers.image.source": "https://github.com/docker-library/postgres.git#d08757ccb56ee047efd76c41dbc148e2e2c4f68f:16/bookworm",
    "org.opencontainers.image.url": "https://hub.docker.com/_/postgres",
    "org.opencontainers.image.version": "16.3"
  }
}

Back to those layers. As I mentioned before, it is possible to change the container image created by oras in some circumstances without re-uploading everything. Those circumstances have a lot to do with those layers. When you run oras push <name> <file> [...], oras creates a separate layer per argument. Let’s upload the same directory as before, but this time, specify every file as a separate argument:

$ oras push ghcr.io/example/data:1 data/50m.bin data/a-directory/hello.txt

While the result on disk is the same when we download the image again, the manifest looks different:

$ oras manifest fetch ghcr.io/example/data:1 | jq                            
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "artifactType": "application/vnd.unknown.artifact.v1",
  "config": {
    "mediaType": "application/vnd.oci.empty.v1+json",
    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
    "size": 2,
    "data": "e30="
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:1e4c2dd682422beba2fa33db0f926935afe1414f722ee54be7788c6a6c40ebca",
      "size": 52428800,
      "annotations": {
        "org.opencontainers.image.title": "data/50m.bin"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:03ba204e50d126e4674c005e04d82e84c21366780af1f43bd54a37816b6ab340",
      "size": 13,
      "annotations": {
        "org.opencontainers.image.title": "data/a-directory/hello.txt"
      }
    }
  ],
  "annotations": {
    "org.opencontainers.image.created": "2024-07-05T13:51:19Z"
  }
}

There is now one layer per file, two in total. That means you can now change the contents of the image in place by adding or removing layers. Let’s add another file, data/10m.bin:

$ oras push ghcr.io/example/data:1 data/50m.bin data/10m.bin data/a-directory/hello.txt

You will see that oras only uploads data/10m.bin because the other two files (layers) are already part of the image. Omit data/50m.bin and oras will only delete its layer, leaving everything else in place:

$ oras push ghcr.io/example/data:1 data/10m.bin data/a-directory/hello.txt

So if you expect that you need to update parts of the container image frequently or you want to save space³ when storing multiple images that share some contents, it might be beneficial to put every file, or at least every folder, in a separate layer as shown in the preceding examples. If you want to save on typing, find and xargs can help:

$ find data -type f -print0 | xargs -r -0 oras push ghcr.io/example/data:1

oras has more to offer, but what we have seen so far should suffice for everyday use.

ORAS in a GitHub Actions Workflow

This is a minimal GitHub Actions workflow to download our data folder onto the runner:

name: Build

on:
  push:

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: read  # Required to access GHCR

    steps:
      - name: Install oras
        run: |
          sudo snap install oras --classic          

      - name: Download data
        run: |
          oras login --username "${{ github.actor }}" --password "${{ secrets.GITHUB_TOKEN }}" ghcr.io
          oras pull ghcr.io/example/data:1

The highlights:

You need to declare the permission packages: read to access GHCR. If you use a different container registry, you can omit it.
I install oras using snap. Any other method is fine, too.
Log into GHCR with ${{ github.actor }} as username and ${{ secrets.GITHUB_TOKEN }} as password. This also saves you some money⁴.

Then, you can use oras as usual.

If the container images you are accessing are private, and they are private by default, you also have to link the image with the repository that the GitHub Actions workflow is part of. Otherwise, you get permission errors. There are two ways to do this:

Connect a repository to a package using the GitHub UI.
You can add the annotation org.opencontainers.image.source to the container image. Assuming you want to access the image in https://github.com/example/my-repository, then the command would look as follows:
```
$ oras push ghcr.io/example/data:1 \
    -a "org.opencontainers.image.source=https://github.com/example/my-repository" \
    data/50m.bin data/10m.bin data/a-directory/hello.txt
```

Docker Can Do This, Too

Before I stumbled upon ORAS, I tried my luck with a normal image builder. My preferred tool is Buildah, and it can actually do it. The key is to use the empty base image scratch. The equivalent to oras push ghcr.io/example/data:1 data looks as follows:

$ export newcontainer=$(buildah from scratch)
$ buildah unshare
$ buildah copy $newcontainer data /data
$ buildah unmount $newcontainer
$ buildah commit $newcontainer data
$ buildah rm $newcontainer
$ buildah push data:latest ghcr.io/example/data:1

When you think that this is kinda gross, it absolutely is.

Docker does not fare better. First, we need a Dockerfile next to the data folder:

FROM scratch
COPY data /data

Then, we can build the image and push it to GHCR:

$ docker build -t ghcr.io/example/data:1 .
$ docker image push ghcr.io/example/data:1

Skopeo is probably the best tool (relatively speaking) to get the data folder back:

$ skopeo copy docker://ghcr.io/example/data:1 dir:output

This command will extract the image into the pre-existing folder output. Unfortunately, we are far from done. We still have to look into the manifest to figure out which file contains the filesystem layer and what compression algorithm was used. Then, we can extract it with tar to get our folder data back.

This is absolutely no fun, and nobody should do it. I only wanted to mention it. After all, you never know when this otherwise useless knowledge might come in handy.

GitHub itself showcases that Homebrew stores at least half a petabyte of binaries on GHCR. If you are curious how Homebrew does it: Homebrew writes the OCI image itself and then uploads it using skopeo. ↩︎
A manifest is a piece of metadata that describes the contents of a container image. ↩︎
Container registries usually store each layer only once, even if it is part of hundreds or thousands of images. ↩︎
Data transfer is free of charge when GHCR is accessed with GITHUB_TOKEN. ↩︎

Helping You Find the JDK That Fits Your Needs

Wed, 27 Sep 2023 12:00:00 +0200

In the last few years, the choice of JDKs has exploded. Almost everyone offers one, even Microsoft. Unsurprisingly, people struggle to pick a JDK and are looking for help.

While we can learn that Java Is Still Free and get told which JDK we should use, nobody seems to provide a comprehensive overview of the differences between the various JDKs. The best I could find is OpenJDK vs Oracle JDK – Comparison Table authored by Azul¹.

Therefore, I have created JDK Comparison.

Introducing JDK Comparison

JDK Comparison allows you to compare most² JDKs on the market, feature by feature. Multiple filters help you narrow your choices. There is even a button that lets you focus on the differences by hiding the commonalities. Dozens of explanations and footnotes should help you understand what each feature means and provide additional sourcing.

While compiling the list, I learnt some interesting things:

It is hard³ to figure out what additional patches the vendors apply to their JDKs. In most cases, it is nothing more than a custom trust store and a handful of backports (Customisations → Customisations).
If you need a JDK for Windows on ARM, you have only two options for JDK 21 and three options for JDK 17 (Platforms: Windows → ARM, 64-bit).
Alibaba Dragonwell and Eclipse Temurin are the only JDKs with a Software Bill of Materials (Security → SBOM).
Eclipse Temurin is the only JDK with Windows-based container images (Platforms: Windows → Container Images).
Neither Amazon nor Microsoft offer support for their JDKs for workloads that do not run on their clouds (Support → Paid Support).

Apart from the comparison itself, there is a page that answers frequently asked questions. For example, it details the options you have if you want to move away from Oracle JDK 8 and need JavaFX or Web Start.

Tell Me What You Think

I am happy to add features to the comparison, expand the FAQ, or fix errors. I would also be thrilled to accept contributions. If you find the comparison helpful, it would be terrific if you spread the word or tell me directly about it.

Unfortunately, it contains some puzzling statements. For example, it claims that Amazon does not have the “Engineering capacity to root-cause & fix bugs” despite employing multiple prolific OpenJDK contributors. ↩︎
The FAQ outlines which JDKs are on the to-do list and why some will not be added. ↩︎
If you work for a vendor: Please publish change logs and link them from your downloads page. Ideally, publish the full source code that you used to build the JDK (not src.zip) on the very same download page. ↩︎

Testing Container Images

Fri, 23 Dec 2022 12:00:00 +0100

More and more software is delivered as a container image for container runtimes like Docker, Podman, and Containerd. Even simple container images require substantial logic for their creation and configuration (for example, the official nginx image). Hence, container images should be automatically tested like all other software deliverables to prevent bugs and security issues from creeping in.

In this article, I present multiple tools that help to examine container images and their configuration.

The basis forms a hypothetical container image for the webserver nginx called myimage:

FROM debian:buster

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
        tini nginx procps

ENTRYPOINT ["/usr/bin/tini", "--", "nginx"]
EXPOSE 80
STOPSIGNAL SIGQUIT
CMD ["-g", "daemon off;"]

With each tool, I want to test the following aspects of the resulting container image:

Is nginx installed?
Do the nginx workers run as an unprivileged user?
Is nginx properly exposed to the host and serves a welcome page when accessed from the host?
Does the container image only expose port 80 by default?

While these are very simple tests, they illustrate whether and how it is possible to assess the internal state of a container, the external state as it can be observed from the host, and the configuration of the container image.

The complete source code can be found in my sample repository on GitHub.

Bash Automated Testing System

Bash Automated Testing System, Bats for short, is a test framework for Bash. Being able to write the tests in Bash is the biggest advantage of Bats, but also its biggest drawback.

What is great about writing tests in Bash is that there is no additional API or framework to learn. You can write the tests using the same commands you type into your terminal. Furthermore, anything that you can do in the terminal can be tested. That is not the case with the other tools presented here that provide dedicated but often limited built-in inspections to assess the state of a system.

What is bad about writing tests in Bash is that you have to use Bash. Not because Bash were a bad choice, but because you suddenly have to deal with unexpected carriage returns returned from Docker that are also invisible¹, lots of quoting, and workarounds required by Bats to capture output. Furthermore, the lack of an API means that some of the tests cannot be portable², and you need to figure out how to use tools like ps.

Apart from that, you get what you would expect from a basic test framework, like setup functions, assertions, and test reports that your build server can process, thanks to TAP support.

The test to check whether all nginx workers run as unprivileged user shows the advantages and problems of using Bats pretty well:

@test "nginx workers do not run as root" {
    MASTER_PID=$(docker exec -i $CONTAINER_ID ps -ouser=,pid= -C nginx | awk '($1=="root"){print $2}')
    RESULT=$(docker exec -e MASTER_PID=$MASTER_PID -i $CONTAINER_ID ps -ouser= --ppid $MASTER_PID | uniq)
    assert_equal "$RESULT" "www-data"
}

Bats in a Nutshell

All tests could be implemented: ✅
Generates test reports: ✅
Supported container runtimes: any with a CLI
Pros: Bash, easy to adopt
Cons: You need to deal with Bash’s quirks, installation requires submodules, no built-in inspections
Complete source code

Goss

Goss is similar to Serverspec and Testinfra in the sense that it is not a test framework but a tool to inspect the state of a system. But it is still very different.

First, Goss runs on the target system. It “sees” the container only from the inside. Consequently, it is not possible to test whether nginx is properly exposed to the host or whether the image is correctly configured. You might not need that functionality, but it is something to be aware of.

Then, Goss is declarative. Instead of writing tests, you describe the desired system state in a YAML file. Goss takes that YAML file and checks whether the system matches the expectations outlined in it. Fortunately, Goss can assist you in creating the YAML file. For example, run

goss add package nginx

and Goss will add an entry like

package:
  nginx:
    installed: true
    versions:
    - 1.17.8

to your YAML file if nginx is already installed on the system. If you do not care about the exact version being installed, you can remove the versions attribute by hand.

Like many declarative tools, Goss provides some means to escape the confines of being entirely declarative. Some attributes accept regular expressions, and there is basic support for some programming constructs like loops:

file:
{{- range mkSlice "/etc/passwd" "/etc/group"}}
  {{.}}:
    exists: true
    mode: "0644"
    owner: root
    group: root
    filetype: file
{{end}}

Unfortunately, the syntax is proprietary and incompatible with goss add (as of Goss 0.3.20).

Like Testinfra and Serverspec, Goss has a decent but still limited number of inspections built-in. If something is not supported by Goss, you can fall back to the command line, like in this test that checks whether all nginx workers run as an unprivileged user:

command:
  ps -ouser= --ppid $(ps -ouser=,pid= -C nginx | awk '($1=="root"){print $2}') | uniq:
    exit-status: 0
    stdout:
    - www-data
    stderr: []
    timeout: 10000

Goss in a Nutshell

All tests could be implemented: ❌
Generates test reports: ✅
Supported container runtimes: Docker, Docker Compose, Kubernetes
Pros: No additional dependencies required, tests can be generated automatically, fast
Cons: Can only assess the container internally, limited expressiveness due to YAML
Complete source code

Serverspec

Serverspec is an extension for RSpec, a BDD framework for Ruby. That means that you write your tests in Ruby as RSpec specifications.

Serverspec provides many built-in inspections, called resource types, that simplify writing tests and give some portability. As a result, checking whether nginx is installed is as simple as

describe package('nginx') do
  it { should be_installed }
end

regardless of the Linux distribution you are using. If the built-in inspections do not cut it, you can still fall back to the command line as I had to check whether all nginx workers run as an unprivileged user:

describe command('ps -ouser= --ppid $(ps -ouser=,pid= -C nginx | awk \'($1=="root"){print $2}\') | uniq') do
  its(:stdout) { should eq("www-data\n") }
end

Thanks to Serverspec being a Rspec extension, you get all the features you can expect from a mature testing framework.

The development of Serverspec has been slow in the past years, and the author only accepts pull requests, but no issues.

Serverspec in a Nutshell

All tests could be implemented: ✅
Generates test reports: ✅
Supported container runtimes: Docker
Pros: Many built-in inspections, based on a mature test framework
Cons: none
Complete source code

Testcontainers

Testcontainers is actually a tool that helps you write integration tests for your applications. But you can still misuse it for testing container images. In that regard, it is most similar to Bats. But instead of writing your tests in Bash, you can use Java.

Testcontainers supports many programming languages besides Java, like C# or Go. I have not tried those and do not know what their capabilities are. This section only discusses the Java version.

Testcontainers excels at simplifying the interaction with Docker. Starting, stopping, and inspecting containers is a breeze. On the other hand, it does not provide any built-in inspections, so you have to write them yourself. For example, the test to check whether all nginx workers run as an unprivileged user looks like that:

@Test
void testNginxWorkersDoNotRunAsRoot() throws IOException, InterruptedException {
	var masterPidCmd = container.execInContainer("/bin/bash", "-c",
			"ps -ouser=,pid= -C nginx | awk '($1==\"root\"){print $2}'");
	var masterPid = Integer.parseInt(masterPidCmd.getStdout().trim());
	var workersUserCmd = container.execInContainer("/bin/bash", "-c",
			String.format("ps -ouser= --ppid %d | uniq", masterPid));

	assertThat(workersUserCmd.getStdout().trim()).isEqualTo("www-data");
}

As we are using Java, we are free to choose between any of Java’s build tools, testing frameworks, and assertion libraries. I picked JUnit 5 and AssertJ.

Testcontainers probably makes the most sense if the container tests are part of a larger Java project because you are probably using some of the tools already and can leverage Gradle or Maven to drive the build. But thanks to JBang, even a standalone usage is possible, as can be seen in the sample source code.

Testcontainers in a Nutshell

All tests could be implemented: ✅
Generates test reports: ✅
Supported container runtimes: Docker
Pros: Simplifies interaction with Docker
Cons: no built-in inspections
Complete source code

Testinfra

Testinfra is a plug-in for pytest, a testing framework for Python. As a result, you write your tests in Python.

Testinfra includes many built-in inspections, called modules, that simplify writing tests and provide some portability. As a result, checking whether nginx is installed is as simple as

def test_nginx_is_installed(host):
    nginx = host.package("nginx")
    assert nginx.is_installed

regardless of the Linux distribution you are using. If any of the built-in inspections is not sufficient, you can fall back to the command line. But I did not have to do this a single time. Consequently, testing whether all nginx workers run as an unprivileged user is surprisingly readable:

def test_nginx_workers_do_not_run_as_root(host):
    master = host.process.get(user="root", comm="nginx")
    workers = host.process.filter(ppid=master.pid)
    users = set([worker.ruser for worker in workers])
    assert set(['www-data']) == users

Another advantage of Testinfra is that it supports more container runtimes than any other tool listed here.

Thanks to Testinfra being a pytest plug-in, you get all the features you can expect from a mature testing framework.

Testinfra in a Nutshell

All tests could be implemented: ✅
Generates test reports: ✅
Supported container runtimes: Docker, Kubernetes, LXC, OpenShift, Podman
Pros: Large number of built-in inspections, based on mature test framework
Cons: none
Complete source code

For some extra fun. ↩︎
While, for example, ps works the same across Linux distributions, each family of distributions uses different package managers. Therefore, there is no single command to check whether a package is installed that works across all Linux distributions. ↩︎

Comparing Parallel Test Execution in JUnit 5, Gradle, and Maven

Thu, 01 Dec 2022 12:00:00 +0100

JUnit 5, Gradle, and Maven: All offer parallel execution of unit tests. With so many options, I wondered whether it makes a difference what I pick. It does.

Capabilities of the Contenders

JUnit 5

Parallel test execution is an experimental feature of JUnit 5 that has been available since version 5.3. Once enabled, you can define whether you want to run top-level test classes or methods sequentially, concurrently or any combination thereof. JUnit 5 runs all tests in a single JVM using a ForkJoinPool. There are various knobs to control how many tests should be run in parallel.

Gradle

Gradle can execute tests in parallel, too. It works with JUnit 5, JUnit 4, TestNG and probably other frameworks. maxParallelForks is used to enable (or disable) parallel test execution and control how many tests should be run in parallel. Any value bigger than 1 enables test parallelism. Compared to the parallel test execution built into JUnit 5, Gradle can only run test classes in parallel, not individual test methods. Each test class is assigned to a single worker process (a forked JVM). In other words: Gradle runs all test methods of a single test class sequentially in the same JVM that is not shared with any other test.

tasks.withType(Test).configureEach {
    // Creates half as many forks as there are CPU cores.
    maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1
}

Maven

Maven’s Surefire plug-in can execute tests in parallel as well. Its capabilities differ by testing framework: JUnit 4, TestNG. However, there is no dedicated support for parallel testing within the same JVM in Maven for JUnit 5. You can use the parallel test execution built into JUnit 5 instead.

It is important to note that all of Maven’s options mentioned so far achieve concurrency within the same JVM as JUnit 5. If you want that Maven runs each test in a separate process, you need the option forkCount with a value higher than 1. It works with all test frameworks supported by Surefire, including JUnit 5.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!-- other options -->

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M7</version>
                <configuration>
                    <!-- Creates half as many forks as there are CPU cores. -->
                    <forkCount>0.5C</forkCount>
                    <reuseForks>true</reuseForks>
                </configuration>
            </plugin>
            <!-- other plug-ins -->
        </plugins>
    </build>
</project>

Concurrency Within the Same JVM Is Dangerous

In summary, there is one big difference between the parallel test execution of JUnit 5, Gradle, and Maven: Whether the concurrently running tests share a single JVM or not. That has a massive impact on your tests.

In short, once your tests share a single JVM, your tests become multi-threaded code which can cause all kinds of issues. For example, changing the default locale in one method impacts all other running tests. Or a single test does no longer have its Testcontainers Container for itself when using the recommended patterns.

JUnit 5 supports several synchronisation mechanisms to prepare your tests for parallel execution within the same JVM. But for me, that requires too much thought and work for very little gain¹ if you can use multiple workers in Gradle and Maven without requiring any² code changes.

Spawning multiple threads is more efficient than creating multiple processes. ↩︎
As long as your tests do not share external resources, like a single database. ↩︎

Fast, Reliable Integration Tests with Spring Boot and Flyway

Fri, 25 Nov 2022 12:00:00 +0100

For integration tests to be effective, they must be reliable and fast. If tests take too long to run, nobody will run them. If tests are unreliable, everybody will ignore them.

To be reliable, tests may not result in false positives or negatives. Furthermore, they should not be flaky. That can be achieved by isolating the tests from each other, for example, by supplying a dedicated test database to each test with the help of Testcontainers. Additionally, tests should start from a known, consistent good state. That can be achieved by resetting the database to a known, good state before every test. Unfortunately, all that additional work makes tests slower.

In this post, we will look at various ways to write reliable integration tests in Spring Boot with the help of Flyway and Testcontainers. And then, we will look at which one is the fastest.

A working implementation of all approaches with the full source code can be found in my sample project on GitHub.

Setting the Stage

If we follow the Spring Boot documentation about writing tests with Testcontainers, we probably end up with something like the following test class to test an AuthorRepository that gives us access to book authors stored in the database:

@JdbcTest
@ContextConfiguration(
	initializers = AuthorRepositoryTest.class,
	classes = AuthorRepository.class
)
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
@Commit
@Testcontainers
public class AuthorRepositoryTest
	implements ApplicationContextInitializer<ConfigurableApplicationContext>
{
	@Containers
	static final PostgreSQLContainer postgres = new PostgreSQLContainer("postgres:15.1");

	@Autowired
	AuthorRepository authorRepository;

	@Override
	public void initialize(@NotNull ConfigurableApplicationContext applicationContext) {
		TestPropertyValues
			.of(
				"spring.datasource.url=" + postgres.getJdbcUrl(),
				"spring.datasource.password=" + postgres.getPassword(),
				"spring.datasource.username=" + postgres.getUsername(),
				"spring.flyway.clean-disabled=false"
			)
			.applyTo(applicationContext);
	}

	@Test
	void add_insertsAuthor() {
		var author = new Author(AuthorId.from("82f4870d-f2e5-4a9f-a2b2-b297f66733a0"), "Brian Goetz");

		this.authorRepository.add(author);

		assertThat(this.authorRepository.findAll())
			.extracting(Author::name)
			.containsExactly("Bert Bates", "Brian Goetz", "Joshua Bloch", "Kathy Sierra", "Trisha Gee");
	}
}

Spring Boot automatically initialises the database before the first test method is run but does nothing after that. Every other test after add_insertsAuthor() has to somehow deal with another author’s presence (or absence, in the case of a test failure) in the database. That makes tests brittle.

One alternative could be that tests clean up after themselves. But that needs work and is problematic if we want to diagnose a test failure. Would it not be much better if the database was automatically brought into a consistent state before every test?

Spring Boot offers at least three different ways to do this that actually work:

Using a custom FlywayMigrationStrategy.
Invoking Flyway from a TestExecutionListener.
Eschewing Flyway and using @Sql with a SQL script instead.

There are ways that do not work properly, too. For example, using a JUnit setup method (a method annotated with @BeforeEach in JUnit 5) to invoke Flyway makes @Sql unusable. The reason is that the TestExecutionListener that processes @Sql annotations runs before @BeforeEach. Thus, the database would still be empty when @Sql is applied.

The Options, Explained

FlywayMigrationStrategy

When Spring Boot starts, its autoconfiguration for Flyway invokes the registered FlywayMigrationStrategy to apply all pending migrations. By supplying your own FlywayMigrationStrategy, you can alter that behaviour and invoke clean() before migrate() and thereby reset the test database:

@Bean
public FlywayMigrationStrategy flywayMigrationStrategy() {
	return flyway -> {
		flyway.clean();
		flyway.migrate();
	};
}

As mentioned before, the FlywayMigrationStrategy is only applied when the application starts. To make Spring apply it again before each test, you have to trigger a context refresh by annotating the test class with @DirtiesContext(classMode = AFTER_EACH_TEST_METHOD).

This approach requires the least amount of code but is also the slowest. @DirtiesContext causes Spring to rebuild its context. That is an expensive operation, especially if you declare many beans or use multiple starters.

For a complete and commented implementation of this approach, see the relevant test in the sample project.

Flyway with TestExecutionListener

TestExecutionListener is an API provided by Spring to react to test-execution events. For example, it allows running custom code before (or after) the execution of each test method. Therefore, it can serve as a replacement for FlywayMigrationStrategy without the need for a context refresh.

public class CleanDatabaseTestExecutionListener
	implements TestExecutionListener, Ordered
{
	@Override
	public void beforeTestMethod(@NotNull TestContext testContext) {
		var flyway = testContext.getApplicationContext().getBean(Flyway.class);
		flyway.clean();
		flyway.migrate();
	}

	@Override
	public int getOrder() {
		// Ensures that this TestExecutionListener is run before
		// SqlScriptExecutionTestListener which handles @Sql.
		return Ordered.HIGHEST_PRECEDENCE;
	}
}

@TestExecutionListeners(
	value = CleanDatabaseTestExecutionListener.class,
	mergeMode = MERGE_WITH_DEFAULTS
)

The last step is to register a FlywayMigrationStrategy that does nothing. Otherwise, the autoconfiguration would apply the migrations and the TestExecutionListener would apply them a second time before the first test runs.

@Bean
public FlywayMigrationStrategy flywayMigrationStrategy() {
	return flyway -> { /* Do nothing. */ };
}

While this approach requires significantly more code than FlywayMigrationStrategy, it is considerably faster, too, because we could get rid of the expensive context refresh after each test.

For a complete and commented implementation of this approach, see the relevant test in the sample project.

@Sql (Almost) Without Flyway

The previous approaches suffer from a common problem: They run Flyway before every test and have to apply each and every migration over and over again to bring the test database into a consistent state. That is fine if a project only has a few migrations. But as soon as there are more than a couple of migrations, their execution time starts adding up. I pulled up an old project with around 50 migrations and measured how long they take to run: between 1 and 2 seconds in total. There are about 600 test methods across all integration tests. Therefore, a single build spends between 10 and 20 CPU minutes applying migrations alone – what a waste.

The solution is to remove Flyway from the integration tests:

Apply all migrations on an empty database, for example, with the help of Flyway’s plug-ins for Gradle and Maven.
Dump the database to a SQL script with pg_dump, mysqldump, and so on. Specify the necessary options so that the generated script includes the commands to drop all database objects if they already exist. That ensures that the database does not contain any leftovers from previous test runs.

Use the SQL script in the integration tests to reinitialise the database before every test instead of Flyway:

@JdbcTest(properties = "spring.flyway.enabled=false")
@Sql(scripts = "/db.sql", config = @SqlConfig(transactionMode = ISOLATED))
// Other annotations
public class AuthorRepositoryTest {
	// Tests
}

This approach significantly cuts down on the total runtime because it creates all database objects once per test without replaying the entire migration history.

Unfortunately, this speed-up comes at a cost: The SQL script needs to be checked into source control. And someone needs to keep the database dump in sync with the Flyway migrations. However, it is possible to avoid the maintenance burden by automating the generation of the database dump. Please see Keeping the Growing Number of Flyway Migrations in Check for how to do this.

For a complete and commented implementation of this approach, see the relevant test in the sample project.

Performance

I did some measurements to give you a very rough idea of how the various approaches perform relative to each other:

Method	Runtime
`FlywayMigrationStrategy`	100%
Flyway with `TestExecutionListener`	~70%
`@Sql` without Flyway	~50%

The numbers are indicative at best because they highly depend on the specifics of your project and the hardware. For example, if your application defines a lot of beans or uses many starters, the FlywayMigrationStrategy will perform worse because the context refresh takes even longer. Therefore, the gap between it and the other two methods widens. If you have many Flyway migrations, @Sql without Flyway will be even faster.

Alternatives

Rollback

One popular method to keep the test database in a consistent state is to automatically rollback the transaction after each test. It is even the default behaviour of the testing support in Spring Framework.

I believe it is not wise to rely on rollback because it might hide errors. Database systems do not necessarily check every constraint after every statement but might defer it until later in the transaction. One example is PostgreSQL’s behaviour with ON DELETE NO ACTION.

Another drawback of testing with rollback is that it might hinder or even make it impossible to test complex scenarios. For example, in batch processing it is common that the transaction is committed after each batch has been processed. How are you supposed to test that only the last (failing) batch is discarded with rollback testing?

Baseline Migrations

Flyway Teams Edition, a paid version of Flyway, offers Baseline Migrations ¹. In a nutshell, you can define a separate SQL script that brings a database to the state of a specific version (the baseline version), skipping all migrations up to and including the baseline version.

If you keep only a handful of migrations around, using Flyway with baseline migrations should perform similarly to @Sql without Flyway. However, someone has to manually create the baseline migration. And you need Flyway Teams Edition.

Not to be confused with Flyway’s baseline command that allows you to introduce Flyway to existing databases and to delete old migrations without Flyway complaining about them being absent. ↩︎

Keeping the Growing Number of Flyway Migrations in Check

Thu, 24 Nov 2022 12:00:00 +0100

Flyway is a fantastic tool for managing an application’s database alongside the application’s source code. But even in small projects, the number of migrations grows inevitably. That is no problem for application startup. But if you use Flyway to initialise the database before every integration test to ensure your tests are independent and reliable, the tests take longer with every migration you add. For example, if you have 50 migrations that take 1 second to run and there are 600 test methods, you would spend 10 CPU minutes alone on creating test databases. There must be a better way.

Fortunately, there is. Unfortunately, it requires a fair number of custom Gradle tasks.

The idea is to automatically do the following before running the tests:

Provision a database container.
Apply all migrations to the database with Flyway.
Create a database dump and place it on the classpath for future consumption.
Remove the container.

Before a test is about to run, the test database can be initialised with the previously created database dump without any involvement of Flyway. That is much faster than applying each migration individually. Furthermore, the time needed to create the test database increases with the number of database objects and no longer with the number of migrations.

The implementation is relatively straightforward thanks to the Gradle Docker plugin:

def dumpPath = project.buildDir.toPath().resolve('resources/test/db.sql')

task pullDatabaseImage(type: DockerPullImage) {
	def stateFile = project.buildDir.toPath().resolve("tmp/squash.txt")

	inputs.files(fileTree('src/main/resources/db/migration'))
		.withPropertyName('migrations')
		.withPathSensitivity(PathSensitivity.RELATIVE)
	outputs.file(stateFile)
		.withPropertyName('stateFile')

	image = 'docker.io/library/postgres:15.1'

	doLast {
		Files.writeString(stateFile, getImage().get(), StandardOpenOption.CREATE,
			StandardOpenOption.TRUNCATE_EXISTING)
	}
}

task createDatabaseContainer(type: DockerCreateContainer) {
	dependsOn pullDatabaseImage

	onlyIf { pullDatabaseImage.didWork }

	targetImageId pullDatabaseImage.getImage()
	hostConfig.portBindings = ['5432:5432']
	hostConfig.autoRemove = true
	withEnvVar('POSTGRES_USER', 'flyway')
	withEnvVar('POSTGRES_PASSWORD', 'flyway')
	withEnvVar('POSTGRES_DB', 'squash')
}

task startDatabaseContainer(type: DockerStartContainer) {
	dependsOn createDatabaseContainer

	onlyIf { createDatabaseContainer.didWork }

	targetContainerId createDatabaseContainer.getContainerId()
}

task stopDatabaseContainer(type: DockerStopContainer) {
	dependsOn startDatabaseContainer

	onlyIf { startDatabaseContainer.didWork }

	targetContainerId startDatabaseContainer.getContainerId()
}

task runFlywayMigrations(type: FlywayMigrateTask) {
	dependsOn startDatabaseContainer

	onlyIf { startDatabaseContainer.didWork }

	driver = 'org.postgresql.Driver'
	url = 'jdbc:postgresql://localhost:5432/squash'
	user = 'flyway'
	password = 'flyway'
	connectRetries = 5
}

task dumpDatabase(type: DockerExecContainer) {
	dependsOn runFlywayMigrations

	onlyIf { runFlywayMigrations.didWork }

	targetContainerId startDatabaseContainer.getContainerId()
	withCommand(['pg_dump', '--clean', '--if-exists', '--inserts',
                 '--disable-dollar-quoting', '--no-owner', '-d', 'squash',
                 '-U', 'flyway', '--file', '/tmp/db.sql'])
}

task copyDatabaseDumpToHost(type: DockerCopyFileFromContainer) {
	dependsOn dumpDatabase
	finalizedBy stopDatabaseContainer

	onlyIf { dumpDatabase.didWork }
	outputs.file(dumpPath)

	targetContainerId startDatabaseContainer.getContainerId()
	remotePath = '/tmp/db.sql'
	hostPath = dumpPath.toString()
}

tasks.withType(ProcessResources).configureEach {
	dependsOn copyDatabaseDumpToHost
}

Two things to highlight:

The database dump is written to build/resources/test/db.sql. As such, it ends up on the test classpath of the project. Tests can pick it up from there by referencing classpath:/db.sql (Spring notation).
The tasks participate in incremental builds. As a result, they only run if any of the migrations change or the database dump is missing. Consequently, the tasks do not impact build times during development once the database dump has been created. And the database dump is never outdated.

The example uses PostgreSQL. It should be possible to adapt it to any other database system (like MariaDB or even Oracle) as long as you can get ahold of a container image with that database system.

A working implementation with the full source code can be found in my sample project on GitHub. The most relevant parts are the build file and the test that uses the created database dump.

Managing Xcode on the Command Line

Thu, 23 Dec 2021 12:00:00 +0100

xcode-install is a nifty tool to install and manage Xcode over the command line. Furthermore, it supports installing multiple Xcode versions side-by-side. It can even install all kinds of simulators and works great with automation tools like Ansible.

Are you more of a GUI person? You might find XcodesApp helpful.

xcode-install uses fastlane under the covers. It downloads everything from the Apple Developer Portal. Consequently, you need a free ADP login to use xcode-install.

Authenticating to the Apple Developer Portal

Unfortunately, automatically logging into the Apple Developer Portal is a bit of a pain since Apple introduced 2FA. If you still have an old account without 2FA enabled, you can get away with a few environment variables:

ec2-user@Mac-mini ~ % export XCODE_INSTALL_USER=yourappleid
ec2-user@Mac-mini ~ % export XCODE_INSTALL_PASSWORD=yourpassword
ec2-user@Mac-mini ~ % export SPACESHIP_SKIP_2FA_UPGRADE=1

If you do not want that fastlane stores your credentials, set FASTLANE_DONT_STORE_PASSWORD=1, too.

If your account has 2FA enabled, you have to create a login cookie first:

ec2-user@Mac-mini ~ % fastlane spaceauth -u yourappleid
ec2-user@Mac-mini ~ % export XCODE_INSTALL_USER=yourappleid
ec2-user@Mac-mini ~ % export XCODE_INSTALL_PASSWORD=yourpassword

The login cookie expires and needs to be renewed from time to time.

If you cannot create the login cookie on the remote machine, you have still options. fastlane spaceauth prints the login cookie, including the necessary export command to the terminal. This allows you to “transfer” the cookie to the remote shell. If this is not possible, either, you can take advantage of fastlane storing the cookie in ~/.fastlane/spaceship. Copy ~/.fastlane/spaceship with all its contents to the remote machine, and you should have the login cookie available.

A Short Tour of xcversion

Before installing Xcode, ensure that you have at least 30 GB of free disk space available.

Installing Xcode is a matter of running

ec2-user@Mac-mini ~ % xcversion install 13.2.1

and some patience. Unpacking and installing Xcode takes roughly half an hour. To see all available Xcode versions, run

ec2-user@Mac-mini ~ % xcversion list
4.3 for Lion
(...)
13.2
13.2.1 (installed)

Installing simulators is easy, too. xcversion simulators gives you a list of all available simulators including their installation status. To install a specific simulator, for example, tvOS 15, run:

ec2-user@Mac-mini ~ % xcversion simulators --install="tvOS 15.0"