Autoscaling Forgejo Runner

Over the last months, the Forgejo community equipped Forgejo and Forgejo Runner with the necessary building blocks for autoscaling Forgejo Runner. These building blocks should enable developers of workload managers and autoscalers to build reliable and secure integrations for Forgejo Actions. This post aims to provide an overview of the new capabilities, explain how they fit together, and highlight some peculiarities to watch out for.

Autoscaling in a Nutshell

To scale Forgejo Runner instances up and down, an external system has to perform the following tasks:

• Monitor Forgejo’s job queue for jobs that are ready to run.
• Create runners in response to jobs that are ready to run.
• Launch Forgejo Runner to run a job.
• Remove runners that are no longer needed.

Start by asking Forgejo for jobs that are ready to run:

$ curl http://localhost:3000/api/v1/repos/andreas/test/actions/runners/jobs
[{"id":982,"attempt":1,"handle":"33ba7d51-59c6-44f8-9d2b-1b94f4033973","repo_id":1,"owner_id":1,"name":"build","needs":null,"runs_on":["debian"],"task_id":0,"status":"waiting"}]

Then, create a runner to run that job:

$ curl -X POST \
	-H "Content-Type: application/json" \
	-d '{"name":"0CEY3JWJ6A4ZK","ephemeral":true}' \
	http://localhost:3000/api/v1/repos/andreas/test/actions/runners
{"id":5,"uuid":"d4637a60-7fea-4075-a81c-7e4ae01bdf0d","token":"e1f3d26e034687a2b6856c4c6c3d87a469ff3bea"}

Run the job:

$ echo -n "e1f3d26e034687a2b6856c4c6c3d87a469ff3bea" > /tmp/runner-token
$ forgejo-runner one-job \
	--url http://localhost:3000/ \
	--uuid d4637a60-7fea-4075-a81c-7e4ae01bdf0d \
	--token-url file:/tmp/runner-token \
	--label debian:docker://node:lts \
	--handle 33ba7d51-59c6-44f8-9d2b-1b94f4033973 \
	--wait
INFO[0000] No configuration file specified; using default settings. 
INFO[2026-04-11T20:18:39+02:00] Starting job
INFO[2026-04-11T20:18:39+02:00] runner: 0CEY3JWJ6A4ZK, with version: v12.8.2, with labels: [debian], ephemeral: true, declared successfully 
INFO[2026-04-11T20:18:39+02:00] single task poller launched
INFO[2026-04-11T20:18:39+02:00] single task poller successfully fetched one task from http://localhost:3000/ 
INFO[2026-04-11T20:18:39+02:00] task 917 repo is andreas/test https://data.forgejo.org http://localhost:3000/ 
INFO[2026-04-11T20:18:47+02:00] Cleaning up network for job build, and network name is: WORKFLOW-891112e47856a708ddac3fdd3d034bc0 
INFO[2026-04-11T20:18:47+02:00] single task poller is shutting down 

And that’s it! Because I created an ephemeral runner, Forgejo will remove the runner automatically after the job has completed.

About Scopes

Forgejo Actions has four scopes: repository, user, organization, and instance. For example, a single runner can be bound to a repository, a user, an organization, or the entire instance. While a runner that is bound to a single repository can only run jobs triggered by that repository, a runner that is bound to an entire organization can run jobs triggered by any repository belonging to that organization. Variables and secrets work similarly.

When managing runners using the web UI, you can manage the runners of the current scope. In addition, all runners of superior scopes are visible. The instance scope works slightly different. It shows all runners because it doubles as admin scope.

The HTTP API uses the same scopes. That means a single endpoint like actions/runners/jobs is available in each scope:

• Admin: /admin/actions/runners/jobs
• Organization: /orgs/{org}/actions/runners/jobs
• Repository: /repos/{owner}/{repo}/actions/runners/jobs
• User: /user/actions/runners/jobs

By default, the HTTP API provides the same data as the web UI. However, some endpoints like actions/runners can only return the data of the current scope if desired.

The examples in this post usually use a single scope for brevity.

Monitoring Forgejo’s Job Queue

You can monitor Forgejo’s job queue either using the HTTP API or, once the PR has landed, webhooks. This section will use the HTTP API exclusively. However, all concepts discussed here also apply to webhooks.

A list of all active¹ jobs can be obtained by querying the endpoint actions/runners/jobs which is exposed in each scope.

Each scope “sees” all active jobs of all subordinate scopes. For example, /user/actions/runners/jobs returns all active jobs of each of the user’s repositories. Similarly, /orgs/{org}/actions/runners/jobs returns all active jobs of each of the organization’s repositories.

To get all active jobs that belong to the repository andreas/test, run:

$ curl http://localhost:3000/api/v1/repos/andreas/test/actions/runners/jobs

The result:

[
	{
		"id": 301,
		"attempt": 1,
		"handle": "9d52c7d8-aebe-426b-b015-dd453aacaada",
		"repo_id": 1,
		"owner_id": 1,
		"name": "test",
		"needs": null,
		"runs_on": [
			"debian"
		],
		"task_id": 0,
		"status": "waiting"
	}
]

Each job can be run multiple times. Forgejo increments the attempt number before each attempt, but the id remains fixed.

If you need to uniquely identify a run attempt, use handle². Do not create an identifier yourself by piecing together multiple fields.

Runner Management

Once a job is ready to run, a new runner has to be created to run the job. That again can be done with the help of Forgejo’s HTTP API.

Forgejo offers the following endpoints for managing runners:

• List all runners: GET /repos/{owner}/{repo}/actions/runners
• Register a runner: POST /repos/{owner}/{repo}/actions/runners
• Get a particular runner: /repos/{owner}/{repo}/actions/runners/{runner_id}
• Delete a particular runner: DELETE /repos/{owner}/{repo}/actions/runners/{runner_id}

The same set of endpoints is available in all other scopes.

To create a new runner, send a POST request to /repos/{owner}/{repo}/actions/runners or one of its equivalents in the other scopes:

$ curl -X POST \
	-H "Content-Type: application/json" \
	-d '{"name":"0CEY3JWJ6A4ZK","ephemeral":true}' \
	http://localhost:3000/api/v1/repos/andreas/test/actions/runners
{"id":5,"uuid":"d4637a60-7fea-4075-a81c-7e4ae01bdf0d","token":"e1f3d26e034687a2b6856c4c6c3d87a469ff3bea"}

You receive the runner’s ID, which is required to use one of the runner-specific endpoints, the runner’s UUID, and its token. You have to supply the UUID and the runner token to forgejo-runner so that it can authenticate itself to Forgejo.

Persistent and Ephemeral Runners

Forgejo distinguishes two types of runners: ephemeral and persistent (the default). Persistent runners can run multiple jobs during their lifetime, whereas ephemeral runners can run at most one job before they get deleted by Forgejo.

If possible, use ephemeral runners in autoscaling scenarios, especially when using Forgejo Runner in host mode. In host mode, the job can get hold of the runner token. With an exfiltrated persistent runner token, an attacker could request additional jobs and, depending on the runner’s scope, access secrets and other sensitive resources of other repositories. With an ephemeral runner token, an attacker can only receive the same job that was already compromised. That means that the damage would be limited to a single repository.

Running Jobs

Forgejo Runner offers two separate commands for running jobs: one-job, and daemon. While daemon runs jobs until it is shut down, one-job does what it says on the tin: it runs exactly one job before it quits.

one-job is purpose-built for autoscaling and should be used whenever possible. It is also the only command that supports ephemeral mode; daemon refuses to work in ephemeral mode.

A typical usage of forgejo-runner one-job looks as follows:

$ forgejo-runner one-job \
	--url http://localhost:3000/ \
	--uuid d4637a60-7fea-4075-a81c-7e4ae01bdf0d \
	--token-url file:/path/to/runner-token \
	--label debian:docker://node:lts \
	--label gpu:docker://node:lts \
	--handle 33ba7d51-59c6-44f8-9d2b-1b94f4033973 \
	--wait

--url, --uuid, and --token-url define the connection to Forgejo, whereas --label defines the labels this runner instance supports. For --handle and --wait, see the next sections.

While it might be handy to use options in many use cases, it is not required. All options except --handle and --wait can also be configured in the runner configuration:

server:
  connections:
    forgejo:
      url: http://localhost:3000/
      uuid: d4637a60-7fea-4075-a81c-7e4ae01bdf0d
      token: e1f3d26e034687a2b6856c4c6c3d87a469ff3bea
      labels:
        - debian:docker://node:lts
        - gpu:docker://node:lts

Together with the runner configuration stored in runner-config.yaml, the invocation above becomes:

$ forgejo-runner -c runner-config.yaml one-job \
	--handle 33ba7d51-59c6-44f8-9d2b-1b94f4033973 \
	--wait

What the Handle is Good For

--handle instructs Forgejo to return the run attempt identified by the handle – and only that run attempt. If the run attempt is not ready to run or already being executed by another runner, Forgejo will not return any other job, even if there are others waiting.

Its purpose is to prevent race conditions, and to simplify the implementation of autoscalers.

As an example for a race condition, consider two waiting jobs: A with the label set [debian], and B with the label set [debian, gpu]. In response to those waiting jobs, two Forgejo Runner instances are started: 1 with the label set [debian] (for A), and 2 with the label set [debian, gpu] (for B). If, for some reason, 2 asks Forgejo first for the next pending job, it will receive A because the label set of 1 is a superset of the label set of A. Once 1 is ready and asks for a waiting job, it cannot run B because its label set does not contain gpu.

If both Forgejo Runner instances were to use --handle, they would both receive the intended job. Crisis averted!

There are more potential concurrency issues that can only be avoided if the workload manager or autoscaler knows which runner is executing a particular job. Without --handle, that would be much harder and require additional APIs in Forgejo.

Waiting Is Unavoidable (for Now)

--wait causes forgejo-runner one-job to ask Forgejo for a job until it receives one. Unfortunately, --wait must be used for now because not all jobs with the status waiting can actually be run. See the section about Unambiguous Job Statuses for details.

Missing Pieces

While the building blocks I have discussed above are a good start, there are still multiple pieces missing.

Webhooks

While polling Forgejo’s HTTP API for jobs that are ready to run is the most reliable mechanism to monitor its job queue, it is not particularly efficient. Operators of larger Forgejo instances like Codeberg would certainly be happy if you would not poll it every few seconds. A well established, more efficient mechanism would be webhooks.

Fortunately, there is a pull request for adding webhooks that are activated by workflow run and job status changes that is close to the finish line.

Runner Removal

If you delete a runner in Forgejo, it will only be marked as deleted, not actually removed from the database. It would make sense if it were used for undo, but it is not. The deleted runners linger forever in Forgejo’s database for no particular reason.

Expect that this soft-deletion will be removed as part of the gradual rollout of foreign keys.

Unambiguous Job Statuses

As of Forgejo 15, it is impossible to determine from the outside whether a job is ready to run. The problem is that a job that is currently waiting might still be blocked by a concurrency group.

There is a proposal to introduce a new status called ready which would signal that a job is ready to run, no exceptions. It would be augmented by a secondary status that would provide additional details.

An Access Token Scope for Managing Runners, Only

In Forgejo, access token scopes are relatively broad. For example, you can select whether a token can only read a repository or have full control over it. There is nothing in between. That means that even if a token is only used to manage runners, it has full administrative privileges over the respective scope. That is unnecessarily risky.

There are ideas swirling around about introducing finer grained scopes, for example, write:repository:runners, but nothing definitive yet.

Organization Tokens

Forgejo only has personal access tokens, but no organization tokens. That, paired with the lack of a token scope for managing runners, means that managing runners using the HTTP API is inconvenient and unnecessarily risky.

Monitoring

There is currently no mechanism to monitor any aspects of Forgejo Actions, like the length of the job queue. There is an open feature request for adding a metrics endpoint. Please tell us what you need.

Rejected Ideas

JIT Configuration

When creating a self-hosted runner, GitHub provides the option to generate a just-in-time configuration, or JIT configuration in short. The JIT configuration is Base64-encoded JSON document that contains the runner configuration and can be passed to GitHub Actions Runner ./run.sh with the option --jitconfig. That avoids the separate configuration step with ./config.sh.

We discussed the idea to create a similar mechanism for Forgejo Runner, but ultimately abandoned it because it would tie Forgejo Runner to Forgejo even more. Furthermore, Forgejo has no knowledge of Forgejo Runner’s label configuration (think debian-latest:docker://node:lts-trixie) which is significantly more involved that GitHub’s.

Kubernetes-native Runners

Forgejo Runner usually runs each job in a separate container that it spawns. Optionally, it creates additional containers to provide services like a PostgreSQL database. That does not work well with Kubernetes, because it requires nesting containers inside containers. Couldn’t Forgejo Runner instead delegate the creation and management of containers to Kubernetes?

While that might indeed be possible, it is unlikely that the capability will be incorporated into Forgejo Runner itself. Forgejo Runner already requires fairly exotic knowledge to work on, and deep knowledge of Kubernetes would narrow the pool of potential maintainers even more. Instead, we are currently investigating the addition of a plug-in interface for back-ends. That would not only enable Kubernetes-native runners, but also using lightweight virtual machines instead of containers for running jobs.

Prototypes of a Kubernetes-native runner and a back-end based on Firecracker already exist and can be tried out.

Recipes

Mixing Persistent and Ephemeral Runners

It is possible to use persistent and ephemeral runners in a single scope without undermining --handle and causing concurrency issues. The key is to use disjoint label sets for ephemeral and persistent runners. For example, debian could activate a persistent runner whereas debian-ephemeral could request an ephemeral runner. It requires that the workload manager or autoscaler ignores debian.

You could also use multiple labels, for example, [debian, ephemeral] (for an ephemeral runner) and [debian] (for a persistent runner). While that might look attractive from the onset, it is tricky to use correctly. The first label that Forgejo Runner encounters defines the execution environment that the job will be executed in. That means that [debian, ephemeral] will be executed in debian, whereas [ephemeral, debian] will be executed in ephemeral. Teaching the workload manager or autoscaler to pass all labels to Forgejo Runner with identical mappings will make it work in any case:

$ forgejo-runner one-job \
	--label debian:docker://node:lts \
	--label ephemeral:docker://node:lts \
	...

Pre-Warmed Runners

Pre-warmed runners are typically virtual machines that are waiting for a job to run. That is supposed to reduce latency. The problem is that without a job to run, there is no handle. The best you can do in this case is to start the virtual machine, but not Forgejo Runner. Only start Forgejo Runner once a job is ready to run.

If that is not an option, see Mixing Persistent and Ephemeral Runners.