forgejo-ephemeral-runner

A Kubernetes operator that runs Forgejo Actions jobs on throwaway runners. Every queued job gets its own runner, which registers, executes exactly that one job, and disappears. Nothing is reused between jobs.

A Runner is a pool: it watches the queue, and for every job it registers a runner with Forgejo, spawns the thing that runs it, and gives everything back once the job is reported. A RunnerInstance is the record of one such runner -- what it registered, and what became of it. The pool writes those records; nothing else does, and there is no second controller behind them.

Two kinds of runner are supported, chosen per resource:

• pod -- a batch/v1 Job. The job image is independent of the runner: an init container copies the forgejo-runner binary out of spec.runnerImage -- or the operator's own default, set chart-wide -- into an emptyDir that the job container runs it from, so jobs can use any image at all.
• vm -- a KubeVirt VirtualMachine booting a prepared guest image. This is how non-Linux jobs (OpenBSD, FreeBSD) get real hardware-level isolation; vms/ holds the image builders.

How it works

namespace of the Runner                              spec.namespace
  Runner       --owns-->  RunnerInstance     Job / VirtualMachine
  token Secret            (the record)        credentials Secret
     |
     +-- registers / deregisters with Forgejo, creates and deletes all of it

The Runner polls GET /api/v1/admin/actions/runners/jobs for jobs whose runs-on labels it serves. Forgejo has no webhook for "job queued" and no queue-depth metric, so polling is the only option.

Every runner is the same thing -- it registers, takes one job and exits -- so the decision is arithmetic. Count the runners that can still take a job, and create enough to cover the queue plus minRunners spare, without passing maxRunners:

spawn = ((waiting + minRunners) - available).min(maxRunners - live)

Runners that have registered but not connected yet count as available, which is what keeps the loop honest: a job polled twice before its runner comes up does not get a second one. minRunners sits on top of demand rather than being consumed by it, so a spare stays ready even while every other runner is busy.

That same poll answers what became of the runners already out there -- Forgejo's own offline, idle and active, plus gone for one it no longer lists. gone is the news the pool is waiting for, because Forgejo deletes an ephemeral runner the moment it reports its one job. Whenever that reading or the child's own ending decides a verdict, the pool asks Forgejo about that one runner again rather than trusting a listing up to a poll old: the difference between a job reported and a runner that died holding it is worth a round trip.

A runner ends on that, on its child exiting, or on jobTimeoutSeconds, which is the backstop for anything wedged -- one left behind by a cancelled job holds its slot until then. However it ends, the pool writes the verdict on the record, deregisters, and deletes the credentials. The child is kept: a finished record and the Job or VirtualMachine beside it are one history entry, so the logs are still there to read. successfulJobsHistoryLimit and failedJobsHistoryLimit say how many of each to keep, newest first, the way a CronJob does.

A runner that fails without ever reporting a job doubles how long the pool waits before making another, up to sixteen minutes at the default interval; one that gets through puts that back to zero. The count lives on the Runner's status rather than being read off the failures still on record, because how many of those there are is failedJobsHistoryLimit's business.

Editing a Runner does not reach the runners it already has out: each record carries the config it was created with and runs on it to the end. So the ones that no longer match are drained and replaced -- all but the one with a job in hand, which is left to finish it. The comparison is between the configs themselves, not the generation, so resizing a pool or changing how much history it keeps throws nothing away.

Registering a runner, recording it, giving it credentials and spawning it are four calls that cannot be made at once, so instead of pretending otherwise the pool sweeps: every pass, anything of its own that no record accounts for -- a child, a credentials Secret, a registration carrying this pool's uid -- is cleaned up. Deleting a record is therefore how a runner is killed by hand.

The child is killed rather than asked to leave: terminationGracePeriodSeconds defaults to 0 for both kinds, where Kubernetes would wait 30 seconds and KubeVirt 180 for a graceful shutdown an ephemeral runner has no use for. A template that sets it keeps its own value.

Credentials are minted per runner: the pool registers an ephemeral runner and puts the result in a Secret beside the child. That Secret cannot be owned by anything in the pool's namespace -- it lives wherever the jobs run, and Kubernetes forbids an owner in another namespace -- so the pool deletes it itself, as soon as the runner is judged and the token is spent. The Forgejo token is never one of the things that crosses that boundary: it is read from the Runner's own namespace and stays there.

How the runner is then told what to do differs by kind. A pod gets an explicit one-job --url ... --uuid ... --label ... argv on the binary in the emptyDir, and the Secret holds only the token file it reads. A VM has no such channel -- it just boots -- so the Secret is attached as a CDROM and carries the whole invocation as files:

file	contents
url	Forgejo base URL
uuid	runner uuid from registration
token	runner token from registration
label	one <name>:host label per line

Reading those and running forgejo-runner one-job is the guest image's job, not the controller's. vms/<os>/bin/image.sh bakes it in: -b installs a script that runs on every boot, -F adds the fstab entry that mounts the CDROM, and -f copies in any extra files.

Usage

kubectl create namespace forgejo-runner
kubectl -n forgejo-runner create secret generic forgejo-pat --from-literal=token=<admin-pat>
helm install forgejo-runner-controller charts/forgejo-runner-controller
kubectl apply -f examples/runner-pod.yaml
kubectl -n forgejo-runner get runners
kubectl -n forgejo-runner get runnerinstances

NAME                    PHASE       REASON           AGE
alpine-pod-4kx9vb2m     Pending                      41s
alpine-pod-8mzq2p7d     Pending                      12s
alpine-pod-vt6bcz1k     Completed   RunnerConsumed   6m

A record is Pending until the pool judges it; what a runner is doing in the meantime is Forgejo's to say, and the pool's own status has the pool-wide count. The Job or VirtualMachine of a finished one is still there:

kubectl -n forgejo-runner logs job/alpine-pod-vt6bcz1k

The token must be a Forgejo admin token: the controller registers runners instance-wide and lists queued jobs across all repositories.

See examples/ for a pod-mode and a VM-mode Runner, and the CRDs in charts/forgejo-runner-controller/templates/ for the full field reference.

Upgrading

Records written before the pool became the only controller kept their registration in the status, where this version does not look for one. It cannot judge them, so it retires them instead: their finalizer -- which nothing processes any more -- is taken off, and once the child they were running has finished, the record goes and the sweeps clean up after it. Jobs in flight are left to finish first, so an upgrade does not fail a build.

The registrations of that generation carry no pool marker, so the sweeps will never touch them. Any left offline in Forgejo after such an upgrade are safe to delete by hand.

historyLimit has been replaced by successfulJobsHistoryLimit and failedJobsHistoryLimit; a Runner still setting the old field keeps the new defaults, since an unknown field is ignored.

Development

cd controller
cargo test
cargo clippy --all-targets -- -D warnings
cargo run --bin crdgen -- --update   # regenerate the CRD in the chart
cargo run --bin forgejo-runner-controller

crdgen writes both CRDs into the chart; CI fails if they are out of date.