One self-hosted runner backs all private-repo CI: bottleneck, flake source, and SPOF

[acamarata]

One runner backs every private-repo CI job in the org

nself-org has exactly one self-hosted runner:

nself-staging-runner   status=online   busy=true

Every private repo that cannot use free GitHub-hosted minutes (web, plugins-pro, and anything else pinned to self-hosted or to the CI_RUNNER variable) queues behind it. Observed on 2026-08-26: ~90 jobs QUEUED simultaneously, none running, for hours.

Three distinct problems, not one

1. Throughput. plugins-pro alone fans out to ~49 jobs (build/test/typecheck across node 18/20/22, plus Rust). web adds ~60 more. A single runner serialises all of it. A PR that should take minutes takes hours, which in practice means people stop waiting for CI before merging.

2. It manufactures flakes. Concurrent jobs on one box contend for the same host resources. Observed directly on web#126:

Auth E2E — unified auth flows
Error response from daemon: failed to set up container networking:
driver failed programming external connectivity

That is a port already bound by a neighbouring job, not a defect in the code under test. A flake of this shape is worse than a plain failure because it trains people to re-run rather than read.

3. Single point of failure. If this box is down, every private-repo PR is unmergeable — the required status checks can never report. That is the same failure mode as the phantom nself-ci required check found in the 2026-08-26 gate sweep, except triggered by hardware rather than config.

It also runs on the staging box

The name says nself-staging. Per the two-box rule in nsentry-server-standard.md, a CI runner has docker.sock access, which is root-on-host, and should not share a machine with anything holding private data or serving traffic. Worth confirming what else that box does.

Options

1. More runners. The cheapest fix for throughput and the SPOF. Even two boxes removes the serialisation cliff and most of the port contention.
2. Reduce fan-out. plugins-pro testing node 18/20/22 for every package is a large matrix; if only one version is actually supported at runtime, the other two are cost with no signal.
3. Move what can be moved back to GitHub-hosted. Public repos already use free minutes. Anything private that does not touch secrets could too.
4. Isolate ports per job. Does not fix throughput, but stops the container-networking class of flake.

1 and 2 are independent and both help. 4 is worth doing regardless, since the flake will recur on any shared runner.

Constraint to respect

Per the standing org rule, paying GitHub for Actions minutes is not an option and should not be proposed. This is about owned hardware and matrix size, not billing.

Found while landing the org-wide CI gate sweep; it is the last thing blocking plugins-pro#70 and web#126.

[acamarata]

Correction and hard data from the box

My original report said "exactly one runner". That was incomplete — it reflected the org-level runner list only. On nself-staging (167.235.233.65) there are two runner services:

actions.runner.nself-org.nself-staging-runner.service       (org-level)
actions.runner.nself-org-web.nself-staging-web.service      (web repo-level)

The more important thing I missed: that same box also runs the entire ntask staging application stack — 31 containers including ntask_postgres, ntask_hasura, ntask_api, ntask_web, ntask_auth, ntask_minio.

Measured state

CPUs	4
Load average	3.65 / 4.98 / 6.03 — sustained above core count
Memory	7 GB total, ~5 GB available
Disk	75 GB, 79% used after cleanup (was 82%)
Running containers	31
Runner services	2

What this means for "add a second runner"

Adding a third runner process to this box would make throughput worse, not better. Load is already ~1.5x core count with two runners; a third would just deepen the queue while adding contention. It is also the mechanism behind the port-collision flakes — concurrent jobs on a box that is simultaneously serving staging.

Docker cleanup recovered only 554 MB. Nearly all images are in use by the 31 running containers, so there is no meaningful headroom to reclaim here.

It also violates the two-box rule

Per nsentry-server-standard.md, a CI runner holds docker.sock (root-on-host) and must not co-locate with an application stack or private data. This box does both today.

So the real options are

1. A separate runner host. The only thing that actually adds concurrency. This is a monthly spend and is explicitly gated by the PPI rule "never provision a third nSelf box — not for CI, not for runners, not for observability — without the user approving it in the same turn." I have not provisioned one: the approval I have was given on the basis of my incorrect "one runner" report, and creating billable infrastructure on a mistaken premise is not something to do silently.
2. Move the staging app stack off this box, leaving it as a dedicated runner host. No new spend, restores the two-box rule, and frees 4 cores from serving 31 containers.
3. Cut the matrix. plugins-pro fans out to ~55 jobs largely from node 18/20/22 across packages. If only one runtime is actually supported, two thirds of that queue is cost with no signal.

Option 2 costs nothing and fixes both the capacity problem and the two-box violation. Worth deciding before spending on option 1.

[acamarata]

Updating with measurements, since the original numbers here were wrong in one direction and the situation has changed in another.

Correction to the original report

It said one runner. There are two runner services, both online and both working:

actions.runner.nself-org-web.nself-staging-web.service    active running
actions.runner.nself-org.nself-staging-runner.service     active running

So this is saturation, not a hang.

What actually loads the box

The staging box (4 cores) is not just a CI runner. It also hosts two full nSelf stacks: nself-web (23 containers) and ntask (10). Load average was measured at 20.79 with ~28 runs queued, and separately at 4.98–6.03 on 4 cores.

Disk was the more urgent half, and is partly fixed

It was at 9% free (/ and /var, 6.7G on a 75G disk), which doctor flags and which took Postgres down once before (2026-08-14). Reclaimed 4.7GB of unreferenced anonymous Docker volumes plus log rotation: now 14G free (81% used). Still under doctor's 20% threshold.

The remaining consumers are all runner caches and all regenerable:

Path	Size
_work	8.8G
.cache/go-build	5.7G
go/pkg/mod	4.9G
pnpm-store	2.8G
.rustup/toolchains	2.2G
.cache/grype/db + .cache/trivy/db	3.3G

Not cleared because both runners were mid-job. This wants a scheduled reaper, not manual intervention — it will recur. That is probably the cheapest real win here and needs no capacity decision.

Still needs a human

The capacity question stands. The PPI restricts nSelf to exactly two servers and forbids paying GitHub, so "add a runner" and "buy minutes" are both off the table by policy. The options that remain are: reap caches on a schedule (do this regardless), move the ɳTask stack off the CI box, or accept the queue depth.

Leaving open. Nothing provisioned.