fix(maintenance): make disk-cleanup CI-aware and stack-safe - #405
Merged
Merged
Conversation
nSelf staging hit 100% disk on 2026-09-11 while nself-disk-cleanup.timer ran daily: DiskCleanup() only pruned docker (unsafely, with -af) and vacuumed 7-day-old journal logs, neither of which touches what actually grows on a runner farm (runner _work job dirs, go-build/module caches, pnpm stores). A wrapper's global `pgrep -f Runner.Worker` busy check also meant one busy runner out of four skipped cleanup for all four. - Reclaim GitHub Actions runner job workspaces under "<root>/_work", discovered via systemd unit introspection with a glob fallback and an env var override (NSELF_MAINTENANCE_RUNNER_ROOTS). _actions/_tool/_temp/ _PipelineMapping are never touched — deleting _actions under a live job is what caused the incident. - Reclaim go/pkg/mod, pnpm-store, and ~/.cache (minus go-build/grype/ trivy, which a running compile or scan reads live). - Replace `docker system prune -af --volumes=false` with dangling-image prune, build-cache prune, and anonymous-volume-only prune (64-hex names via filterAnonymousVolumes) — this can no longer remove an image a stopped stack container depends on, unlike the command that caused the 3-day ntask Hasura outage. - Detect busy state per runner root (worker process path contains that runner's own directory) instead of globally. - Add a disk-pressure threshold (default 85%, overridable) above which the idle-preferred shared-cache tier runs regardless of busy state — runner job workspaces stay idle-gated at every tier since deleting one breaks its job outright. - CleanupResult now carries BytesReclaimed/Reclaimed/Skipped so a timer run is diagnosable, plus a DryRun mode via DiskCleanupDryRun(). - Keep the Windows build green with a parallel, guarded implementation that gets the same docker-prune fix; runner-farm reclaim is POSIX-only.
A per-runner busy check is necessary but not sufficient: the check and the
RemoveAll are not atomic. Cleaning workspaces on runners that reported idle
destroyed three live jobs on nSelf staging on 2026-09-11, which failed with
'Directory .../_work/web/web does not exist' — jobs had started in the window
between the scan and the removal.
A running job writes into its workspace constantly, which is the signal a
process scan cannot give. Require a workspace to have gone untouched for
workspaceStaleAfter (30m) before removing it. A merely slow job still touches
its files; a finished one cannot. Existing tests build fresh fixtures and so
opt out via withoutStalenessGuard.
Also drops DefaultPressureThreshold from 85 to 75. That number has to keep a
different check satisfied: 'nself doctor --deep' fails the host at 80% used
('Disk free: /: N% free (<20%)'). Escalating at 85 leaves a band where cleanup
is content and doctor is red, which is the state staging was in that day — the
dogfood gate failing on disk while the cleanup timer reported nothing to do.
A test pins the two thresholds in the correct order.
golangci-lint fails the build on it: 'const tierAlways is unused'. go vet does not flag unused constants, which is why it passed locally. The always-safe reclaims (docker dangling images and build cache, anonymous volumes, old compressed logs, journald history) genuinely have no ReclaimEntry records to tag. Those commands report their own freed space through CleanupResult's DockerPruneOut / LogRotationOut / JournalVacuumOut and give no per-path byte attribution, so synthesising zero-byte entries purely to carry a tier label would make the reclaimed list read as though nothing was freed. Keeps the tier documented in prose and says why it has no constant.
The glob fallback covered /opt/actions-runner*, /home/*/actions-runner* and /home/*/*/actions-runner*. nSelf staging installs the runner serving the web repo at /home/runner/github-runner, which matches none of them. That install held 5.1G, 2.3G of it job workspaces, and was invisible to cleanup: not a candidate at any tier, under any pressure, because discovery never returned it. The box filled to 100% on 2026-09-11 with that space sitting unreclaimable. Adds the github-runner naming in both /home and /opt, and a test that pins every convention actually in use. A runner discovery never returns is a runner cleanup can never reclaim, so the test asserts the glob set matches each real install path rather than asserting the list's contents.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nSelf staging (167.235.233.65) hit 100% disk on 2026-09-11 while
nself-disk-cleanup.timerran daily and reclaimed nothing that mattered. Two defects:DiskCleanup()only diddocker system prune -af --volumes=false+journalctl --vacuum-time=7d— neither touches what actually grows on a CI box (runner_workjob dirs 7.8G,~/.cache12G,~/go/pkg4.7G, pnpm stores 6.8G combined). A wrapper's globalpgrep -f Runner.Workerbusy check also meant one busy runner out of four skipped cleanup for all four runners.docker system prune -afis dangerous on an nself box:-aremoves all unused images, including ones a stopped stack container still depends on — a past run of this exact command deleted ntask's postgres/redis images and caused a 3-day Hasura outage.Changes
actions.runner.*), falling back to a documented glob list, with anNSELF_MAINTENANCE_RUNNER_ROOTSenv override. Only direct children of<root>/_workare candidates;_actions,_tool,_temp,_PipelineMappingare permanently excluded (deleting_actionsunder a live job caused the incident).~/go/pkg/mod,~/pnpm-store,/opt/pnpm-store, and~/.cache/*— exceptgo-build,grype,trivy, which a running compile/scan reads live (deleting these mid-job broke plugins-pro#113/fix(embedded-pg): correct sha256 pin and add CDN fallback for pglite WASM #114 previously).docker system prune -afreplaced with dangling-image prune + build-cache prune + anonymous-volume-only prune (64-hex names viafilterAnonymousVolumes, never a named*_datavolume).pgrep.DiskCleanupOptions.PressureThreshold), above which the idle-preferred shared-cache tier runs regardless of busy state. Runner job workspaces stay idle-gated unconditionally — deleting one breaks that job outright regardless of pressure.CleanupResultnow carriesBytesReclaimed,Reclaimed []ReclaimEntry,Skipped []SkipEntryso a timer run is diagnosable.DiskCleanupDryRun()/DiskCleanupOptions{DryRun: true}reports what would be removed without removing anything.DiskCleanupWithOptionsthat applies the same docker-prune safety fix; runner-farm reclaim is POSIX-only and reported as skipped on Windows rather than attempted.Test plan
gofmt -l internal/maintenance/— emptymake vet— cleango test ./internal/maintenance/... -v— 12/12 pass, covering: protected runner subdirs preserved, go-build/grype/trivy preserved (including under pressure escalation), per-runner busy detection cleans the idle runner and skips the busy one, pressure escalation triggers above threshold, dry-run removes nothing, anonymous-volume filtering excludes named volumesGOOS=windows go build ./internal/maintenance/...— cleango build ./...— cleant.TempDir()fixtures and injectable hooks (listRunnerWorkerProcesses,dockerReclaimFunc,logRotationFunc,journalVacuumFunc) — no real docker daemon, runner, or system paths are touched.