Repository navigation
ci: cache the Go gate's lint and libvips, and bound PR cache churn - #242
Conversation
The "Go lint (router recovery, full tree)" job was the CI critical path at ~19.6 min. Three changes cut it down. It never restored golangci-lint's analysis cache, so `make lint-router-recovery` re-analyzed the whole tree from scratch every run (~14 min). The go-lint job runs the identical command over a restored cache in under a minute, so this job now restores that cache too, including the entries go-lint writes. It also carried three Postgres tests with no database service and no SILO_TEST_DATABASE_URL, so every one skipped while still paying its compile time on the critical path. They move to a new parallel go-integration job with a pgvector service and a migration step, so they actually run. The job now uses the repo's composite setup-go action, which restores a per-commit build cache, instead of a bare setup-go that restored only its own stale go.sum-keyed cache.
Every GitHub-hosted Go job installs libvips-dev from apt: an apt-get update plus a ~108 MB dependency tree, about three minutes. It also occasionally stalls on a slow mirror; one run sat on that step for over twenty minutes. The composite setup-go action now caches the downloaded .deb files per runner image. A warm cache installs them offline with --no-download, so there is no apt-get update and no downloads. The key is the runner image, not the commit, and the cache is saved only on a push to main, where PRs can restore it. The private runner image still short-circuits on pkg-config. The changed-lines lint job stops hand-rolling its own apt install and uses the composite action, so it gets the same cached packages and build cache.
fcee0da to
a4d221e
Compare
The composite action restored the build and module cache and then saved a per-PR copy: 0.6-1.4 GB per lane, per run. Two commits of one PR left about 10 GB of PR-scoped caches, filling GitHub's 10 GB budget. GitHub evicted the small caches first, including the golangci-lint analysis cache, so the lint lanes ran cold: router-recovery took about 9 minutes in both jobs even with go-lint's build cache warm. PR runs now restore the build and module cache without saving it; only a push to main saves. The golangci-lint analysis caches follow the same rule, so every PR restores one shared cache instead of saving a copy no other PR can read. A new prune job on main keeps only the newest cache per lane, bounding main's own per-commit saves.
randrini
changed the title
randrini
changed the title
The composite's libvips install ran apt-get update and then install. On one run the update stalled fetching package indexes from the mirrors for 22 minutes (azure.archive.ubuntu.com was ignored and archive.ubuntu.com was slow), wedging the job. The runner image ships package lists, so try the install first and only run the update if it cannot resolve. Add Acquire timeouts and retries so a stalled mirror fails fast instead of holding the job open, and skip the translation indexes.
Review — COMMENT (approve after 1 real fix)Good work on a real quota failure. All 10 checks green (incl. new Must-fix1. High — prune script crashes on any cache outside its lanes. The jq filter uses Should-fix (non-blocking but worth it)2. Stale comment re-introduced at 3. Verified correct
Body is complete. No security surface beyond the standard ephemeral test password. Gate: fix the prune |
The prune jq used capture() without a guard, so a cache whose key matched no lane (golangci-lint-binary-, setup-go-, node-cache-, docker) aborted the whole map and the job pruned nothing. Skip non-lane keys before the capture. Delete through the REST API instead of `gh cache delete`, whose bare-number argument has parsed as a key on some gh versions. Refresh the stale "three jobs" comment now that the gate has four lanes.
|
Addressed in 817d67a.
|
Final: APPROVE — ready to mergeVerified |
Two cold-path stalls hit the libvips install: apt-get update sat on a slow mirror for 22 minutes, and an install-first attempt against a mirror returning 502s spent 16 minutes retrying per package (Acquire::Retries=3 multiplied the per-package retries). Drop the install-first shortcut, let the update refresh the runner's mirror selection, and bound the fetches with Retries=1 and 20s timeouts. The .deb cache used a restore-only step on PRs, so a PR could never warm it and every run paid the flaky apt path. Use a single actions/cache step: a run that restores the entry does not re-save, so it only writes when the shared entry is missing, which lets a PR warm it before main ever runs.
|
One more change from watching the run. The cold apt path stalled twice: Also switched the deb cache to a single |
The deb cache never saved, so every run started cold and paid the flaky apt path. sudo apt-get creates lock/ and partial/ in the archive dir as root, and the cache step runs as the runner user, so tar failed with "Cannot open: Permission denied" and the save was skipped. chown the archive dir back to the runner user on exit, including the early cache-hit exit, so the save succeeds and later runs install offline.
|
Found why the deb cache never helped: it never saved. |
Re-review of
|
Address the re-review: - The .deb cache goes back to restore-only on PRs and save-on-main, so a cold key cannot be raced into a duplicate save by every job in a PR run. Now that the save bug is fixed, main warms it and PRs restore it. - The composite header now says the warm path skips apt while the cold path runs the bounded update and install. - The chown trap emits a warning instead of swallowing a failure, so a broken save is visible rather than silently re-emptying the cache.
|
Addressed in 2afd820:
Gate evidence:
|
Final re-review of
|
2 participants
Problem
Related issue: N/A
Validation tasks: none
This supersedes #241, which is closed: this branch is stacked on it, so its lint-cache and DB-job changes are included here.
The Go gate had three cache and setup problems.
The full-tree router-recovery lint never restored golangci-lint's analysis cache, so
make lint-router-recoveryre-analyzed the whole tree from scratch every run (~14 min) — the job was the CI critical path at ~19.6 min. It also ran three Postgres tests with no database service and noSILO_TEST_DATABASE_URL, so all three skipped while still paying their compile time on the critical path.Every GitHub-hosted Go job installs
libvips-devfrom apt each run: anapt-get updateplus a ~108 MB dependency tree, about three minutes, and it occasionally stalls on a slow mirror (one run sat on it for over twenty minutes).The composite action then saved a per-PR copy of its build and module cache: 0.6–1.4 GB per lane, per run. Two commits of one PR left ~10 GB of PR-scoped caches, filling GitHub's 10 GB budget. GitHub evicted the small caches first, including the analysis cache, so the lint lanes ran cold even with a warm build cache.
Approach
The three Postgres tests move to a new
go-integrationjob: apgvector/pgvector:pg18service,SILO_TEST_DATABASE_URL, ago run ./cmd/silo/ --migrate-onlystep, then the tests, in parallel with the rest of the gate.lint-router-recoverykeeps the full-tree lint, restores the golangci-lint analysis cache, and uses the composite setup action.The composite action caches the apt
.debfiles per runner image and installs them offline withapt-get --no-downloadon a warm cache. On a cold cache it tries the install beforeapt-get update— the runner image ships package lists — and bounds the update with Acquire timeouts and retries, because a stalled mirror once held that step for 22 minutes. The changed-lines lint job drops its hand-rolled apt install and uses the composite.PR runs now restore the build and module cache without saving it; only a push to main saves. The golangci-lint analysis caches follow the same rule, so every PR restores one shared cache instead of saving a copy no other PR can read. A new
prune-go-cachesjob on main keeps only the newest cache per lane.Validation
actionlintonci.yml: clean; both edited YAML files parse.go-integrationsteps locally in the repo's test image (Go 1.26.8 + libvips) against a freshpgvector/pgvector:pg18: migrations applied 522/522, and all three tests passed (internal/adminjob,internal/recommendations,cmd/silo).ubuntu:24.04container: populated the cache with the real apt command (285 packages, 108 MB), then installed from it in a fresh container with--network noneand--no-download; headers present.Risks
CI scheduling and cache policy only. The first run after this lands is cold and populates the main-scoped caches; later PR runs restore them. If GitHub bumps the runner image, the libvips key changes and the packages reinstall once.
Checklist
AI Disclosure
ghCLI and Docker