CI: per-run cache rotation writes ~5 GB per full dispatch and exceeds the 10 GB quota

[mobileskyfi]

Current state

Live usage on 2026-07-30 is 11.49 GB across 33 active caches against the repository's 10 GB quota:

active_caches_size_in_bytes=11493386757
active_caches_count=33

GitHub is therefore evicting entries. Cold image/package downloads are now an uncontrolled variable in integration timing, directly confounding #76/#106 and the cold-download headroom formerly tracked by #91.

Cause

#101 correctly fixed the old static-key failure: an exact actions/cache hit skips the post-job save, so newly-resolved channel versions were never persisted.

The replacement over-corrected by keying every run/attempt and splitting integration/examples:

chr-images-${os}-${arch}-v2-${target}-int-${run_id}-${attempt}
chr-images-${os}-${arch}-v2-${target}-ex-${run_id}-${attempt}

A full dispatch writes near-duplicate -int- and -ex- entries for every platform × target. Recent large entries are roughly 400–520 MB each; one all-platform/three-target dispatch writes about 5 GB, half the quota.

Proposed approach

1. Resolve each channel to the concrete RouterOS version before the cache key is chosen, and key durable content by OS/arch/resolved version rather than run id.
2. Avoid duplicate integration/examples writes. They consume the same cache path in parallel, so choose an explicit owner/prewarm scheme instead of two full saved entries.
3. Audit cached contents (.img, extracted images, derived qcow2/package archives) and retain only the expensive source artifacts needed to reconstruct the rest.
4. Add bounded retention/eviction only as a safety net, not as the primary design.
5. Measure a real cold-cache download/package path separately and carry that measurement into CI: retune oversized boot envelopes from path-specific timing data without losing forensic headroom #106's nested timeout budget. Cache hits must improve speed, not be required for correctness.

Done-when

• Repeated full dispatches for unchanged resolved versions do not write a new multi-GB cache set.
• Integration and examples do not persist duplicate full caches for the same OS/arch/version.
• The active cache set stays predictably below quota with documented retention.
• A forced cold-cache run completes or fails with a named infrastructure/download outcome inside its enclosing test/file budget.
• .github/instructions/ci.instructions.md documents the key ownership and why it is safe.

Tracked in Wave 1 of #110. #91 is closed because its fixed test timeout is gone; remaining cold-download budgeting belongs here and in #106.

[mobileskyfi]

Now over the limit — 11.49 GB / 10 GB

Measured 2026-07-30 after run 30507484030:

$ gh api repos/tikoci/quickchr/actions/cache/usage
active_caches=33 size_bytes=11493386757     # 11.49 GB

Filed at 9.08 GB; it has crossed the 10 GB cap, so GitHub is now actively evicting LRU entries — cache misses are no longer hypothetical.

Per-run key rotation confirmed unchanged — the largest entries are all this run's fresh set:

522092590  chr-images-Linux-ARM64-v2-testing-int-30507484030-1
522092530  chr-images-Linux-ARM64-v2-testing-ex-30507484030-1
419214956  chr-images-macOS-ARM64-v2-testing-int-30507484030-1
419214731  chr-images-macOS-ARM64-v2-testing-ex-30507484030-1
419214618  chr-images-macOS-ARM64-v2-testing-int-30479032655-1
402002294  chr-images-Linux-ARM64-v2-long-term-ex-30507484030-1

Note the -int- / -ex- split doubles the write: each platform × target writes two near-identical ~400–520 MB entries per run. One full dispatch is ~5 GB of new cache, i.e. half the quota per run.

Why this now interacts with other issues

Eviction means a leg can lose its image cache and fall back to downloading inside the test step. That lands on the two legs with the least headroom:

• CI: macos-x86 full-suite runner loses communication near 60 min and destroys all evidence (HVF) #76 — macos-x86 dies at ~50 min; a cold image download inside Run integration tests pushes it closer, and there is no evidence trail to tell the two apart.
• CI: provisioning.test.ts fixed 360s timeout can be shorter than the download-retry worst case #91 — provisioning.test.ts's fixed 360 s timeout vs the download-retry worst case is precisely this scenario.

So this is no longer only a housekeeping issue; it is a plausible contributor to slow-platform flakes and should be weighted accordingly.

🤖 Measured by Claude Code while reviewing run 30507484030