Skip to content

ci: investigate sccache with external storage backend (S3/R2) for hosted runners #374

@staging-devin-ai-integration

Description

Context

Our hosted-runner CI jobs get 0% sccache hit rate because sccache falls back to local disk, which is empty on each ephemeral runner. The self-hosted GPU runner achieves 100% hit rate via persistent local disk.

We tried enabling SCCACHE_GHA_ENABLED=true (#372) to use the GitHub Actions cache as sccache's backend, but it creates one cache entry per compiled artifact (~1000 tiny entries per CI run), which evicts the much more valuable Swatinem/rust-cache target/ snapshots via LRU. Reverted in #373.

Research: How large Rust projects handle this

Project sccache backend Notes
Bevy (46K★) None — no sccache on hosted runners Uses actions/cache for target/ only; dedicated update-caches.yml workflow saves cache on main only
rust-analyzer (16K★) None actions/cache for target/ + cargo registry
Deno (106K★) Azure Blob Storage Moved off S3 for cost; does NOT use GHA cache for sccache
Next.js/Turbopack (139K★) Vercel Artifacts API (custom fork) Multi-level disk,vercel_artifacts; self-hosted runners

The sccache GHA backend's per-artifact granularity is a known problem — GitHub staff warned about cache thrashing. A local-aggregation approach was proposed but never merged.

Current state

  • Swatinem/rust-cache handles target/ directory caching (effective)
  • shared-key groups reduce cache entries from 7 to 3
  • save-if: main prevents PR branch cache thrashing
  • sccache is installed but only useful on the self-hosted GPU runner (persistent disk)

Options to explore

  1. Cloudflare R2 bucket as sccache remote backend — free egress, S3-compatible API. Requires provisioning a bucket + access keys as GitHub Actions secrets.
  2. S3 bucket — standard approach, small cost for storage + egress.
  3. Do nothing — current setup is functional; sccache only helps incrementally on top of target/ cache. The main bottleneck (full recompilation) is already addressed by rust-cache.

Expected impact

With an external sccache backend, hosted runner compilation cache hit rate would go from 0% to ~50-60% for unchanged crates, saving ~2-5 minutes per job on incremental builds.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions