CAS GC: read-ahead only over listed ref logs; resolve mounts from listed server roots

[pauloschade]

Changelog category (leave one):

• Performance Improvement

Changelog entry:

CAS GC: read-ahead hints only ref-log ids that the round's own LIST returned, so a walk no longer requests the positions past every epoch seal (which never exist). The orphan sweep resolves a namespace's mount from the server roots that the round already listed, instead of probing every prefix of the namespace.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

---

Problem

We run CAS in production on 26.6 Antalya: one writer, two readers, and about 800 tables. The GC leader sends about 2.4 M GETs per day that return 404 under cas/ns/stream/<life>/_log/<epoch>-<seq>.zst. That is 58 % of the leader's GETs, and we pay for each of them in S3 requests and GuardDuty S3 events.

What we measured:

• A median of 63.9 404s per epoch crossing per stream. That is 4 × cas_gc_io_concurrency at the default, which is the read-ahead window.
• 792 streams. 568 of them have ≥ 2 epochs, and only 74 have a _snap.
• 3,876 epochs consist of a single epoch_seal. Recovery closes every table's epoch on every remount, and a table smaller than the snapshot threshold never publishes a _snap, so these epochs accumulate forever. They also pile up above an existing _snap.
• 96 % of the streams belong to live tables.

GcReadAhead documents this waste as bounded: "overshooting by at most one window per epoch crossing". Per walk that is true. But the orphan sweep rebuilds each namespace's protection view from _snap, or from genesis when there is none, on every page. So the cost is window × epochs × visits, and every remount adds an epoch to every table. The cost model in docs/en/antalya/cas/architecture/garbage-collection.md ("What a round costs") does not include it.

There is a second, smaller cost. floorForNamespace reads the mount key of every /-prefix of a namespace, longest first. A namespace like writer/store/913/<uuid>@cas@ costs 2 extra 404s per page (96 k per day for us).

Change

The round already holds both key sets:

• phase 4 lists cas/ns/stream/ once (RefScanSummary::logs_by_life, which is a hint, not a census);
• phase 3 lists gc/server-roots/.

The change uses them. It adds no new state, setting or format.

1. hintRefLogsWithinEpoch(..., listed) stops at the first id that is not in the life's listing. Within an epoch the ids are dense, so an unlisted id either lies past the seal or was written after the LIST. In the second case the walk takes it inline.
   • The intake's lookahead stops the same way, using the RefTableListing that its witnessAbove already reads.
   • The sweep's recovery walk and its committed-tail walk receive the listed ids through recoverRefTableDetailedFromAuthority(..., reader, listed_logs) and activeManifestKeys.
2. Mounts come only from listed server roots. floorForNamespace, prefixEligible and sweepNamespace take the set of server roots the caller listed under gc/server-roots/ as a required argument, and only those prefixes are tried. There is one rule and no "try every prefix" mode.
   • A GC round passes phase 3's own LIST (HeartbeatFloor::listed_srids, the set computeHeartbeatFloor already builds to prune obs). This applies to the orphan sweep page and to the fold's dead-precommit check.
   • Callers outside a round (fsck, rebuild, decommission) take one listServerRoots pass per call, outside their per-manifest and per-prefix loops. listServerRoots is a thin wrapper over the existing read-only listMounts and adds no new parse.
   • A mount created after the caller's listing is seen only on the next listing. This is the conservative direction: "not found" means retain, keep the barrier, or over-protect. It is reachable only after DROP POOL MEMBER.

With hinting bounded by the listing, no id past a seal is ever hinted, so the seal-crossing discards in the recovery and committed-tail walks became dead code and are removed.

The read-ahead contract is unchanged. It caches results, never decisions: every take, decode, counter, hold and event stays at its site. Concurrency 1 hints nothing. A LIST that omits a key costs one inline read, and the decision is the same.

Tests

• CASOrphanSweepRequests.EpochCrossingsWasteNoHintAndReadExactlyTheInlineKeys covers a birth epoch, four seal-only epochs and a 40-log epoch. It asserts that nothing is wasted, that nothing past any seal is read, and that the GET set, outcome and nominations are identical inline and at concurrency 16.
• CASGCArithmeticIntake.ReadAheadAcrossRemountEpochsWastesNothingAndDecidesAsInline runs a full round over six crossings through a lying LIST, at concurrency 1 and 16. Decisions are identical, nothing is wasted, and the read-ahead hits.
• CASOrphanSweepRequests.FloorIsReadOncePerNamespacePerPage (one read instead of three; none when the mount is absent).
• Integration: test_cas_gc_s3::test_gc_read_ahead_wastes_nothing_across_epoch_changes, with 5 restarts (3 with inserts, 2 idle). It asserts CASGCReadAheadWasted == 0 and Hit > 0 before the DROP, and that the dropped table is reclaimed.

On our builder (aarch64, Clang 21.1.8) at the same tree on top of v26.6.4.20001.altinityantalya, the CAS gtests passed 2527/2527 and test_cas_gc_s3 passed 2/2. This branch is the same five commits rebased onto antalya-26.6: two comment and call-site conflicts with the later io_pool/gc_io_concurrency rename were resolved, and the new test uses gc_io_concurrency. It has not been compiled on this base yet, so CI will be the first build.

Production result

We rolled the patch out on 2026-10-11 to all three members, at the default read-ahead concurrency (16). We measured on the GC leader, with one round every 600 s:

	before (stock 26.6.4 Antalya)	after
CASGCReadAheadWasted	11.2 M since process start, against 7.1 M hits	19 → 1 → 0 → 0 per round (hits 2,908 / 2,213 / 1,871 / ~2,670)
404s on reads (leader)	~1,000 per hour, even with read-ahead disabled (1); ~2.4 M per day at the default	2 in a steady-state round
GC decisions	—	unchanged: every round kept deleting (192, 189 and 4 deleted in rounds 2–4)

The 19 in the first round came right after a leadership change, with a backlog (that round deleted 1,698). CASGCReadAheadWasted also counts hints that are legitimately left untaken, such as "a namespace held below its lookahead" or "a HEAD candidate that kept an edge", so we read it together with the 404 count.

Separate point, not in this PR

On every page, the protection view still rebuilds a namespace from genesis when it has no _snap. For us that explains most of the ~956 k successful reads per day. Two options are a cheaper rebuild source (for example, the fold seal's last_folded_ref_id plus a cached view) or letting a seal count toward the snapshot threshold. We would like to discuss which direction you prefer before we write it.

🤖 Generated with Claude Code

[filimonov]

Thanks a lot for the early testing, the detailed report and the PR. The per-epoch numbers and the "window × epochs × visits" cost model are exactly right.

This PR fixes the consequence, and we would rather remove the cause. The cause is that the orphan-manifest sweep replays each namespace from _snap or genesis on every page, even though the fold has already processed everything below its cursor. Bounding the hints shrinks the 404s but keeps the replay (your ~956k successful GETs/day are that replay), and it adds a LIST dependency we are removing (the global cas/ns/stream/ LIST is going away).

The direction we want: the sweep should not replay at all. The fold already extracts +1/-1 manifest edges per transaction (manifestEdgesOfTxn); a durable owned-manifest set per namespace, carried between generations with a cursor, lets the sweep check a key against the set and read only the committed tail. Two constraints: the state must live in new objects under the generation prefix, not in the fold seal (an old server must ignore it and never write it), and a namespace whose set is not current is retained, never deleted from. If you would like to work on that task I can share the draft design.

Two pieces of your PR we will keep: bounding the hint overshoot by prev_epoch_seal from {E, 1} (no LIST needed), and adding the replay term to the GC cost model doc.

If you want to continue, we'd be glad to have you: a call to go over your numbers and the plan, and PRs in that direction. Feedback from early adopters at your scale is what we need most right now.