gateway: allowance stays fail-closed on a partial snapshot; atomic snapshot rebuild (D257)

[jaredLunde]

Summary

D257 (Reliability/Security, med). rebuild_snapshot removed the file, appended Puts, and wrote the cursor last, all in place. A crash midway left some Puts and no cursor. The next boot with NATS unreachable installed that file as a ready allowance-set holding only part of the exhausted tenants: the rest were served, and /readyz returned 200.

• Allowance policy, set through the trait: WatchedSet::SEED_FROM_CURSORLESS_SNAPSHOT. Deny (fail-open) still seeds from any snapshot. Allowance (fail-closed) seeds only from a snapshot that has a cursor record, and this is decided before the slot or the ready gauge is touched. Only a revision > 0 cursor skips the scan on connect, same as before.
  • Deviation from the brief: the brief gated on is_resumable. That made an edge gateway with an empty allowance bucket (empty scan, cursor revision 0) unready with NATS down, and the existing e2e on_disk_snapshot_enforces_across_restart_without_nats failed. Every writer puts the cursor after the data it covers, so a cursor record of any revision proves the read was complete.
• Atomic rewrite for every set: the rebuild writes {path}.tmp, fsyncs it, renames it over path, fsyncs the directory, and reopens path for appends. It removes any stale .tmp first, and any failure (removal or rename) aborts to snapshot-less mode.
• beyond-slipstream 0.1.0 → 0.8.0: same v2 on-disk format and API-compatible for us. 0.8 adds no public atomic rebuild API (its compact_to_file is private), so the temp-file-and-rename lives here.

Test plan

• reliability_snapshot.rs (hermetic, NATS on a closed port):
  • cursorless allowance snapshot → 503 and /readyz 503;
  • with a cursor → 402 for the exhausted tenant, 200 for others, ready;
  • cursorless deny snapshot still returns 403.
• Unit tests:
  • a crash between staging and rename → load() returns the previous complete snapshot with its cursor;
  • a stale .tmp is discarded, and the reopened writer appends to the renamed file.
• With the allowance policy flipped to true, the fail-closed test fails (402 served).
• cargo nextest run -p beyond-ai: 1093/1093 passed. cargo clippy --all-targets -D warnings, cargo fmt --check, dprint check and verify gate (static) all pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk