Skip to content

gateway: allowance stays fail-closed on a partial snapshot; atomic snapshot rebuild (D257) #119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jaredLunde merged 2 commits into from
Oct 3, 2026
Merged

gateway: allowance stays fail-closed on a partial snapshot; atomic snapshot rebuild (D257) #119

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
320db0e
gateway: allowance stays fail-closed on a cursorless snapshot; atomic…
jaredLunde Oct 3, 2026
5300210
verify: D257 fixed (320db0e), tagged on REL-5 and REL-13
jaredLunde Oct 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 16 additions & 1 deletion crates/gateway/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,21 @@ snapshot exists so _enforcement_ survives a cold start before NATS reconnects. C
enforcement — "captured nothing for the first few seconds after a restart" is a non-event — so
`Capture::snapshot_path` returns `None`.

**A snapshot seeds a set by its fail direction** (`WatchedSet::SEED_FROM_CURSORLESS_SNAPSHOT`).
Every writer puts the cursor after the data it covers, so a snapshot with a cursor record (even the
revision 0 of an empty scan) is a complete read, and one without is not. Deny (fail-open) installs
whatever a snapshot holds, cursor or not, because any surviving deny entry is strictly better than
the empty set. Allowance (fail-closed) installs a snapshot only when it has a cursor; otherwise it
stays unseeded (503, `/readyz` 503) until a complete snapshot or a NATS scan, so a partial file can
never flip it ready while missing exhausted tenants (D257). Skipping the scan on connect is a
separate test for every set: only an `is_resumable` cursor (revision > 0) does that.

**The rebuild after a scan is atomic.** `rebuild_snapshot` writes every entry and then the cursor to
`{path}.tmp`, fsyncs it, renames it over `path`, fsyncs the directory, and reopens `path` for
appends. A crash leaves either the old complete file or the new complete one, never `Put`s without
their cursor. A stale `{path}.tmp` from an earlier crash is removed first, and any failure (removal,
rename) aborts the rebuild so the gateway runs snapshot-less rather than append to a stale log.

Deny is **fail-open** on an unread store (empty map = allow). Allowance is **fail-closed** until
`from_entries` has run (even on an empty scan): managed traffic gets a retryable `503` with
`ai_rejections_total{reason="allowance_unavailable"}`. After seed, a NATS blip keeps the last-known
Expand Down Expand Up @@ -2833,7 +2848,7 @@ already gone gets nothing, and a response that has started cannot be replaced.

| Failure | What Actually Happens | Recovery |
| ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| NATS unreachable at boot | Deny-set starts empty (fail-open). Allowance is unread → managed traffic gets a retryable 503 (`Retry-After: 5`), fail-closed. Auth still works — keys from config. | Watcher reconnects; seeds from NATS or disk snapshot on connect. Allowance flips remaining-ok once `from_entries` runs (empty scan included). Until then `/readyz` is 503 so the pod receives no traffic. |
| NATS unreachable at boot | Deny-set starts empty (fail-open), or from its snapshot when `snapshot_path` is set (cursor or not). Allowance is unread unless `{snapshot_path}.allowance` carries a cursor → managed traffic gets a retryable 503 (`Retry-After: 5`), fail-closed. Auth still works — keys from config. | Watcher reconnects; seeds from NATS or disk snapshot on connect. Allowance flips remaining-ok once `from_entries` runs (empty scan included). Until then `/readyz` is 503 so the pod receives no traffic. |
| NATS disconnects mid-run | Last-known deny-set and allowance-set stay active. New entries not applied until reconnect. Unready allowance (never seeded) keeps refusing managed traffic with a retryable 503. | Watcher reconnects (1s→30s exponential backoff, reset only after a watch that ran ≥30s — _connecting_ is not success, or a reachable NATS with a broken watch loops at 1 Hz forever) and resumes from the saved revision. Rescans instead when the seed found no entries, since revision 0 is not resumable — see above. |
| NATS history compacted past snapshot cursor | `CursorExpired` → full re-scan from current NATS state. | After re-scan, new cursor set; delta watch resumes normally. |
| Virtual key tampered or forged | Prefix matches (`bai_v1`) and Ed25519 verify fails → **401**, never BYO. No billing event. The error does not name which part of the token failed. | Client retries with a valid key. A public listener that forwarded a failed verify as BYO would open junk-auth egress while the rate guard had already exempted the token from the BYO aggregate. |
Expand Down
2 changes: 1 addition & 1 deletion crates/gateway/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ workspace = true
[dependencies]
# slipstream is published — consume it from crates.io, aliased to `store` so the code's
# `use store::...` is unchanged. No path deps into the `beyond` repo: this crate builds standalone.
store = { package = "beyond-slipstream", version = "0.1.0" }
store = { package = "beyond-slipstream", version = "0.8.0" }

# The provider table (`route`'s `KNOWN_PROVIDERS`/`AuthScheme`/`Dialect`), shared with `crates/agent`
# so the gateway's proxy routing and the agent's direct routing can never disagree about a provider's
Expand Down
Loading
Loading