Repository navigation
gateway tests: timing assertions hold under heavy host load - #127
Merged
Merged
Conversation
Wall-clock assertions failed at load averages of 60-230 without any code getting slower. Each fragile one now asserts something load cannot fake or break: - D217 merge test: thread CPU time (rustix clock_gettime, dev-dep; the crate forbids unsafe) and scaling, 4x the messages < 8x the CPU, in place of < 2 s wall. The reintroduced quadratic merge reads 15.7x. - deadline, breaker window, cache TTL, key cooldown unit tests: measured elapsed time, a stepped clock, or re-running a run that a stall made inconclusive, in place of fixed sleeps with an implicit upper bound. - streaming claims: the scripted provider writes the next event only once the client holds the last one (new Step::Until), so the ordering is the proof rather than inter-arrival gaps. - request deadlines, header stall, dead h2 PING, early answer, large-body failover: the bound is pushed far below the alternative ending, which is pushed far out (a trickled body that lasts over a minute), and the cause is asserted from the answer, row, metric or drain log line. - SEC-17 deny bound: 2 s on an idle host, stretched by 40 round trips measured in the same run when the host is loaded (common::stretched). - rate-limit and log-cap floods: bursts re-sent until one provably landed inside the windows the assertion needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk
jaredLunde
enabled auto-merge (squash)
October 4, 2026 19:52
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wall-clock assertions failed under heavy host load (load 60–230 during local mutation runs) without any code being slower — CI noise, and worse, false "caught" verdicts in mutation runs. Tests only; one dev-dependency feature (
rustix/time, already in the graph) for a safe thread-CPU-time read underunsafe_code = "forbid".D217 linear merge: now thread CPU time at N and 4N (min of 3 runs each), asserting ratio < 8. Unloaded ≈ 4.0; the old quadratic merge put back gives 15.7 and fails.
18 fragile timing checks converted (event/ordering waits, hand-stepped clocks, re-run when a stall makes a run prove nothing, bounds stretched from a round-trip measured in the same run —
common::stretched, exactly the old 2 s on an idle host), including two rate-limit tests the load runs exposed (requests spread across window boundaries). NewStep::Untilintests/commonmakes scripted providers write the next event only after the client holds the previous one, replacing timing gaps in the streaming claims. Safe checks (generous "didn't hang" bounds, lower bounds only, virtual or injected time) left as is.Verification: 160 busy-loop processes (load 105–168) — 5 full loaded runs green over lib + claims_streaming, request_deadlines, reliability_lifecycle, claims_security, replicas, reliability_log_stall, large_bodies, reliability_early_response, reliability_breaker; full
cargo test -p beyond-aigreen unloaded (71 binaries). Every converted test re-checked against a hand-broken subject (quadratic merge, early/late cooldowns, deadline checks off, drain never exits, no keep-alives, buffered SSE, reject cap off, rate limit off, deny 3 s late): all fail.Side finding (not changed):
ai_session_pinned_totalrises on essentially every catalog walk, soreliability_breaker::an_undecidable_first_kib…'s assertion on it is weak regardless of load.🤖 Generated with Claude Code
https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk