Make memory guardrail restart verdict reachable via chronological params - #3939
Make memory guardrail restart verdict reachable via chronological params#3939tomerweller wants to merge 2 commits into
Conversation
Add a consumer-side regression that extracts the live two-line eval_memory_guardrail invocation from each monitor-tick SKILL.md (.claude and .agents copies) and evals it with a +600/+600 MB heap growth trajectory (RSS 47 GB / avail 7 GB on a 61 GB host). Asserts the restart verdict fires. Fails on main with report-high-mem because the function binds (prev, curr, prev2) while SKILL.md passes heap args chronologically, inverting the second growth delta. Refs #3846 Co-authored-by: Claude Code <claude-code@anthropic.com>
Reorder eval_memory_guardrail's three heap parameters from (prev, curr, prev2) to chronological (prev2, prev, curr) so the signature matches the order both live monitor-tick SKILL.md call sites already pass. On main the mismatch bound the oldest snapshot to heap_prev_mb and the newest to heap_prev2_mb, so the second growth delta (heap_prev - heap_prev2) was computed oldest-minus-newest — negative for any rising heap — leaving the restart arm dead on genuine OOM trajectories. The delta arithmetic is unchanged (it references variable names), so after the reorder both deltas are positive under growth and the restart tier fires when the RSS/avail thresholds are also met. No SKILL.md edits needed: both copies already pass HEAP_PREV2_MB HEAP_PREV_MB HEAP_CURR_MB. Update the six direct-call test cases' heap args to the chronological order and the signature doc comments to match. Refs #3846 Co-authored-by: Claude Code <claude-code@anthropic.com>
🔍 Reviewer: CorrectnessVerdict: APPROVE Summary: Correct, minimal Option-B fix — the function signature is reordered to chronological Full reviewCycle 1 (no prior Linked issue kind: bug-fix. Test-verification gate → requires a regression test that provably fails at the pre-fix commit. Verified against
CI signal: No blocking concerns. |
🔍 Reviewer: RiskVerdict: APPROVE Summary: Low-risk, well-scoped fix confined to monitor bash scripts (no crate / observable-surface code). It restores a previously-dead operational safety arm (controlled restart before OOM-kill) rather than changing any live behavior that has been firing, and it is guarded by a new consumer-side regression test. Full reviewCycle 1 (no prior Reviewed
No blocking concerns. |
Review: Bounce-Back Cycle 1Reason: CI failed (unrelated, will rebase) Reviewer A (Correctness): APPROVE — Option-B signature reorder aligns the function with both SKILL.md copies; new consumer-side regression test provably fails pre-fix and passes post-fix. Both agent reviewers APPROVE and there are no external change requests, but CI is red on two Quickstart integration probes:
These failures are not diff-attributable. The PR touches only Exit 124 is a wall-clock timeout of the friendbot/RPC/horizon services — infrastructure flakiness independent of the change. The monitor-specific checks ( Routing back to |
Do: Re-review (CI-only bounce)Both agent reviewers already APPROVE'd; the sole blocker on cycle 1 was two unrelated flaky Go quickstart integration jobs ( Actions taken:
Ready for re-review once CI goes green. |
🔍 Reviewer: CorrectnessVerdict: APPROVE Summary: Correct, minimal fix — reorders Full reviewCycle 1 (no prior Root-cause verified [correctness]: With the old binding ( All call sites checked [regression-risk]: grepped the repo — the only invocations are the two live SKILL.md sites ( Test-verification gate (bug-fix) PASSED [test-coverage]: The PR adds a consumer-side regression that greps the live two-line invocation out of each SKILL.md and
Readability/error-handling [none]: The awk extraction captures both continuation lines robustly; the test resets No blocking concerns. |
🔍 Reviewer: RiskVerdict: APPROVE Summary: Scripts-only bash change to a monitor-tick decision helper. No observable/interop surface, no Rust, no data format touched. Behavioral change is strictly a fix (makes the Full reviewCycle 1 (no prior Blast radius [regression-risk]: Only Behavioral delta [correctness]: The only runtime effect is that a genuine OOM trajectory (RSS>0.75·RAM AND avail<0.12·RAM AND two consecutive heap deltas >500 MB) now yields Operational safety [ops]: Making CI note [ci-failure]: The two red checks are Quickstart No blocking concerns. |
Review: Bounce-Back Cycle 1Reason: CI failed (unrelated, will rebase) Reviewer A: APPROVE — correct, minimal fix; regression test provably fails at pre-fix commit and passes post-fix (488/488). Both agent reviewers APPROVE and the fix is verified locally (post-fix 488/488 exit 0; pre-fix regression cases 439/440 fail exit 1). The only red checks are unrelated flaky Quickstart Docker integration jobs:
These spin up external Docker services (friendbot/rpc/horizon) and are unrelated to this 4-line bash reorder. All directly-relevant checks are green: Monitor-Tick Skill Snippets, Script self-tests, Project-Loop Skill Snippets, Clippy, Test, Format. Routing back to |
Do: Re-review (CI-only bounce, cycle 2)No code change required. Both agent reviewers already APPROVE, and there are no inline or actionable review comments since the last push. The sole blocker remains the two flaky Quickstart Docker integration jobs. Actions taken:
Failure diagnosis (unchanged, confirmed from the fresh 09:34 re-run logs): both red jobs are the Quickstart Ready for re-review once CI settles. |
tomerweller
left a comment
There was a problem hiding this comment.
Addressed feedback (CI-only bounce; no code change needed). Re-triggered flaky Quickstart jobs. Ready for re-review.
Review: Bounce-Back Cycle 3Reason: CI failed (unrelated — flaky Quickstart Docker integration timeout, will re-run) Reviewer A (Correctness): APPROVE — clean chronological Both agent reviewers APPROVE on the current head (
Escalation note: this is the 3rd bounce on the current (unchanged) code state, all for the same unrelated flaky-CI reason. Per the head-scoped cap, if the flaky Quickstart recurs on the next review without a fresh CI pass, the PR should be blocked for operator attention (re-run the flaky job, or treat Quickstart as non-required for a scripts-only change). Routing back to |
Do: CI Re-trigger (Mode B, bounce cycle 3)No code changes — both agent reviewers APPROVE the current head and there are zero actionable inline review comments. The sole red signal was the flaky Quickstart Docker integration jobs ( Action taken: re-ran the failed Quickstart jobs via Ready for re-review once CI settles. |
🔍 Reviewer: CorrectnessVerdict: APPROVE Summary: Re-review on unchanged head Full reviewCycle N≥2 — prior
No blocking concerns. Only red signal is unrelated Quickstart CI (see the review decision comment). |
🔍 Reviewer: RiskVerdict: APPROVE Summary: Re-review on unchanged head Full reviewCycle N≥2 — prior
No blocking concerns. Only red signal is unrelated Quickstart CI. |
Review: Bounce-Back Cycle 2Reason: CI failed (unrelated, will rebase) Reviewer A (Correctness): APPROVE — Option-B signature reorder aligns the function with both SKILL.md copies; consumer-side regression provably fails pre-fix and passes post-fix. Both agent reviewers APPROVE and there are no external change requests, but CI is red on two Quickstart integration probes:
These are not diff-attributable. The PR touches only Exit 124 is a friendbot/RPC service timeout — infrastructure flakiness independent of a bash param reorder. The monitor-specific checks ( Note: the cycle-1 bounce was resolved by a CI-only re-trigger (head unchanged at Routing back to |
Do: CI Re-trigger (Mode B, full Quickstart re-run)No code changes — both agent reviewers APPROVE the current head ( The sole red signal remains the two flaky Quickstart Docker integration jobs ( Action taken: Prior re-runs of just the failed jobs kept timing out in the same run context, so I triggered a full re-run of the Quickstart workflow (run Ready for re-review once CI settles. Per the head-scoped bounce cap, if Quickstart is still red on the next |
tomerweller
left a comment
There was a problem hiding this comment.
Full Quickstart workflow re-run triggered (fresh containers) to clear the flaky friendbot/rpc/horizon startup timeouts. No code change — rebase is a no-op and both reviewers already APPROVE. Ready for re-review.
Review: Bounce-Back Cycle 2Reason: CI failed (unrelated, will rebase) Reviewer A (Correctness): APPROVE — 4-line heap-param reorder to chronological The only red checks are the Quickstart shards This is a docker/infra startup flake in the Quickstart harness, entirely unrelated to this PR's diff, which touches only Manual job re-triggers (Quickstart attempt 6) keep hitting the same friendbot flake. Routing back to |
Do: Blocked — CI-only bounce loop, no actionable fixMode B ran with zero actionable inline review comments and no code change to make. Both agent reviewers (Correctness + Risk) APPROVE the current head; the only external review is a neutral non-blocking COMMENT. I am not performing another CI re-trigger, because doing so is what has kept this PR in an infinite loop. Why this is a loop, not progress:
All directly-relevant checks are green: Monitor-Tick Skill Snippets, Script self-tests, Project-Loop Skill Snippets, Clippy, Test, Format, CodeQL, Build, Doctest. Operator action needed (out of
Moving to |
Closes #3846
Summary
The monitor-tick memory guardrail's
restartverdict was unreachable:eval_memory_guardrail's heap parameters were ordered(prev, curr, prev2), but both live monitor-tickSKILL.mdcall sites pass the three heap snapshots chronologically (HEAP_PREV2_MB HEAP_PREV_MB HEAP_CURR_MB). This bound the oldest snapshot toheap_prev_mband the newest toheap_prev2_mb, so the second leak-evidence delta (heap_prev - heap_prev2) was computed oldest-minus-newest — negative for any rising heap — and the-gt 500predicate was never true. On a genuine OOM trajectory the monitor would reportreport-high-memforever and never escalate to a controlled restart.Fix (Option B from the issue): reorder the function's three heap
localbindings to chronological(prev2, prev, curr)so the signature matches what the consumers already pass. The delta arithmetic is byte-identical (it references variable names), so after the reorder both growth deltas are positive under a rising heap and therestarttier fires when the RSS/avail thresholds are also met. Thresholds and the report-only arm are untouched. NoSKILL.mdedits needed.Plan reference
Converged Plan comment
Test plan
Regression test
scripts/test-monitor-skill-snippets.sh— new consumer-side case "eval_memory_guardrail: SKILL.md documented call order fires restart on +600/+600" (runs against both.claude/and.agents/SKILL.md copies)f9d4320— verified FAILED with:got report-high-mem(expectedrestart)30db904The regression is deliberately consumer-side: it greps the live two-line invocation out of each SKILL.md and evals it, because a direct-call test structurally cannot catch a SKILL.md/function argument-order divergence (the #3844 defect class).
Deviations from plan
None.
🤖 Generated with Claude Code