feat(bench): phase-7 tooling + 4 Python cascade scenarios (#104)#124
Merged
Conversation
PR-A of the cascade expansion (4 -> ~14): the polarization tooling plus the first four new scenarios, each a NEW drift archetype in a NEW domain. Tooling (foundation for the harder PRs): - tools/validate_scenario.py: offline grader-polarization self-test. author.py proves the *drift* is real; this proves the *graders discriminate*. Reuses the live grade_code/grade_qa on author-provided reference solutions in .author/ (solution_correct.* -> ok & not misled; solution_stale.* -> not ok & misled), catching mis-polarized graders before any spend. - scenarios.py: `edit_path` meta key (the visible file the agent returns), so the self-test can build the FILE: block; backfilled into the 4 existing cascades. - scenarios/CHECKLIST.md: the repeatable authoring steps + neutrality rules. - tests/test_scenarios.py: parametrized guard that every scenario with reference solutions has correctly-polarized graders. New scenarios (Python, each oracle-safe: drift detectable by surf, graders probe the real hidden dependency): - cascade-ttl-units-code (T1, cache) units s->ms (lifetime 30s->5s) - cascade-money-rounding-code (T2, currency) rounding precision cents->dimes - cascade-backoff-offbyone-code (T2, retries) attempt indexing 1-based->0-based - cascade-signal-threshold-code (T2, monitoring) alert sign flip rise->drop Note: surf alpha-renames identifiers when hashing, so a drift must change a literal/operator/structure, not just swap a named constant — each scenario's drift is chosen accordingly. Verified offline: author.py seals all four (genuine "changed" divergence), validate_scenario --all green, a mock run over all four is error-free, and uv run pytest passes (35). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Connorrmcd6
added a commit
that referenced
this pull request
Jun 14, 2026
PR-B of the cascade expansion: three TS code cascades (graded by `node --test`, Node >=22.18 type-strip), each a new drift archetype that surf detects via a structural/literal change (not an identifier swap). - cascade-serialize-key-order-ts-code (T2, serialization) — canonicalString key order insertion -> sorted (Object.keys().sort() added); the visible signer must reproduce the canonical string. - cascade-default-timeout-ts-code (T1, timeouts) — totalDeadlineMs default param 30000 -> 5000; the visible caller reports the default budget. - cascade-validate-guard-ts-code (T2, validation) — isValidName dropped its `length > 0` guard (empty names now accepted); the visible intake filter must mirror the rule. Each: graders probe the real hidden dependency for ground truth, task.md is neutral, author.py seals a genuine "changed" divergence, and validate_scenario proves grader polarity. Verified offline: validate_scenario --all green (7/7), a mock run over the three is error-free, uv run pytest passes (38). Stacked on PR-A (#124) for the validate_scenario tooling + edit_path; will retarget to main once #124 merges. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR-A of the cascade-family expansion (4 → ~14), which kills the "the headline rests on 4 hand-built fixtures" critique. Ships the polarization tooling plus the first 4 new scenarios — each a new drift archetype in a new domain. Fully offline, no spend.
Tooling (foundation for PR-B/C)
tools/validate_scenario.py— an offline grader-polarization self-test.author.pyproves the drift is real; this proves the graders discriminate. It reuses the livegrade_code/grade_qaon author-provided reference solutions (.author/solution_correct.*→ ok & not misled;.author/solution_stale.*→ not ok & misled), catching mis-polarized graders before any API spend — the single biggest authoring risk across many fixtures.scenarios.py— anedit_pathmeta key (the visible file the agent returns) so the self-test can build theFILE:block; backfilled into the 4 existing cascades.scenarios/CHECKLIST.md— repeatable authoring steps + neutrality rules.tests/test_scenarios.py— parametrized guard that every scenario with reference solutions is correctly polarized.New scenarios (Python)
cascade-ttl-units-codecascade-money-rounding-codecascade-backoff-offbyone-codecascade-signal-threshold-codeEach: drift is detectable by
surf(a genuine"changed"divergence seals), graders probe the real hidden dependency for ground truth, andtask.mdis neutral (no leaked value).Verification (offline)
python tools/author.pyseals all four (genuinechangeddivergence).python tools/validate_scenario.py --all— green (graders discriminate).uv run pytest— 35 passed;cargo fmt --all --checkclean.Scope
PR-A of Connorrmcd6/surface-bench#12. PR-B (3 TS code cascades) and PR-C (3 QA cascades) follow, to reach ~14 total and unblock the phase-8 pre-registration freeze.
🤖 Generated with Claude Code