fix(optimization): run official benchmark reproducibly

[drewstone]

What changed

• install the pinned upstream GEPA and SkillOpt projects before live comparisons
• replace stale workflow inputs with controls the current benchmark reads
• match candidate-case evaluation counts and per-method optimizer-model spend
• keep a separate whole-run spending stop and record both limits
• require explicit model and token-price metadata for paid runs
• fail when the result artifact is missing
• test the workflow-to-program contract

Checks

• pnpm exec vitest run tests/official-optimizer-benchmark-workflow.test.ts
• pnpm typecheck:examples
• pnpm typecheck (commit hook)
• pnpm exec biome check tests/official-optimizer-benchmark-workflow.test.ts examples/compare-optimization-methods/index.ts
• actionlint .github/workflows/official-optimizer-benchmark.yml
• git merge-tree --write-tree origin/main HEAD

[tangletools]

✅ Auto-approved drewstone PR — ab7e69f5

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-07-24T17:43:23Z}

[tangletools]

🟡 Value Audit — sound-with-nits

Verdict	sound-with-nits
Concerns	3 (3 weak-concern)
Heuristic	0.0s
Duplication	0.0s
Interrogation	129.8s (2 bridge agents)
Total	129.8s

💰 Value — sound-with-nits

Replaces a silently-broken scheduled workflow with one that installs pinned GEPA/SkillOpt sources and uses the env vars the benchmark actually reads, plus splits one cost knob into per-method and whole-run limits — a sound, in-grain fix with one minor test-robustness nit.

• What it does: Two things. (1) Deletes .github/workflows/empirical-gate.yml and adds .github/workflows/official-optimizer-benchmark.yml: the new workflow runs uv sync --frozen --group skillopt-source --group gepa-source to install the pinned upstream optimizers, passes the env vars the current benchmark reads (SKILLOPT_EPOCHS, SKILLOPT_BATCH_SIZE, MAX_OPTIMIZER_MODEL_COST_USD, MAX_TOTAL_COST_USD), hard-fails t
• Goals it achieves: Make the weekly official-optimizer comparison reproducible and not silently broken: the old workflow passed POPULATION/GENERATIONS/EPOCHS which the benchmark never reads, never installed the Python optimizer sources, and only warned (didn't error) when the result artifact was missing — so it could 'succeed' while producing nothing comparable. Secondary goals: equal, separately-auditable cost contr
• Assessment: Sound. The split of MAX_RUN_COST_USD into two knobs is semantically correct: costCeiling (src/campaign/presets/compare-optimization-methods.ts:447) guards the whole run (optimization + final test), while OptimizerModelBudget.maxCostUsd caps each method's own model spend — conflating them under one number was the real defect. The new workflow reuses existing machinery in-grain: the skillopt-source/
• Better / existing approach: No existing equivalent to reuse — this is the first test in the repo to read a .github/workflows/*.yml (git grep found none), and no other workflow does pinned-source install + cost-metadata gating. The only materially-available improvement is test robustness: the contract test uses raw string substring/regex assertions on the YAML (tests/official-optimizer-benchmark-workflow.test.ts:16-44) rather
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound-with-nits

A coherent fix that replaces a stale scheduled workflow with one wired to the controls the benchmark actually reads, correctly separates per-method optimizer spend from whole-run spend, and locks the workflow↔program contract with a test.

• Integration: Fully wired and reachable. The new .github/workflows/official-optimizer-benchmark.yml (added in this same commit ab7e69f, replacing the deleted empirical-gate.yml) is the live caller: it sets SKILLOPT_EPOCHS, SKILLOPT_BATCH_SIZE, MAX_OPTIMIZER_MODEL_COST_USD, MAX_TOTAL_COST_USD as env vars (workflow:108-111) and runs pnpm tsx examples/compare-optimization-methods/index.ts (workflow:113), which r
• Fit with existing patterns: Fits the grain. The example already used positiveNumberEnv/positiveIntegerEnv + optimizerModelBudgetFromEnv; this PR keeps that pattern and only renames one control into two. The two-budget split is strictly more correct than the prior single MAX_RUN_COST_USD, which previously served BOTH as the per-method optimizer budget default AND the whole-run ceiling (old index.ts used MAX_RUN_COST_USD for o
• Real-world viability: Holds up. Paid runs are gated behind explicit MODEL + PRICE_IN + PRICE_OUT (workflow:88-91), neutral-skip when no key secret exists (workflow:93-96), and the artifact upload fails closed on a missing result (workflow:121). The shared CostLedger enforces the ceiling atomically across concurrent methods (cost-ledger.ts:364-371). The stale-name guard test uses regex ^\s+NAME: so SKILLOPT_EPOCHS is
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

💰 Value Audit

🟡 Workflow contract test asserts on raw YAML strings instead of parsed structure [better-architecture] ``

tests/official-optimizer-benchmark-workflow.test.ts:24-37 uses ${name}: substring checks and ^\s+${staleName}: regex on the whole-file text, so it can't distinguish whether a required env var (e.g. MAX_OPTIMIZER_MODEL_COST_USD) is wired into the correct step's env block versus appearing in a comment, a description, or the wrong step. The same guard against drift could be made structural by parsing the YAML (js-yaml is a common choice) and asserting that each required name appears in the 'Com

🎯 Usefulness Audit

🟡 Companion README still documents the removed MAX_RUN_COST_USD [ergonomics] ``

examples/compare-optimization-methods/README.md:107 documents MAX_RUN_COST_USD (default 12), which this PR replaces with MAX_OPTIMIZER_MODEL_COST_USD (5) + MAX_TOTAL_COST_USD (20). A caller copying the README sets an env var the renamed example no longer reads. Update the table in the same PR so docs match the shipped controls.

🟡 Sibling benchmarks still conflate the two budgets this PR separates [problem-fit] ``

examples/benchmarks/gsm8k/compare-optimization-methods.ts:83,371,419 and examples/benchmarks/appworld/run-bench.ts:78,400,429 still use a single MAX_RUN_COST_USD as both per-method optimizer budget and whole-run costCeiling — the exact conflation this PR fixes for the main example. Not a blocker (each example is independent), but they're the natural next adopters of the two-budget split.

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260724T174735Z}

[tangletools]

✅ No Blockers — ab7e69f5

Review health 100/100 · Reviewer score 67/100 · Confidence 75/100 · 7 findings (1 medium, 6 low)

glm: Correctness 67 · Security 67 · Testing 67 · Architecture 67

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 3/3 planned shots over 3 changed files. Global verifier still owns final merge decision.

🟠 MEDIUM No concurrency group on a paid scheduled benchmark — .github/workflows/official-optimizer-benchmark.yml

The workflow has no concurrency: block. A manual workflow_dispatch (or two dispatches) overlapping the weekly cron, or overlapping dispatches, each run independently and each can spend up to MAX_TOTAL_COST_USD ($20 default). For a paid LLM benchmark this is a real billing-footgun, not just a hygiene issue. Add e.g. concurrency: { group: official-optimizer-benchmark, cancel-in-progress: false } so runs serialize. Fix is two lines.

🟡 LOW No explicit permissions: block — default token broader than needed — .github/workflows/official-optimizer-benchmark.yml

The job only needs to checkout (contents:read) and upload artifacts (implicit actions token). Without an explicit permissions: block it inherits the repo/org default, which may be write-all. Add permissions: { contents: read } at job or workflow level for least-privilege.

🟡 LOW if-no-files-found:error under if:always() produces a misleading secondary failure — .github/workflows/official-optimizer-benchmark.yml

When the 'Compare official GEPA and SkillOpt' step exits non-zero (API timeout, cost-ceiling abort, integrity assertion), no comparison.json is written. The upload step then runs because of if: always() and fails with if-no-files-found: error, surfacing an artifact-upload error that can mask the real benchmark failure in the job summary. Intended to guarantee output visibility, but consider if-no-files-found: warn (or gating the upload on steps.compare.outcome == 'success' once the compare step gets an id) so the primary failure stays legible.

🟡 LOW MAX_RUN_COST_USD dropped with no warning for existing automation — examples/compare-optimization-methods/index.ts

The env var MAX_RUN_COST_USD (previously default 12) is removed entirely. Any existing CI or operator script that sets MAX_RUN_COST_USD will silently lose effect: the total ceiling jumps from the old value to the new default 20, and per-method optimizer budget drops from the old value to 5 — a net cost-ceiling INCREASE (12→20) that could surprise a budget-constrained runner. The repo's 'no backward-compat shims' doctrine makes this acceptable for an example script, but a one-line console.warn when process.env.MAX_RUN_COST_USD is set would prevent silent misconfiguration. Impact: cost overrun surprise for existing users; no correctness defect.

🟡 LOW No cross-validation that per-method budget × numMethods fits under total ceiling — examples/compare-optimization-methods/index.ts

costCeiling is set to MAX_TOTAL_COST_USD (20) and each method's optimizer budget is MAX_OPTIMIZER_MODEL_COST_USD (5), but nothing asserts that MAX_OPTIMIZER_MODEL_COST_USD × selectedNames.length (plus worker/scoring headroom) stays under MAX_TOTAL_COST_USD. If a user sets MAX_OPTIMIZER_MODEL_COST_USD=30 with 2 methods but MAX_TOTAL_COST_USD=20, the per-method budgets (60) far exceed the total ceiling (20), so compareOptimizationMethods will abort mid-optimization when the shared costLedger hits its ceiling — the same incoherence this refactor set out to remove, just inverted. This validation did not exist before either (not a regression), but this refactor is the natural place to add a guard like: if (MAX_OPTIMIZER_MODEL_COST_USD * selectedNames.length > MAX_TOTAL_COST_USD) throw. Low seve

🟡 LOW Module-level readFileSync degrades error reporting on missing files — tests/official-optimizer-benchmark-workflow.test.ts

Both readFileSync calls execute at module load time (lines 6-13), outside any describe/it/beforeAll. If either file is missing (e.g., renamed in a future refactor), vitest reports a collection-time error with all three tests showing as 'errored' rather than a clean named failure. Moving the reads into beforeAll or the test body would preserve the assertion name in the failure output. Minor robustness nit, not a blocker.

🟡 LOW Stale-name negative check only covers workflow YAML, not benchmark TS — tests/official-optimizer-benchmark-workflow.test.ts

The negative assertion expect(workflow).not.toMatch(...) for POPULATION/GENERATIONS/EPOCHS only scans the workflow YAML. The positive assertions check both workflow and benchmark, but if someone reintroduces process.env.POPULATION / process.env.GENERATIONS reads in examples/compare-optimization-methods/index.ts, this test will not catch it. Adding expect(benchmark).not.toContain(...) for the stale names would close the gap and match the bilateral symmetry of the positive checks.

---

_{tangletools · 2026-07-24T17:54:37Z · trace}

[tangletools]

✅ Approved — 7 non-blocking findings — ab7e69f5

Full multi-shot audit completed 3/3 planned shots over 3 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-07-24T17:54:37Z · immutable trace}

[tangletools]

✅ Auto-approved drewstone PR — c714bf63

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-07-24T18:07:46Z}

[tangletools]

🟢 Value Audit — sound

Verdict	sound
Concerns	0 (none)
Heuristic	0.0s
Duplication	0.0s
Interrogation	169.8s (2 bridge agents)
Total	169.8s

💰 Value — sound

Splits a conflated cost knob into distinct per-method and whole-run ceilings, and replaces a stale CI workflow with a reproducible, contract-tested benchmark harness — correct and in-grain.

• What it does: Splits the single MAX_RUN_COST_USD env var (previously fed to BOTH the per-method optimizer-model budget via optimizerModelBudgetFromEnv AND the whole-run costCeiling) into two distinct controls: MAX_OPTIMIZER_MODEL_COST_USD (equal per-method optimizer-model spend) and MAX_TOTAL_COST_USD (whole-run stop). Replaces the old empirical-gate.yml workflow — which passed dead inputs (`POPULAT
• Goals it achieves: Three goals, all read from the change: (1) Fix a budget-accounting bug — the old code gave each method an optimizer-model spend cap equal to the total-run ceiling, so two methods could collectively spend 2× the 'total' on optimizer calls alone before any final scoring. (2) Make the scheduled benchmark actually reproducible — install the exact pinned upstream optimizers, require model+price metadat
• Assessment: The core insight is correct and well-grounded. The preset at src/campaign/presets/compare-optimization-methods.ts:234 documents costCeiling as the shared limit across 'every method's optimizer and evaluation calls plus final scoring', while each method's optimizer.budget.maxCostUsd (from optimizerModelBudgetFromEnv) is a separate per-method surface. Feeding both surfaces the same single va
• Better / existing approach: none — this is the right approach. The budget split matches two genuinely distinct surfaces in the preset API. The workflow↔benchmark contract test is the correct lightweight guardrail against drift. The yaml devDep is justified solely by that test. I checked whether the env-var triple (MAX_OPTIMIZER_MODEL_COST_USD / MAX_TOTAL_COST_USD / GEPA_MAX_PROPOSER_COST_USD) repeated across three sc
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

Replaces a genuinely broken scheduled benchmark (feeding env vars the benchmark ignores, silently warning on missing artifacts, conflating two cost limits) with a correct, reproducible, fail-loud equivalent locked by a contract test.

• Integration: Fully wired and reachable. The new .github/workflows/official-optimizer-benchmark.yml:7-9 runs on a weekly cron + workflow_dispatch, installs two real pinned uv dependency groups (clients/python/pyproject.toml:42-47 defines skillopt-source and gepa-source at pinned commits), invokes a real entry point (examples/compare-optimization-methods/index.ts:372-373 writes comparison.json to .evolve/...), a
• Fit with existing patterns: Fits the codebase's grain precisely. compareOptimizationMethods already separates a whole-run costCeiling (src/campaign/presets/compare-optimization-methods.ts:235) from per-method optimizer-model budgets (examples/_shared/optimizer-model-budget.ts via optimizerModelBudgetFromEnv). The old code fed the SAME MAX_RUN_COST_USD number into both (examples diffs), conflating two distinct limits; this PR
• Real-world viability: Holds up on non-happy paths. Verified the old workflow was actively feeding dead inputs: grep shows examples/compare-optimization-methods/ and examples/_shared/ never read POPULATION/GENERATIONS/EPOCHS, so the old POPULATION/GENERATIONS/EPOCHS inputs were no-ops. The new workflow fail-closes on missing model/price metadata before any paid call ([ -z "$MODEL" ] / [ -z "$PRICE_IN" ] / [ -z "$PRICE_O
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

No concerns — sound change, no better or existing approach found. ✅

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260724T181051Z}

[tangletools]

✅ No Blockers — c714bf63

Review health 100/100 · Reviewer score 65/100 · Confidence 90/100 · 10 findings (10 low)

glm: Correctness 65 · Security 65 · Testing 65 · Architecture 65

Reviewer score is advisory once the run is complete and the verdict has no blockers.

Full multi-shot audit completed 6/6 planned shots over 9 changed files. Global verifier still owns final merge decision.

🟡 LOW Artifact name collides on workflow re-run — .github/workflows/official-optimizer-benchmark.yml

Artifact name 'official-optimizer-benchmark-${{ github.run_number }}' is stable across re-runs because github.run_number does not change between attempts of the same run. actions/upload-artifact@v4 rejects duplicate artifact names within a run, so 'Re-run failed jobs' / 'Re-run all jobs' will fail at this step with 'an artifact with this name already exists' if the first attempt uploaded anything. Fix: include github.run_attempt in the name, e.g. 'official-optimizer-benchmark-${{ github.run_number }}-attempt${{ github.run_attempt }}'. Low severity: scheduled runs (the primary trigger) always get a fresh run_number; only affects manual/failed re-runs.

🟡 LOW First execution is unverified (no dry-run evidence) — .github/workflows/official-optimizer-benchmark.yml

This is a new workflow with no recorded green run. Env-var wiring is correct by static inspection, but the end-to-end path (uv sync of both gepa-source + skillopt-source groups in CI, pnpm tsx invocation, Python subprocess at clients/python/.venv/bin/python) has not been demonstrated. Recommend a single workflow_dispatch smoke run with max_total_cost_usd=1 and max_optimizer_model_cost_usd=0.5 before relying on the Monday cron, to confirm the venv path and group co-install resolve on the runner.

🟡 LOW README GEPA_MAX_PROPOSER_COST_USD default is stale after coupling to MAX_OPTIMIZER_MODEL_COST_USD — examples/benchmarks/appworld/README.md

Line 53 still documents GEPA_MAX_PROPOSER_COST_USD default as 5, but run-bench.ts:60-63 now defines it as positiveNumberEnv('GEPA_MAX_PROPOSER_COST_USD', MAX_OPTIMIZER_MODEL_COST_USD). The sibling rows (GEPA_MAX_MODEL_COST_USD line 56, SKILLOPT_MAX_MODEL_COST_USD line 63) were correctly updated in this same diff to show MAX_OPTIMIZER_MODEL_COST_USD as the default, but this row w

🟡 LOW Silent env var rename MAX_RUN_COST_USD -> MAX_TOTAL_COST_USD with no migration notice — examples/benchmarks/appworld/run-bench.ts

MAX_RUN_COST_USD was removed and replaced by MAX_TOTAL_COST_USD (and MAX_OPTIMIZER_MODEL_COST_USD). An operator who has MAX_RUN_COST_USD exported will now silently get the default (125) instead of their value, with no warning — and their intended ceiling becomes per-method-optimizer-budget-shaped, not total. The README documents the new names, so this is a documentation-only migration, but a one-line deprecated-alias check (read MAX_RUN_COST_USD and warn if set) would prevent a silent 125 USD run when someone intended 50. Low impact since these are example benchmarks, not library API.

🟡 LOW No cross-validation that per-method optimizer budgets fit within the total ceiling — examples/benchmarks/gsm8k/compare-optimization-methods.ts

With two methods, optimizerModelBudgetFromEnv('GEPA', MAX_OPTIMIZER_MODEL_COST_USD) + optimizerModelBudgetFromEnv('SKILLOPT', MAX_OPTIMIZER_MODEL_COST_USD) can each be set to values whose sum exceeds MAX_TOTAL_COST_USD (costCeiling). The shared CostLedger will still enforce the ceiling, so the run fails mid-optimization rather than over-spending — but the failure surfaces as an opaque optimizer call rejection deep in the run. A startup assertion (e.g. numMethods * MAX_OPTIMIZER_MODEL_COST_USD + smokeEstimate <= MAX_TOTAL_COST_USD) would fail fast with a clear message. Defaults (10*2=20 <= 25) are fine, but env-overridable footgun remains. Pre-existing class of issue, not introduced by this PR's logic.

🟡 LOW README GEPA_MAX_PROPOSER_COST_USD default shows literal 5, not the inherited env var — examples/compare-optimization-methods/README.md

Row 94 lists GEPA_MAX_PROPOSER_COST_USD | 5. The code now defaults it to MAX_OPTIMIZER_MODEL_COST_USD (index.ts:57-60), so if a user raises MAX_OPTIMIZER_MODEL_COST_USD the proposer default follows — but the doc shows a flat 5. The sibling rows GEPA_MAX_MODEL_COST_USD (line 97) and SKILLOPT_MAX_MODEL_COST_USD (line 105) correctly show MAX_OPTIMIZER_MODEL_COST_USD as their default for exactly this inheritance reason. For consistency row 94 should read MAX_OPTIMIZER_MODEL_COST_USD instead of 5. E

🟡 LOW Pinned version breaks caret convention — package.json

"yaml": "2.9.0" is the only dep in the file without a ^ caret (all other deps/devDeps use ^). For a test-only devDep this is harmless and pnpm.lock already pins the resolved version, but the inconsistency suggests an oversight. Either pin intentionally elsewhere or align to ^2.9.0 for consistency. No functional impact.

🟡 LOW Compare step's paid-work gate (if: steps.config.outputs.run == 'true') is not asserted — tests/official-optimizer-benchmark-workflow.test.ts

Test 3 asserts the Upload step's if (line 68) but never asserts the Compare step's if: steps.config.outputs.run == 'true' (workflow line 106). The Compare step is the actual paid LLM work; the Upload if only gates artifact publishing. If someone removed the Compare step's if, paid work would run on every trigger regardless of the config gate (the script would still throw on a missing API key, but only after process startup). Fix: add `expect(stepNamed('Compare official GEPA and SkillOpt

🟡 LOW Exact-string toBe on run commands is brittle to benign edits — tests/official-optimizer-benchmark-workflow.test.ts

expect(...run).toBe('uv sync --frozen --group skillopt-source --group gepa-source') and the identical exact-match on the Compare run fail on any flag reorder, whitespace change, or added flag. For a paid-work lock this strictness is intentional and acceptable, but it means every benign workflow tweak forces a test edit. Consider asserting the load-bearing tokens (--frozen, both --group names, pnpm tsx, the script path) via toContain if you want lower churn; leave as-is if you want maximum rigidity.

🟡 LOW Type cast of document.on omits schedule; cron trigger untested — tests/official-optimizer-benchmark-workflow.test.ts

The cast on: { workflow_dispatch: { inputs: ... } } hides the schedule: - cron: '17 6 * * 1' trigger (workflow lines 8-9). Runtime is unaffected (parse returns the full object), but TypeScript won't catch a deleted/changed schedule because nothing reads it. If the weekly cadence matters, add a schedule field to the cast and assert the cron string.

---

_{tangletools · 2026-07-24T18:19:36Z · trace}

[tangletools]

✅ Approved — 10 non-blocking findings — c714bf63

Full multi-shot audit completed 6/6 planned shots over 9 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-07-24T18:19:36Z · immutable trace}