Skip to content

fix(optimization): record optimizer model provenance#433

Merged
drewstone merged 1 commit into
mainfrom
fix/optimizer-provenance-model
Jul 24, 2026
Merged

fix(optimization): record optimizer model provenance#433
drewstone merged 1 commit into
mainfrom
fix/optimizer-provenance-model

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

What changed

  • record the configured optimizer model in official GEPA and SkillOpt provenance
  • omit the field for GEPA engines that own no configured optimizer model
  • validate and clone the new field through method comparison and self-improvement results
  • release the npm and Python packages as 0.126.6

Proof

  • 99 focused tests passed
  • full suite: 3,382 passed, 3 environment-gated skipped
  • typecheck, examples, lint, build, package verification, and Python lock check passed
  • branch merges cleanly into current main

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — b942ab08

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

tangletools · auto-approval · reason: drewstone_author · 2026-07-24T20:49:36Z

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Value Audit — sound

Verdict sound
Concerns 0 (none)
Heuristic 0.0s
Duplication 0.0s
Interrogation 103.8s (2 bridge agents)
Total 103.8s

💰 Value — sound

Adds the configured optimizer model as an optional provenance field on optimization-method results (GEPA conditionally, SkillOpt always), validated in-grain — small, coherent, no existing equivalent to reuse.

  • What it does: Introduces an optional optimizerModel?: string field on OptimizationMethodProvenance (src/campaign/presets/compare-optimization-methods.ts:86) and writes the configured model identifier into the run provenance: GEPA emits it only when an optimizer is configured (src/campaign/gepa-optimization-method.ts:479, conditional spread so engineModules-only runs omit it), SkillOpt emits it unconditional
  • Goals it achieves: Reproducibility/audit gap: the provenance object recorded package, bridge, module, and Python runtime identity, but NOT which model actually drove candidate generation. The model string was already fed into the opaque compatible-run-key hash (gepa-optimization-method.ts:263-269) but was never surfaced as a human-readable output field — so a comparison or self-improvement result named the optimizer
  • Assessment: Good change, built in the grain of the codebase. The provenance object is the established home for 'what implementation produced this result', and the model identifier is a first-class identity input for these optimizers, so adding it there (rather than a new side-channel) is correct. Validation idiom is identical to surrounding fields. The conditional emission for GEPA's no-optimizer path is hand
  • Better / existing approach: none — this is the right approach. I checked for duplication: the model lives in config.optimizer.model (input config) and inside the hashed run-material at gepa-optimization-method.ts:263 (opaque, not a readable output) — neither is a provenance output, so no existing field to reuse. The assertOptimizerModel helper in src/campaign/optimizer-model.ts:16 validates the whole {model, baseUrl, api
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 2
  • Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content

🎯 Usefulness — sound

Adds the optimizer model identifier to optimization provenance — a fully wired, correctly validated field that fills an obvious reproducibility gap without duplicating anything.

  • Integration: Fully reachable end-to-end. Both official methods set it: SkillOpt unconditionally (skillopt-optimization-method.ts:362, config.optimizer is required per skillopt-optimization-config.ts:131) and GEPA conditionally (gepa-optimization-method.ts:479, config.optimizer is optional per gepa-optimization-config.ts:117). It flows through compareOptimizationMethods (validated at compare-optimization-method
  • Fit with existing patterns: Matches the existing provenance grain exactly. OptimizationMethodProvenance (compare-optimization-methods.ts:76-94) already records source, bridge, modules, python runtime, runId — the exact model identifier is the natural missing piece. The conditional-vs-unconditional split correctly mirrors established config semantics. No competing field: backend.models (provenance.ts:486) records dispatch/jud
  • Real-world viability: Robust across all paths. Optional field breaks no existing consumer. The GEPA conditional spread omits the key cleanly when no optimizer is configured, with an explicit test asserting its absence (gepa-optimization-method.test.ts). Validation rejects untrimmed strings (compare-optimization-methods.ts:563-569), consistent with every other string field in the same validator. Deep-cloned via structur
  • Model: opencode/zai-coding-plan/glm-5.2
  • Bridge attempts: 1

No concerns — sound change, no better or existing approach found. ✅


What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass What it asks
Heuristic Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication Do added function/class names already exist elsewhere in the repo?
Value Audit What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

value-audit · 20260724T205313Z

@drewstone
drewstone merged commit ad59952 into main Jul 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants