Skip to content

[codex] add uncertainty-aware paired fit evidence - #15

Draft
divo12 wants to merge 3 commits into
codex/confirmed-failure-curationfrom
codex/paired-statistical-evidence
Draft

[codex] add uncertainty-aware paired fit evidence#15
divo12 wants to merge 3 commits into
codex/confirmed-failure-curationfrom
codex/paired-statistical-evidence

Conversation

@divo12

@divo12 divo12 commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Stack

Depends on #14. Review against codex/confirmed-failure-curation.

Research finding

Agent outcomes vary across trials. Anthropic recommends repeated trials and separates pass@k from reliability; model-selection literature recommends uncertainty and multiple-comparison correction. OFW already paired champion and candidate by case/repeat, but it promoted from point estimates alone.

Implementation

  • Adds typed PairedEvidence per partition: real wins, losses, ties, discordant pairs, net pass delta, candidate win rate, and exact one-sided sign-test probability.
  • Excludes synthetic benchmark copies from significance claims while retaining their declared weight in effect-size metrics.
  • Adds explicit StatisticalGateMode: EFFECT_SIZE_ONLY or EXACT_SIGN_TEST.
  • Adds a frozen PairedEvidencePolicy to FitPolicy and all policy/cache digests.
  • Rejects raw string modes at the direct Python runtime boundary instead of relying on annotations.
  • Exact mode requires a declared minimum discordant-pair count and maximum probability.
  • Applies Bonferroni correction across all candidates evaluated in the campaign.
  • Keeps critical regression, regression score, target effect, latency, cost, selection, and admission gates unchanged and primary.
  • Persists paired evidence in every CandidateOutcome, including rejected candidates.

TDD evidence

  • Five real target wins produce exact probability 0.03125.
  • Ties do not count as discordant evidence.
  • Synthetic wins are excluded.
  • Insufficient evidence fails exact mode.
  • The same evidence fails after correction for two candidate comparisons at alpha 0.05.
  • Five repeated target wins pass an end-to-end exact gate and reach one-shot admission.
  • Effect-size-only mode preserves existing small-fixture behavior explicitly.

Verification

  • pytest -q — 159 passed
  • Ruff and strict mypy across src and tests — clean
  • forbidden construct scan — no Any, raw dict, object, getattr, setattr, lambdas, or __import__
  • Bandit — zero findings
  • source distribution and wheel build successfully

Deliberate scope

  • This is paired offline evidence, not a claim of randomized production A/B impact.
  • No SciPy dependency, bootstrap framework, or online experimentation service is added.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant