signoff

Statistical sign-off for the model swap. N-run, nondeterminism-aware A/B equivalence testing for LLM migrations, with an audit-ready report.

Model deprecations force migrations on someone else's schedule. "We eyeballed ten outputs" is not evidence a regulated enterprise can take to a risk committee. signoff runs your real prompt suite on the old and new model N times each, measures per-case flip rates (nondeterminism), scores answers through layered matching (exact, normalized, numeric, then LLM-judge), and applies a paired sign test before declaring the swap safe.

Vendor-neutral by design: it judges an OpenAI to Anthropic migration with exactly the same rigor as the reverse.

Install

pip install -e .
export OPENAI_API_KEY=...      # for openai: models and/or the judge
export ANTHROPIC_API_KEY=...   # for anthropic: models

Use

signoff run \
  --suite examples/extraction.yaml \
  --model-a openai:gpt-4o \
  --model-b anthropic:claude-sonnet-4-5 \
  --runs 5 \
  --judge openai:gpt-4o-mini \
  --out report.html

Output: a one-line verdict in the terminal, plus report.html (human) and report.json (machine).

cases=48 agreement=95.8% flip_a=2.1% flip_b=1.7%
accuracy: A=93.8% B=95.8% p=0.512
verdict: PASS   report: report.html / report.json

Verdict logic: with ground truth (expected per case), FAIL only when model B is less accurate and the regression is statistically significant (exact sign test on discordant pairs, p <= 0.05). Without ground truth, 90%+ majority-answer agreement passes; below that, REVIEW.

Suite format

cases:
  - id: facility-amount
    prompt: "Extract the total facility amount. ..."
    expected: "$250,000,000"   # optional; omit for agreement-only mode

Why N runs?

Single-run comparisons confuse nondeterminism with regression. A prompt that flips answers 30% of the time will "regress" or "improve" by luck. signoff reports the flip rate per case per model, so instability is visible before it's misattributed to the migration.

Judge options

The semantic match layer accepts two judge kinds:

--judge openai:gpt-4o-mini                 # classic yes/no LLM judge
--judge jev --judge-threshold 0.8          # probability judge (Jev API, JEV_AI_API_KEY)

The jev judge counts a match when the stated same-information probability clears the threshold. Caveat from our own published test (github.com/ashish24142/jev-clm): this signal is a reasonable match decision but a weak error detector, so do not use the probability to skip human QA of extractions.

Prompt stability (flakiness) mode

Measure a single model's flakiness, and find which prompt section causes it:

signoff stability --suite suite.yaml --model openai:gpt-4o --runs 10 --ablate --gate 0.1

Give cases sections instead of prompt to enable ablation:

cases:
  - id: governing-law
    sections:
      - name: role
        text: "You extract facts from legal documents."
      - name: task
        text: "Extract the governing law state from the excerpt below."
      - name: format
        text: "Reply with only the state name."

For each case you get the flip rate (fraction of N runs disagreeing with the majority answer), and with --ablate, the flip rate with each section removed: the section whose removal makes instability jump is the section doing the stabilizing work, and the one whose removal cures a flip is the section causing it. --gate turns it into a CI check: exit 1 if any case flips more than the threshold.

Roadmap

• Semantic match layer (embeddings) between numeric and judge
• Governance evidence annex: mapping results to ISO/IEC 42001 change control, NIST AI RMF MEASURE, EU AI Act Annex IV
• Adapters: Bedrock, Azure OpenAI, Gemini
• Equivalence bands (TOST) as an alternative to the sign test

License

Apache-2.0