Skip to content

feat(evidence): add versioned bounded reports (#151) - #165

Draft
tcballard wants to merge 1 commit into
codex/enterprise-shadow-routingfrom
codex/enterprise-shadow-evidence
Draft

feat(evidence): add versioned bounded reports (#151)#165
tcballard wants to merge 1 commit into
codex/enterprise-shadow-routingfrom
codex/enterprise-shadow-evidence

Conversation

@tcballard

Copy link
Copy Markdown
Contributor

Summary

Adds the bounded, versioned evidence report surface for deterministic shadow-routing evaluations tracked by #151.

Scope

Included

  • prompt-free, bounded evidence-label retention with deterministic deduplication
  • human and versioned automated evaluator provenance
  • reproducible JSON (wf-evidence-v1) and bounded text reports
  • explicit missingness, confidence intervals, disagreement, provider health, cost class, and tri-state outcomes
  • operator-authenticated evidence report and label endpoints
  • audit events for report reads and label submissions
  • rejection of unknown/prompt/response fields and atomic batch-capacity enforcement
  • documentation and ADR for the evidence contract

Excluded

Product / Architecture Decisions

  • Evidence is advisory and cannot alter the production data path.
  • Insufficient, mixed, missing, or inconclusive evidence remains keep-shadowing; it is never presented as a quality claim.
  • Provider observations and quality labels are separate, with explicit accounting class and evaluator provenance.
  • Operator surfaces remain behind the existing operator-authentication boundary; labels never accept prompt or response content.

User-Facing Contract

CLI / Human Output

/v1/evidence.txt (and equivalent router paths) returns a bounded report with schema version, sample counts, reasons, provider observations, quality labels, and tri-state outcome.

JSON Output

GET /v1/evidence returns the versioned wf-evidence-v1 report. POST /v1/evidence/labels accepts bounded EvidenceLabelBatch payloads and returns the accepted-label count.

Exit Codes

No CLI exit-code changes.

Verification

Ran

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • cargo test --workspace --all-targets --locked (including socket tests with local permission)

Covered

  • sparse/missing/inconclusive evidence
  • deterministic all-win evidence and human/automated agreement
  • conflicting/unknown content rejection
  • bounded deduplication and atomic batch rejection
  • authenticated bounded endpoint behavior and prompt-free text output

Review Path

Stacked on #164 (shadow routing), targeting codex/enterprise-shadow-routing. Review this PR after #164.

Notes For Reviewer

The report is deliberately fail-safe: it only recommends enforcement after the minimum decisive sample and interval criteria pass; provider error rates above the configured bound result in do-not-enforce.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant