Add a holdout-safe Fit experience index

docs/research/2026-08-22-failure-mining-evals-tuning-ab-research.md

@@ -37,7 +37,7 @@ Meta-Harness stores each candidate’s source, scores, and execution traces in a
 
 AHE resolves the scale problem with layered experience observability: a compact corpus supports navigation, while the raw evidence remains available for drill-down. It also adds decision observability, binding an edit’s predicted effects to later task-level outcomes. [Agentic Harness Engineering](https://arxiv.org/abs/2604.25850)
 
-**OFW decision:** create a typed experience index containing source trace/snapshot references, baseline and candidate verifier feedback, case deltas, and raw benchmark-result paths. It is an index, not a lossy replacement for the existing manifests.
+**OFW decision:** create a typed experience index containing developer source trace/snapshot references, paired baseline and candidate verifier feedback, case deltas, and raw developer benchmark-result paths. Selection and admission remain sealed and contribute only their pass/fail decisions. The index is not a lossy replacement for the existing manifests.
 
 ### 1.3 Trace shape and privacy are part of mining correctness
 
@@ -182,7 +182,7 @@ LangSmith’s comparative evaluation API can randomize answer order to mitigate
 ### PR17 — Drill-down optimization experience index
 
 - Write one content-bound experience manifest per Fit campaign.
-- Index cluster/source trace/snapshot, case partition, baseline/candidate verdict, all verifier feedback, prediction error, and raw benchmark result paths.
+- Index developer cluster/source trace/snapshot, case partition, paired baseline/candidate verdicts, all developer verifier feedback, prediction error, and raw developer benchmark result paths.
 - Validate the index on cached Fit reads and expose a typed reader for provider-specific proposers.
 - TDD: feedback preserved byte-for-byte, source trace linkage, rejected/winner histories, artifact tamper rejection, no holdout payload copied into proposer-visible fields.
 

docs/specs/2026-08-23-fit-experience-index.md

@@ -0,0 +1,32 @@
+# Fit experience index
+
+Each completed `FitCampaign` writes one content-bound `experience.json` beside its result manifest. The index is the provider-neutral observation surface for the next candidate proposer. It does not choose edits or summarize away source evidence.
+
+## Developer-visible evidence
+
+For every attempted candidate, the index preserves:
+
+- the frozen candidate manifest and prediction attribution;
+- final candidate status and gate reason;
+- the candidate diff artifact reference;
+- content-digested references to the raw champion and candidate developer benchmark results;
+- each developer case’s cluster family, source trace, trace family, snapshot reference, and partition; and
+- every paired champion/candidate run result, verifier verdict, score, textual feedback, metric, evidence reference, and case delta.
+
+Textual verifier feedback is stored unchanged. The compact index is navigational: the raw developer snapshot, benchmark results, and candidate diff remain available for drill-down.
+
+## Holdout boundary
+
+Selection and admission cases are evaluator-only. The proposer-visible index records only whether each stage ran, its completion status, and whether its frozen threshold passed. It never stores holdout case IDs, trace IDs, family IDs, snapshot references, prompts, outputs, verifier feedback, or raw benchmark paths.
+
+This asymmetry is intentional. Developer evidence teaches the next proposer; holdouts decide whether a frozen candidate survives. Returning holdout diagnostics would turn repeated selection into training and invalidate the gate.
+
+## Integrity
+
+`FitResult.experience_digest` binds the exact index bytes. `read_fit_experience(result)` validates campaign, export bundle, input digest, harness revision, candidate ordering, statuses, gate reasons, prediction attribution, developer-result linkage, and every referenced developer artifact digest. Cached `FitCampaign.run()` additionally rebuilds the developer case index from the raw champion/candidate benchmark artifacts and export cases before returning a prior result.
+
+Tampering with the index, candidate diff, or raw developer benchmark result therefore fails with `FitErrorCode.RESULT_INVALID`. Existing candidate/revision validation remains authoritative for the harness, export snapshots, and candidate manifest.
+
+## Why this is the minimum useful shape
+
+A scalar leaderboard cannot tell an optimizer which tool, prompt, skill, subagent, or middleware behavior caused a failure. Copying all artifacts into a second store adds drift without adding evidence. The index instead supplies AHE-style component and decision observability while retaining Meta-Harness-style drill-down to raw developer traces. A filesystem manifest and exact identifiers are sufficient for local v0; no vector database, summarizer, or provider-specific proposer belongs in this PR.

src/ofw/__init__.py

@@ -88,20 +88,29 @@
     PrivacyTransform,
 )
 from ofw.fit import (
+    CandidateExperience,
     CandidateOutcome,
     CandidateStatus,
     CaseDelta,
+    DeveloperCaseExperience,
+    FitArtifactReference,
     FitCampaign,
     FitError,
     FitErrorCode,
+    FitExperience,
+    FitExperienceSchemaVersion,
     FitPolicy,
     FitResult,
     GateReason,
+    HoldoutDecision,
+    HoldoutStage,
+    PairedAttemptExperience,
     PairedEvidence,
     PairedEvidencePolicy,
     StatisticalGateMode,
     paired_evidence,
     paired_evidence_passes,
+    read_fit_experience,
 )
 from ofw.harness import EditableFile, Harness, Subagent, Tool, editable
 from ofw.mine import (
@@ -307,6 +316,7 @@ def promote(
     "CandidateError",
     "CandidateErrorCode",
     "CandidateEvidence",
+    "CandidateExperience",
     "CandidatePolicy",
     "CandidateOutcome",
     "CandidateStatus",
@@ -342,6 +352,7 @@ def promote(
     "DeploymentAdapter",
     "DeploymentReference",
     "DeploymentRequest",
+    "DeveloperCaseExperience",
     "EditableFile",
     "EvidenceAnchor",
     "EvidenceAnchorKind",
@@ -353,9 +364,12 @@ def promote(
     "FailureCluster",
     "FailureDisposition",
     "FileEdit",
+    "FitArtifactReference",
     "FitCampaign",
     "FitError",
     "FitErrorCode",
+    "FitExperience",
+    "FitExperienceSchemaVersion",
     "FitPolicy",
     "FitResult",
     "GitCommit",
@@ -375,10 +389,13 @@ def promote(
     "HermesAgentVersion",
     "HermesDiagnoser",
     "hermes_python_command",
+    "HoldoutDecision",
+    "HoldoutStage",
     "FunctionName",
     "GateReason",
     "PairedEvidence",
     "PairedEvidencePolicy",
+    "PairedAttemptExperience",
     "Langfuse",
     "LangfuseOtelSpanAttributes",
     "LangfuseProject",
@@ -482,6 +499,7 @@ def promote(
     "paired_evidence_passes",
     "promote",
     "propagate_attributes",
+    "read_fit_experience",
     "read_observation_content",
     "read_snapshot_content",
     "read_trace_observations",

src/ofw/benchmarking.py

@@ -79,6 +79,7 @@ def digest(self) -> Sha256Digest:
 @dataclass(frozen=True, slots=True)
 class CaseAttempt:
     case_id: str
+    source_case_id: str
     partition: ExportPartition
     critical: bool
     repeat: int
@@ -242,6 +243,7 @@ def _run_suite(
                             attempts.append(
                                 CaseAttempt(
                                     case_id,
+                                    case.id,
                                     case.partition,
                                     case.critical,
                                     repeat,
@@ -365,6 +367,7 @@ def _semantic(attempts: tuple[CaseAttempt, ...]) -> tuple[CaseAttempt, ...]:
     return tuple(
         CaseAttempt(
             attempt.case_id,
+            attempt.source_case_id,
             attempt.partition,
             attempt.critical,
             attempt.repeat,