Skip to content

Add a holdout-safe Fit experience index - #18

Open
divo12 wants to merge 3 commits into
codex/hermes-agent-diagnoserfrom
codex/fit-experience-index
Open

Add a holdout-safe Fit experience index#18
divo12 wants to merge 3 commits into
codex/hermes-agent-diagnoserfrom
codex/fit-experience-index

Conversation

@divo12

@divo12 divo12 commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Summary

  • write one content-bound, typed experience index per Fit campaign
  • preserve paired developer runs, verifier feedback, trace/snapshot lineage, candidate manifest, prediction attribution, diff and raw developer benchmark references
  • expose only selection/admission completion and pass decisions; never copy holdout identifiers, snapshots, outputs, feedback, or artifact paths
  • validate canonical source paths, digests, and rebuilt developer evidence on cached Fit reads

Verification

  • 167 tests passed
  • Ruff clean
  • mypy --strict clean
  • Bandit clean
  • pip-audit: no known vulnerabilities
  • wheel and sdist build
  • forbidden dynamic/raw type scan clean
  • independent reviewer: APPROVE

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 6 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/ofw/benchmarking.py">

<violation number="1" location="src/ofw/benchmarking.py:82">
P2: When a repository already contains a baseline or Fit manifest from before this change, cached reads reject its attempts because `source_case_id` is missing, and legacy baseline verification reports semantic drift after the field is added to `_semantic`. Add backward-compatible deserialization and preserve the legacy digest for old artifacts, or explicitly version and migrate/invalidate these persisted results.</violation>
</file>

You're on the cubic free plan with 18 free PR reviews remaining this month. Upgrade for unlimited reviews.

Re-trigger cubic

Comment thread src/ofw/benchmarking.py
@dataclass(frozen=True, slots=True)
class CaseAttempt:
case_id: str
source_case_id: str

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When a repository already contains a baseline or Fit manifest from before this change, cached reads reject its attempts because source_case_id is missing, and legacy baseline verification reports semantic drift after the field is added to _semantic. Add backward-compatible deserialization and preserve the legacy digest for old artifacts, or explicitly version and migrate/invalidate these persisted results.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/ofw/benchmarking.py, line 82:

<comment>When a repository already contains a baseline or Fit manifest from before this change, cached reads reject its attempts because `source_case_id` is missing, and legacy baseline verification reports semantic drift after the field is added to `_semantic`. Add backward-compatible deserialization and preserve the legacy digest for old artifacts, or explicitly version and migrate/invalidate these persisted results.</comment>

<file context>
@@ -79,6 +79,7 @@ def digest(self) -> Sha256Digest:
 @dataclass(frozen=True, slots=True)
 class CaseAttempt:
     case_id: str
+    source_case_id: str
     partition: ExportPartition
     critical: bool
</file context>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant