Skip to content

feat(evaluation): bridge installed datasets into production runner - #258

Draft
daniele21 wants to merge 9 commits into
devfrom
agent/eval-dataset-runner-bridge
Draft

feat(evaluation): bridge installed datasets into production runner#258
daniele21 wants to merge 9 commits into
devfrom
agent/eval-dataset-runner-bridge

Conversation

@daniele21

Copy link
Copy Markdown
Owner

Scope

Adds the concrete production dataset-to-runner bridge required to move EVAL-2/EVAL-4 beyond fake case sources without coupling evaluation:engine to filesystem or parser implementations.

  • introduces evaluation:dataset-adapter as a concrete integration module depending on evaluation:datasets and evaluation:engine;
  • implements the existing runner EvaluationDatasetPreflight and EvaluationCaseDefinitionSource ports from registry-published installed packs;
  • resolves only the exact dataset ID/version from EvaluationDatasetRegistry and verifies the requested content digest;
  • reparses cases.jsonl through the canonical bounded EvaluationDatasetJsonlParser and verifies canonical content digest before exposing cases;
  • validates sample-set dataset identity and rejects selected case IDs not present in the installed pack;
  • caches only a validated immutable in-memory snapshot keyed by exact dataset identity, so scored cases do not reparse the file repeatedly;
  • exposes manifest category definitions from the same validated snapshot for the subsequent R-10 aggregation wiring;
  • adds deterministic tests for production pack consumption, exact case lookup, category exposure, digest mismatch, invalid sample membership and missing packs;
  • registers explicit module ownership/navigation and keeps dependency direction one-way through the adapter.

Boundary

This PR does not add filesystem access to evaluation:engine, does not create an alternate parser, and does not make evaluation:datasets depend on the engine. The adapter is the composition seam between those two existing ownership boundaries.

This slice also does not yet wire EvaluationRunAggregator into the engine terminal path; the exposed category definitions are the input for that next composition slice after R-09 converges.

Gate

Keep draft until module registration/navigation, scoped adapter tests/lint and repository validation are green. EVAL-2 should not be marked DONE until this bridge is integrated and used by production evaluation composition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant