Roadmap item 5: Add inactive eval record validator (Codex-authored)

[yihanzhu]

Summary

• evals/v1/framework.jq and run.sh: an eval-record validator (evaluate BUNDLE.json) built by the Codex construction session before it stopped.
• Authored by the Codex construction session; adopted for review by the operator's decision on 2026-09-05 (keep both unpushed Codex units and publish them once the eval framework landed). Published as-is from the local branch; no content changes.
• Base: main (built against a637451; needs a merge of current main before it can land, which the operator-authorized merge chain performs only after a clean review).

Exact candidate

• Head: b4ce778
• Roadmap: item 5, Agent evals and telemetry (Codex unit)

Fresh CI and a fresh independent review are required for this head. Findings are folded here.

[cloudflare-workers-and-pages]

Deploying ystack with    Cloudflare Pages

Latest commit:	d4ae469
Status:	✅  Deploy successful!
Preview URL:	https://42ab9303.fabrica-6yx.pages.dev
Branch Preview URL:	https://codex-eval-framework-v1.fabrica-6yx.pages.dev

View logs

[yihanzhu]

Codex reviewer (cross-vendor, read-only)

Reviewed-head: b4ce778
Reviewed-base: 96f96e4
reviewer: gpt-5.5 @ high

Posted verbatim by codex-review.sh (codex exec review --json --base refs/codex-review/241-13981/base in an isolated temp worktree, sandbox forced read-only). Comments only — Codex never pushes, approves, or merges.

The evaluator rejects valid multi-trial identity layouts and fails under a common executable invocation path. These are functional issues in the newly added runner/evaluator behavior.

Full review comments:

• [P2] Stop treating trial identity lists as sorted sets — /var/folders/rm/bkx4f4b91vqfn79vk02pcd140000gn/T/tmp.guVdeS5TgM/evals/v1/framework.jq:61-62
  When a case uses trial or attempt IDs whose lexicographic order differs from trial-index order, such as trial.1 ... trial.10, this shape check rejects the case even though the relation check later expects the IDs in trial-index order. That makes otherwise valid multi-trial model cases up to the advertised limit of 16 fail unless callers know to zero-pad or otherwise choose IDs that sort the same way as the trial indexes.

• [P2] Normalize the executable path before the repo check — /var/folders/rm/bkx4f4b91vqfn79vk02pcd140000gn/T/tmp.guVdeS5TgM/evals/v1/run.sh:49-49
  When the executable is invoked as ./evals/v1/run.sh evaluate ..., BASH_SOURCE[0] is kept as /repo/./evals/v1/run.sh, while source_dir is canonicalized to /repo/evals/v1; this strict string comparison then emits E_RUNTIME before any evaluation. This is a common way to run an executable script and is not covered by the current test, which calls the runner through an absolute path.

[yihanzhu]

Codex reviewer (cross-vendor, read-only)

Reviewed-head: 43a1d56
Reviewed-base: 96f96e4
reviewer: gpt-5.5 @ high

Posted verbatim by codex-review.sh (codex exec review --json --base refs/codex-review/241-5732/base in an isolated temp worktree, sandbox forced read-only). Comments only — Codex never pushes, approves, or merges.

I found no actionable correctness issues in the added eval-record evaluator, runner, documentation, or test wiring.

[yihanzhu]

Codex reviewer (cross-vendor, read-only)

Reviewed-head: 9539266
Reviewed-base: 8c1ad22
reviewer: gpt-5.5 @ high

Posted verbatim by codex-review.sh (codex exec review --json --base refs/codex-review/241-1533/base in an isolated temp worktree, sandbox forced read-only). Comments only — Codex never pushes, approves, or merges.

The evaluator can treat a canonical JSON stream as valid input and emit more than one report, which breaks the documented one-bundle/one-report behavior.

Review comment:

• [P2] Reject multi-root JSON streams — /private/var/folders/rm/bkx4f4b91vqfn79vk02pcd140000gn/T/tmp.ebWwTKXwV1/evals/v1/run.sh:95-98
  Because jq accepts JSON streams, this canonicalization step proves each JSON text is canonical but not that BUNDLE.json contains exactly one bundle. If an input contains multiple canonical eval_bundle documents with wrapper hashes computed from the streamed extraction, the later jq filters run once per document and the final output can contain multiple eval_report objects, violating the new single-report contract. Please add a single-root check before hashing and evaluation.

[yihanzhu]

Codex reviewer (cross-vendor, read-only)

Reviewed-head: d4ae469
Reviewed-base: 8c1ad22
reviewer: gpt-5.5 @ high

Posted verbatim by codex-review.sh (codex exec review --json --base refs/codex-review/241-6251/base in an isolated temp worktree, sandbox forced read-only). Comments only — Codex never pushes, approves, or merges.

I did not find any discrete, actionable defects in the changed evaluator, runner, tests, or restore/manifest documentation.