Roadmap item 5: Add the flow/quality dashboard over validated run results - #238
Merged
Conversation
One offline, deterministic eval pass over a caller-supplied seed set, returning one canonical run result. - evals/v1/eval-catalog.json names the nine regression families the roadmap requires before any autonomous write, each with its grader kinds, trial policy, and core evidence kinds. Two are seeded here; the other seven are declared and wait for their own seeds. - evals/v1/seed-set.json replays eight canonical core-v2 stage runs through the real portable core (scripts/core-contract.sh validate-stage-run): stale and moved artifacts, and empty, fake, and degraded review evidence. The core is the only judge; the framework records accept/reject plus token and grades that against expectation. - A wrong expectation grades failed. A family only a model or human can grade stays inconclusive. Malformed, moved, or edited inputs fail closed with one token. - Bootstrap, launcher, and driver mirror the state scanner: env -i, pinned jq 1.6 (Rosetta on arm64), private runtime snapshots with exact digests for program, catalog, driver, and the core closure, and an independent re-validation of the delivered document. - Trace events carry the Observability shape (tool, adapter, gate, identity, latency, cost). Latency and cost are absent in this unit: no model runs, so nothing honest is charged. Inactive boundary: no candidate or adapter runs, no model call, no credential or network, no write outside scratch, no qualification, no profile activation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
portable-core-schema.test.sh keeps a closed allowlist of tracked paths that may name the corrective core generation or import the schema module. The framework's program, driver, and launcher now appear there, as every prior component did when it landed. No guard rule is weakened; three paths are added to two closed lists. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review P2: a case in a family without a deterministic grader was graded inconclusive with grader_kind none, while its trace event still said deterministic. The trace now derives grader_kind the same way the case result does, the trace shape accepts either value, and the run-result contract requires every trace event to agree with its case on grader kind and family. The test asserts it on the model-only family. Program and driver digests re-pinned. 13/13 focused checks pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two review findings on the previous head. A seed id could be a valid 128-character id, yet "evals.run." + id then broke the same limit, so the run was built and only refused afterwards. The seed-set shape now requires the prefixed id to fit, so the input is refused before any case runs. validate-run-result accepted any well-formed sha256 in catalog_ref, seed_set_ref, and evaluator. It now takes the exact digests the launcher and driver already hold and requires the refs, the seed id, and the run id to match them. A saved result from another seed set, catalog, or evaluator no longer validates. Program and driver digests re-pinned. Regressions: over-long seed id -> E_SHAPE, longest fitting id runs, and a run result with any moved ref or foreign seed id is rejected. scripts/test/evals-framework.test.sh 15/15, portable-core-schema guard 0 failures, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The new run-result binding check hardcoded the corrective generation id, which the schema activation guard only allows on a closed list of paths. The test now reads the generation from the launcher's own pin, so the guard's allowlist stays unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: the validator only checked shapes and top-level refs, so a candidate with altered verdicts, summary, or trace order still validated. The launcher now hands the driver's recorded observations to the independent re-validation, and validate-run-result rebuilds the run result from the same catalog, evaluator, seed set, and observations and requires the candidate to equal it byte for byte, the same pattern the state scanner uses for its observation check. Program and driver digests re-pinned. Regressions: a flipped verdict with a consistent summary, a reordered trace, and an altered recorded observation are all rejected. scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…y ran Review finding on the previous head: the evaluator shape only checked those refs for form, so an archived result could name a program, driver, or catalog that never ran if its evaluator digest was recomputed. The launcher and driver now pass the exact program, driver, and catalog digests they verified, and evaluator_shape requires the refs to equal them. The driver hashes its own snapshot for the driver digest and carries the program pin in one constant. Program and driver digests re-pinned. Regression: for each of the three refs, a result with the ref changed and the evaluator digest recomputed is rejected. scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Second eval unit. The "repeated, cancelled, and missed events" family
moves from declared to seeded. Its twelve cases in
evals/v1/seed-set-events.json carry canonical orchestrator state
snapshots and replay them through the real inactive state scanner
(orchestrator/v1/scan-state.sh), staged inside the private evals runtime
at pinned digests so the scanner reads its core and jq from the runtime
mirror only.
Cases: a missed attempt deadline strands; a cancelled stage stays
terminal even when the target moves; a failed stage is retryable until
its retry limit and blocked after it; a moved target is stale; a
repeated stage, a live attempt beside a terminal result, and a snapshot
from another revision are refused with one token.
Contracts: a seed set now names one of two active seed sources and its
shape follows the source. Scanner cases carry expected_revision and a
snapshot pair; expectations are observed{class,action,reason_id} or
rejected{error_token}. Observations, case results (subject_ref), trace
tool refs, and the run result's seed_source follow the source, and the
evaluator pins the scanner closure. Grading adds
evals.classification-mismatch. Unit 1 stage-run seeds are unchanged.
Proof: scripts/test/evals-events.test.sh 9/9,
scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Third eval unit. The "repeated, cancelled, and missed events" family gains a second seed source: evals/v1/seed-set-plans.json replays thirteen observation-plus-ledger bundles through the real inactive reconciliation planner (orchestrator/v1/reconciliation-plan.jq), staged in the private runtime at a pinned digest. Cases: a repeated delivery of the same key is a redelivery, not a second effect; an acknowledged delivery is suppressed; a failed stage is planned as the next attempt and refused past the retry limit; a stranded attempt is recovered; deliveries beyond the in-flight limit are deferred with redeliveries first; operator-only classes produce messages and no deliveries; duplicate classifications or ledger entries are refused. Contracts: a catalog family now lists the seed sources that feed it (seed_sources), and a seed set may only feed families the catalog says draw on its source. Planner cases carry an input bundle pair and are graded on a plan summary (deliveries, deferred, suppressed, operator messages). Grading adds evals.plan-mismatch. The unit 1 model-only check now asserts the refusal of a misfiled family and exercises the inconclusive grade through the program with a catalog that does seed that family. Proof: scripts/test/evals-plans.test.sh 9/9, scripts/test/evals-events.test.sh 9/9, scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean, rename gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: the plan summary kept only the initiative id per delivery and reduced deferred, suppressed, and operator-handled items to counts, so a planner that deferred the wrong stage with the same count would still pass. The summary now carries the full delivery key (stage key, request digest, operation, attempt) for every delivery, deferral, and suppression, each deferral's and suppression's reason id, and the stage key, class, and action of every operator message. Expectations were regenerated with those identities and each was checked against the real planner before being written. Program and driver digests re-pinned. Regression: an expectation that defers the other stage with identical counts is graded failed. scripts/test/evals-plans.test.sh 9/9, evals-events 9/9, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…aluator
Fourth eval unit. The "protected-path, credential, network, and
publisher boundaries" family moves from declared to seeded. Its sixteen
cases in evals/v1/seed-set-boundaries.json replay execution-environment
claims through the real inactive sandbox-policy evaluator
(control/v1/evaluate-sandbox.sh), staged with its policy-set validator
and policy in the private evals runtime at pinned digests. The
evaluator finds its jq on PATH and its policy beside itself; both come
from the runtime, nothing from the repo.
Cases: a cleared, allowlisted sandbox is satisfied; a publisher role, an
allowed network or endpoint, a tool that asks for network, an inherited
environment or a secret-looking variable, a credential reference, a
write root outside the fixed sandbox, any target or external write, and
a stale policy-set reference are violated with their exact reason sets;
unknown network or sensitive-material state is inconclusive; a claim
with an unknown field or a wildcard path is refused. Each expectation
was checked against the real evaluator before being written.
Contracts: seed source control.sandbox-policy.v1; sandbox cases carry
policy-set, duty, and claim pairs; expectations are
evaluated{verdict,reason_ids} or rejected{error_token}; the evaluator
pins a control closure of six files; grading adds
evals.verdict-mismatch, and an incomplete reason set fails.
Proof: scripts/test/evals-boundaries.test.sh 9/9,
scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fifth eval unit. The "adapter contract compliance" family moves from
declared to seeded. Its thirty-seven cases in
evals/v1/seed-set-adapters.json replay recorded provider snapshots with
caller bindings through the real inactive default normalizers for the
GitHub forge, GitHub Actions CI, and Codex native reviewer adapters,
each staged in the private evals runtime at a pinned digest. The
normalizers are pure jq with no modules and read nothing else.
Cases: open, blocked, merged, and closed change requests; queued,
running, passed, failed, cancelled, timed-out, and action-required runs;
clean, findings, dismissed, timed-out, and failed reviews; stale
bindings named exactly, singly and in sets; incomplete or unknown
provider state kept inconclusive; provider text that can never decide a
state; and malformed envelopes, bindings, or snapshots refused with the
normalizer's own error id. Each expectation was checked against the real
normalizer before being written.
Contracts: seed source adapters.provider-normalizers.v1; cases name the
one normalizer they run and carry an opaque input pair; expectations are
normalized{state,reason_id,stale_bindings} or rejected{error_token};
each trace event names the exact normalizer that ran, and the evaluator
pins a three-file adapter closure. Grading adds
evals.normalization-mismatch; an incomplete stale-binding set fails.
Proof: scripts/test/evals-adapters.test.sh 9/9, evals-framework 15/15,
evals-events 9/9, evals-plans 9/9, evals-boundaries 9/9, shellcheck
0.11.0 clean, rename gate clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ults Sixth eval unit. `run-evals.sh dashboard OBSERVED_AT SEED RESULT [SEED RESULT]...` aggregates one to sixteen run results into one canonical dashboard document. The launcher snapshots each pair and binds the driver to a digest manifest; the driver re-validates every result against its own seed set and this exact program before it counts, then builds the dashboard and re-validates it; the launcher repeats both checks independently after the driver exits. The dashboard reports seeded-family coverage, per-family and overall pass, fail, and inconclusive counts, and the recovery evidence the events family produced (missed attempts recovered, cancellations kept terminal, repeats redelivered once or suppressed, retry limits enforced, malformed or over-limit events refused). Every number is a count over the results handed in. Latency, cost, and token telemetry are recorded absent because no live run exists, and the operating-flow metrics the roadmap names are recorded absent with the reason that there is no operating history yet. Nothing is estimated. Fail closed: a result with an altered verdict, a result paired with the wrong seed set, a duplicate result, a non-canonical or truncated result, and unpaired arguments are each refused with one token. Proof: scripts/test/evals-dashboard.test.sh 6/6, evals-framework 15/15, evals-events 9/9, evals-plans 9/9, evals-boundaries 9/9, evals-adapters 9/9, shellcheck 0.11.0 clean, rename gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deploying ystack with
|
| Latest commit: |
9ae96db
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://2da3515a.fabrica-6yx.pages.dev |
| Branch Preview URL: | https://ystack-roadmap-5-evals-dashb.fabrica-6yx.pages.dev |
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: 570c50a Posted verbatim by The new dashboard path rejects valid inputs in common cases: archived run results with their own timestamps and multiple valid seed files without trailing whitespace. These are functional issues in the added dashboard operation. Full review comments:
|
… apart Two review findings on the previous head. A run result recorded at a different time than the dashboard was rebuilt with the dashboard's timestamp and refused. The driver and the launcher now rebuild each result at the observed_at the result itself records; the dashboard keeps this invocation's time. Seed and result files were appended byte for byte, so a file without a trailing newline ran into the next document. Each is now re-emitted as one canonical document per line. Driver digest re-pinned. Regression: a dashboard over a result recorded five days earlier, from a seed file with no trailing newline, beside a current result, is accepted and lists both times. scripts/test/evals-dashboard.test.sh 7/7, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: 772ea78 Posted verbatim by The dashboard validation trusts a digest embedded in the untrusted result, allowing a tampered run result to pass re-validation and be included. This breaks the integrity guarantee added by the patch. Review comment:
|
Review finding on the previous head: the dashboard's pair validation read the evaluator digest from the result under check, so a result with a rewritten evaluator digest and matching trace refs still counted. The driver and the launcher now hash the embedded evaluator content themselves and validate against that. Driver digest re-pinned. Regression: a result whose evaluator digest and trace refs are rewritten to a claimed value is refused as E_RELATION. scripts/test/evals-dashboard.test.sh 7/7, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: 948cfa7 Posted verbatim by I did not find any discrete, actionable bugs in the dashboard implementation or its integration with the existing eval runner. The new paths preserve the existing validation model and add appropriate tests and restore-manifest coverage. |
yihanzhu
changed the base branch from
ystack/roadmap-5/evals-adapters-v1
to
main
September 5, 2026 22:07
Review nit on the stack: the fixture copied only evals, core, and scripts, so the launcher refused it for missing component files before reaching the edit. It now copies every component directory the launcher stages. evals-framework 15/15. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the alternative-adapters PR, which carries this code: the dashboard's canonical check compared two command substitutions, and those strip trailing newlines, so a supplied result with trailing whitespace or no final newline passed as canonical. The driver now renders the canonical form to a file and compares bytes with cmp; the launcher does the same for its own output. Driver digest re-pinned. Regressions: a result with trailing whitespace and a result with no final newline are each refused as E_CANONICAL. scripts/test/evals-dashboard.test.sh 7/7, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: d05f0b4 Posted verbatim by The dashboard feature rejects a class of valid run results containing inconclusive cases from missing observations. That breaks the advertised aggregation of run-result documents. Review comment:
|
Review finding: the dashboard extracted every case's observation value, so a result built from a partial observation set (which build-run-result supports by marking the unobserved case inconclusive) produced a null observation and was refused. The driver and the launcher now hand back only present observations, and the rebuild reproduces the absent one. Driver digest re-pinned. Regression: a result built from a partial observation set yields one inconclusive case, is accepted by the dashboard, and is counted as inconclusive. evals-framework 16/16, evals-dashboard 7/7, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: 6faca65 Posted verbatim by I did not identify any discrete correctness, security, or maintainability issues introduced by the diff. The dashboard path appears to validate inputs, preserve existing run behavior, and update the restore/test manifest consistently. |
Review finding: the dashboard derived observations from the supplied result itself, so a forged result whose observations, verdicts, and summary agreed with each other validated without any replay. The dashboard now replays every seed set inside the same private runtime at the result's own recorded time, through the same driver path a run uses, and accepts a result only when the replay reproduces it byte for byte. The launcher re-validates each supplied result against the driver's fresh observations and this runtime's evaluator. Nothing embedded in a supplied result is trusted, so a result from another framework version or platform never counts either. The run path is one function now, called once for a run and once per pair for a dashboard. The dashboard loop keeps its own variable names so the replay's globals cannot clobber its comparison targets. Driver digest re-pinned. Regressions: a forged self-consistent result is refused as E_RELATION; a partially observed result, which the program grades inconclusive, is refused because the replay observes every case. evals-framework 16/16, evals-dashboard 7/7, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: 9ae96db Posted verbatim by The implementation appears functionally sound; the only issue found is a misleading test comment that does not affect runtime behavior or test outcomes. Review comment:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evals/v1/run-evals.sh dashboard OBSERVED_AT SEED-SET.json RUN-RESULT.json [...]aggregates one to sixteen run results into one canonicaleval_dashboarddocument.evals.no-live-runs), and the fifteen operating-flow metrics the roadmap names are recorded absent (evals.no-operating-history). Nothing is estimated.main; no rebase of published work.Honesty of the numbers
Counts only, all derived from validated results. No live measurement exists, so no latency, cost, or flow figure is invented; each absent metric carries its reason. The single-result dashboard proves the numbers are that run's alone.
Inactive boundary
Unchanged: no candidate, adapter, model, credential, network, write outside private scratch, qualification, authority, or profile activation. The dashboard is a read-only aggregation of documents the caller supplies.
Exact candidate
570c50aystack/roadmap-5/evals-adapters-v1ata916980(Roadmap item 5: Seed the adapter family from the default normalizers #237)evals/v1/evals.jq,evals/v1/evals-driver.sh,evals/v1/evals-launcher.sh,scripts/test/evals-dashboard.test.sh(new),scripts/test/evals-framework.test.sh,README.md,RESTORE.md,ci/required-files.txtCurrent-head local proof
bash scripts/test/evals-dashboard.test.sh— 6/6 passed (five runs plus dashboards)evals-framework15/15,evals-events9/9,evals-plans9/9,evals-boundaries9/9,evals-adapters9/9Fresh CI and a fresh independent review are required for this head.