Roadmap item 5: Seed the approval family from the risk-gates evaluator - #243
Merged
Conversation
One offline, deterministic eval pass over a caller-supplied seed set, returning one canonical run result. - evals/v1/eval-catalog.json names the nine regression families the roadmap requires before any autonomous write, each with its grader kinds, trial policy, and core evidence kinds. Two are seeded here; the other seven are declared and wait for their own seeds. - evals/v1/seed-set.json replays eight canonical core-v2 stage runs through the real portable core (scripts/core-contract.sh validate-stage-run): stale and moved artifacts, and empty, fake, and degraded review evidence. The core is the only judge; the framework records accept/reject plus token and grades that against expectation. - A wrong expectation grades failed. A family only a model or human can grade stays inconclusive. Malformed, moved, or edited inputs fail closed with one token. - Bootstrap, launcher, and driver mirror the state scanner: env -i, pinned jq 1.6 (Rosetta on arm64), private runtime snapshots with exact digests for program, catalog, driver, and the core closure, and an independent re-validation of the delivered document. - Trace events carry the Observability shape (tool, adapter, gate, identity, latency, cost). Latency and cost are absent in this unit: no model runs, so nothing honest is charged. Inactive boundary: no candidate or adapter runs, no model call, no credential or network, no write outside scratch, no qualification, no profile activation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
portable-core-schema.test.sh keeps a closed allowlist of tracked paths that may name the corrective core generation or import the schema module. The framework's program, driver, and launcher now appear there, as every prior component did when it landed. No guard rule is weakened; three paths are added to two closed lists. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review P2: a case in a family without a deterministic grader was graded inconclusive with grader_kind none, while its trace event still said deterministic. The trace now derives grader_kind the same way the case result does, the trace shape accepts either value, and the run-result contract requires every trace event to agree with its case on grader kind and family. The test asserts it on the model-only family. Program and driver digests re-pinned. 13/13 focused checks pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two review findings on the previous head. A seed id could be a valid 128-character id, yet "evals.run." + id then broke the same limit, so the run was built and only refused afterwards. The seed-set shape now requires the prefixed id to fit, so the input is refused before any case runs. validate-run-result accepted any well-formed sha256 in catalog_ref, seed_set_ref, and evaluator. It now takes the exact digests the launcher and driver already hold and requires the refs, the seed id, and the run id to match them. A saved result from another seed set, catalog, or evaluator no longer validates. Program and driver digests re-pinned. Regressions: over-long seed id -> E_SHAPE, longest fitting id runs, and a run result with any moved ref or foreign seed id is rejected. scripts/test/evals-framework.test.sh 15/15, portable-core-schema guard 0 failures, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The new run-result binding check hardcoded the corrective generation id, which the schema activation guard only allows on a closed list of paths. The test now reads the generation from the launcher's own pin, so the guard's allowlist stays unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: the validator only checked shapes and top-level refs, so a candidate with altered verdicts, summary, or trace order still validated. The launcher now hands the driver's recorded observations to the independent re-validation, and validate-run-result rebuilds the run result from the same catalog, evaluator, seed set, and observations and requires the candidate to equal it byte for byte, the same pattern the state scanner uses for its observation check. Program and driver digests re-pinned. Regressions: a flipped verdict with a consistent summary, a reordered trace, and an altered recorded observation are all rejected. scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…y ran Review finding on the previous head: the evaluator shape only checked those refs for form, so an archived result could name a program, driver, or catalog that never ran if its evaluator digest was recomputed. The launcher and driver now pass the exact program, driver, and catalog digests they verified, and evaluator_shape requires the refs to equal them. The driver hashes its own snapshot for the driver digest and carries the program pin in one constant. Program and driver digests re-pinned. Regression: for each of the three refs, a result with the ref changed and the evaluator digest recomputed is rejected. scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Second eval unit. The "repeated, cancelled, and missed events" family
moves from declared to seeded. Its twelve cases in
evals/v1/seed-set-events.json carry canonical orchestrator state
snapshots and replay them through the real inactive state scanner
(orchestrator/v1/scan-state.sh), staged inside the private evals runtime
at pinned digests so the scanner reads its core and jq from the runtime
mirror only.
Cases: a missed attempt deadline strands; a cancelled stage stays
terminal even when the target moves; a failed stage is retryable until
its retry limit and blocked after it; a moved target is stale; a
repeated stage, a live attempt beside a terminal result, and a snapshot
from another revision are refused with one token.
Contracts: a seed set now names one of two active seed sources and its
shape follows the source. Scanner cases carry expected_revision and a
snapshot pair; expectations are observed{class,action,reason_id} or
rejected{error_token}. Observations, case results (subject_ref), trace
tool refs, and the run result's seed_source follow the source, and the
evaluator pins the scanner closure. Grading adds
evals.classification-mismatch. Unit 1 stage-run seeds are unchanged.
Proof: scripts/test/evals-events.test.sh 9/9,
scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Third eval unit. The "repeated, cancelled, and missed events" family gains a second seed source: evals/v1/seed-set-plans.json replays thirteen observation-plus-ledger bundles through the real inactive reconciliation planner (orchestrator/v1/reconciliation-plan.jq), staged in the private runtime at a pinned digest. Cases: a repeated delivery of the same key is a redelivery, not a second effect; an acknowledged delivery is suppressed; a failed stage is planned as the next attempt and refused past the retry limit; a stranded attempt is recovered; deliveries beyond the in-flight limit are deferred with redeliveries first; operator-only classes produce messages and no deliveries; duplicate classifications or ledger entries are refused. Contracts: a catalog family now lists the seed sources that feed it (seed_sources), and a seed set may only feed families the catalog says draw on its source. Planner cases carry an input bundle pair and are graded on a plan summary (deliveries, deferred, suppressed, operator messages). Grading adds evals.plan-mismatch. The unit 1 model-only check now asserts the refusal of a misfiled family and exercises the inconclusive grade through the program with a catalog that does seed that family. Proof: scripts/test/evals-plans.test.sh 9/9, scripts/test/evals-events.test.sh 9/9, scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean, rename gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: the plan summary kept only the initiative id per delivery and reduced deferred, suppressed, and operator-handled items to counts, so a planner that deferred the wrong stage with the same count would still pass. The summary now carries the full delivery key (stage key, request digest, operation, attempt) for every delivery, deferral, and suppression, each deferral's and suppression's reason id, and the stage key, class, and action of every operator message. Expectations were regenerated with those identities and each was checked against the real planner before being written. Program and driver digests re-pinned. Regression: an expectation that defers the other stage with identical counts is graded failed. scripts/test/evals-plans.test.sh 9/9, evals-events 9/9, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…aluator
Fourth eval unit. The "protected-path, credential, network, and
publisher boundaries" family moves from declared to seeded. Its sixteen
cases in evals/v1/seed-set-boundaries.json replay execution-environment
claims through the real inactive sandbox-policy evaluator
(control/v1/evaluate-sandbox.sh), staged with its policy-set validator
and policy in the private evals runtime at pinned digests. The
evaluator finds its jq on PATH and its policy beside itself; both come
from the runtime, nothing from the repo.
Cases: a cleared, allowlisted sandbox is satisfied; a publisher role, an
allowed network or endpoint, a tool that asks for network, an inherited
environment or a secret-looking variable, a credential reference, a
write root outside the fixed sandbox, any target or external write, and
a stale policy-set reference are violated with their exact reason sets;
unknown network or sensitive-material state is inconclusive; a claim
with an unknown field or a wildcard path is refused. Each expectation
was checked against the real evaluator before being written.
Contracts: seed source control.sandbox-policy.v1; sandbox cases carry
policy-set, duty, and claim pairs; expectations are
evaluated{verdict,reason_ids} or rejected{error_token}; the evaluator
pins a control closure of six files; grading adds
evals.verdict-mismatch, and an incomplete reason set fails.
Proof: scripts/test/evals-boundaries.test.sh 9/9,
scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fifth eval unit. The "adapter contract compliance" family moves from
declared to seeded. Its thirty-seven cases in
evals/v1/seed-set-adapters.json replay recorded provider snapshots with
caller bindings through the real inactive default normalizers for the
GitHub forge, GitHub Actions CI, and Codex native reviewer adapters,
each staged in the private evals runtime at a pinned digest. The
normalizers are pure jq with no modules and read nothing else.
Cases: open, blocked, merged, and closed change requests; queued,
running, passed, failed, cancelled, timed-out, and action-required runs;
clean, findings, dismissed, timed-out, and failed reviews; stale
bindings named exactly, singly and in sets; incomplete or unknown
provider state kept inconclusive; provider text that can never decide a
state; and malformed envelopes, bindings, or snapshots refused with the
normalizer's own error id. Each expectation was checked against the real
normalizer before being written.
Contracts: seed source adapters.provider-normalizers.v1; cases name the
one normalizer they run and carry an opaque input pair; expectations are
normalized{state,reason_id,stale_bindings} or rejected{error_token};
each trace event names the exact normalizer that ran, and the evaluator
pins a three-file adapter closure. Grading adds
evals.normalization-mismatch; an incomplete stale-binding set fails.
Proof: scripts/test/evals-adapters.test.sh 9/9, evals-framework 15/15,
evals-events 9/9, evals-plans 9/9, evals-boundaries 9/9, shellcheck
0.11.0 clean, rename gate clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ults Sixth eval unit. `run-evals.sh dashboard OBSERVED_AT SEED RESULT [SEED RESULT]...` aggregates one to sixteen run results into one canonical dashboard document. The launcher snapshots each pair and binds the driver to a digest manifest; the driver re-validates every result against its own seed set and this exact program before it counts, then builds the dashboard and re-validates it; the launcher repeats both checks independently after the driver exits. The dashboard reports seeded-family coverage, per-family and overall pass, fail, and inconclusive counts, and the recovery evidence the events family produced (missed attempts recovered, cancellations kept terminal, repeats redelivered once or suppressed, retry limits enforced, malformed or over-limit events refused). Every number is a count over the results handed in. Latency, cost, and token telemetry are recorded absent because no live run exists, and the operating-flow metrics the roadmap names are recorded absent with the reason that there is no operating history yet. Nothing is estimated. Fail closed: a result with an altered verdict, a result paired with the wrong seed set, a duplicate result, a non-canonical or truncated result, and unpaired arguments are each refused with one token. Proof: scripts/test/evals-dashboard.test.sh 6/6, evals-framework 15/15, evals-events 9/9, evals-plans 9/9, evals-boundaries 9/9, evals-adapters 9/9, shellcheck 0.11.0 clean, rename gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… apart Two review findings on the previous head. A run result recorded at a different time than the dashboard was rebuilt with the dashboard's timestamp and refused. The driver and the launcher now rebuild each result at the observed_at the result itself records; the dashboard keeps this invocation's time. Seed and result files were appended byte for byte, so a file without a trailing newline ran into the next document. Each is now re-emitted as one canonical document per line. Driver digest re-pinned. Regression: a dashboard over a result recorded five days earlier, from a seed file with no trailing newline, beside a current result, is accepted and lists both times. scripts/test/evals-dashboard.test.sh 7/7, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: the dashboard's pair validation read the evaluator digest from the result under check, so a result with a rewritten evaluator digest and matching trace refs still counted. The driver and the launcher now hash the embedded evaluator content themselves and validate against that. Driver digest re-pinned. Regression: a result whose evaluator digest and trace refs are rewritten to a claimed value is refused as E_RELATION. scripts/test/evals-dashboard.test.sh 7/7, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Seventh eval unit. The "approval invalidation and no push after approval" family moves from declared to seeded. Its sixteen cases in evals/v1/seed-set-approvals.json each carry a whole decision tuple (policy set, core request, resolved profile, result, duty evaluation, decision claim) and replay it through the real inactive risk-gates evaluator (control/v1/evaluate-risk-gates.sh), which regenerates the duty-separation evaluation and validates the core tuple from a mirror built out of the private evals runtime. The duty-separation and risk-gates files join the pinned control closure (now fourteen files). Cases: a request whose basis moved after the decision is violated (decision.stale); a decision recorded after the request, a missing, rejected, downgraded, wrong-role, wrong-kind, unbound-actor, unbound, ambiguous, or malformed claim is violated with its exact reason set; a duty violation is violated; honest routine, high, and bootstrap accept claims stay inconclusive because no qualified decision-provenance adapter exists; a forged duty evaluation is refused with E_DUTY. Each verdict and primary reason was hand-written and checked against the real evaluator before the full reason set was recorded. Contracts: seed source control.risk-gates.v1 shares the control evaluator expectation and observation shapes with the sandbox source, each bound to its own error-token set. Proof: scripts/test/evals-approvals.test.sh 9/9, and evals-framework 15/15, evals-events 9/9, evals-plans 9/9, evals-boundaries 9/9, evals-adapters 9/9, evals-dashboard 7/7, shellcheck 0.11.0 clean, rename gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deploying ystack with
|
| Latest commit: |
9d48f24
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://7d910e96.fabrica-6yx.pages.dev |
| Branch Preview URL: | https://ystack-roadmap-5-evals-appro.fabrica-6yx.pages.dev |
The seed set embeds control policy sets that name the selected core generation by id, so the guard must know this tracked path may carry it, as it already knows for the eval program, driver, and launcher. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: 83037a0 Posted verbatim by The new risk-gates eval path accepts an impossible Review comment:
|
Review finding on the previous head: the risk-gates shapes reused the sandbox verdict set, which includes satisfied, so an archived risk-gates result claiming a satisfied verdict could pass shape validation. The control evaluation, expectation, and observation shapes now take the evaluator's own verdict set: sandbox keeps inconclusive, satisfied, and violated; risk gates allow only inconclusive and violated, matching the evaluator, which has no qualified decision provenance and so no satisfied result. Program and driver digests re-pinned. Regression: a seed claiming a satisfied risk-gates verdict is refused as E_SHAPE. scripts/test/evals-approvals.test.sh 9/9, evals-boundaries 9/9, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The wrong-expectation check claimed satisfied, which the risk-gates shape now refuses before grading; it claims violated with a plausible reason instead and is graded failed as intended. The tampered-program fixture copies every component directory the launcher stages. evals-approvals 9/9, evals-framework 15/15. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the alternative-adapters PR, which carries this code: the dashboard's canonical check compared two command substitutions, and those strip trailing newlines, so a supplied result with trailing whitespace or no final newline passed as canonical. The driver now renders the canonical form to a file and compares bytes with cmp; the launcher does the same for its own output. Driver digest re-pinned. Regressions: a result with trailing whitespace and a result with no final newline are each refused as E_CANONICAL. scripts/test/evals-dashboard.test.sh 7/7, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding: the dashboard extracted every case's observation value, so a result built from a partial observation set (which build-run-result supports by marking the unobserved case inconclusive) produced a null observation and was refused. The driver and the launcher now hand back only present observations, and the rebuild reproduces the absent one. Driver digest re-pinned. Regression: a result built from a partial observation set yields one inconclusive case, is accepted by the dashboard, and is counted as inconclusive. evals-framework 16/16, evals-dashboard 7/7, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding: the dashboard derived observations from the supplied result itself, so a forged result whose observations, verdicts, and summary agreed with each other validated without any replay. The dashboard now replays every seed set inside the same private runtime at the result's own recorded time, through the same driver path a run uses, and accepts a result only when the replay reproduces it byte for byte. The launcher re-validates each supplied result against the driver's fresh observations and this runtime's evaluator. Nothing embedded in a supplied result is trusted, so a result from another framework version or platform never counts either. The run path is one function now, called once for a run and once per pair for a dashboard. The dashboard loop keeps its own variable names so the replay's globals cannot clobber its comparison targets. Driver digest re-pinned. Regressions: a forged self-consistent result is refused as E_RELATION; a partially observed result, which the program grades inconclusive, is refused because the replay observes every case. evals-framework and evals-dashboard suites pass, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review nit: the comment said the dashboard must accept a partially observed result; the assertions require it to refuse one, because the dashboard replays every case. The comment now states that. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s-approvals-v1 # Conflicts: # README.md # RESTORE.md # ci/required-files.txt # evals/v1/eval-catalog.json # evals/v1/evals-driver.sh # evals/v1/evals-launcher.sh # evals/v1/evals.jq # scripts/test/evals-boundaries.test.sh # scripts/test/evals-dashboard.test.sh # scripts/test/evals-framework.test.sh # scripts/test/portable-core-schema.test.sh
yihanzhu
changed the base branch from
ystack/roadmap-5/evals-dashboard-v1
to
main
September 6, 2026 07:37
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: b0ac001 Posted verbatim by The code changes appear internally consistent, but the patch leaves restore-critical documentation with contradictory old and new eval counts/instructions. Those introduced documentation conflicts should be fixed before the patch is considered correct. Full review comments:
|
…ments Merging main kept the old family, path, and suite counts next to the updated ones in README.md and RESTORE.md, and listed the evals directory twice in the file map. Only the current statements remain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: 9d48f24 Posted verbatim by The changes consistently add the approvals seed set and risk-gates replay path, with updated hashes, catalog metadata, manifest entries, and tests. I did not find a blocking correctness issue in the diff. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evals/v1/seed-set-approvals.jsoneach carry a whole decision tuple (policy set, core request, resolved profile, result, duty evaluation, decision claim) and replay it through the real inactive risk-gates evaluator (control/v1/evaluate-risk-gates.sh), which regenerates the duty-separation evaluation and validates the core tuple from a mirror built out of the private evals runtime.decision.stale); a decision recorded after the request, a missing, rejected, downgraded, wrong-role, wrong-kind, unbound-actor, unbound, ambiguous, or malformed claim is violated with its exact reason set; a duty violation is violated; honest routine, high, and bootstrap accept claims stay inconclusive because no qualified decision-provenance adapter exists; a forged duty evaluation is refused withE_DUTY.decision.unbound).control.risk-gates.v1shares the control-evaluator expectation and observation shapes with the sandbox source, each bound to its own error-token set; the control closure grows to fourteen pinned files.main; no rebase of published work.Honesty of the grade
The risk-gates evaluator is the only judge. The framework records its verdict and exact reason set, or its one refusal token, and grades that against the case. A wrong verdict or an incomplete reason set is graded
failed.Inactive boundary
Unchanged: no candidate, adapter, model, credential, network, write outside private scratch, qualification, authority, or profile activation. The replayed evaluator is itself observation-only and cannot produce
satisfied. An edited evaluator is refused asE_STALEbefore anything runs.Exact candidate
83037a0ystack/roadmap-5/evals-dashboard-v1at948cfa7(Roadmap item 5: Add the flow/quality dashboard over validated run results #238)evals/v1/evals.jq,evals/v1/evals-driver.sh,evals/v1/evals-launcher.sh,evals/v1/eval-catalog.json,evals/v1/seed-set-approvals.json(new),scripts/test/evals-approvals.test.sh(new),scripts/test/evals-framework.test.sh,scripts/test/evals-boundaries.test.sh,scripts/test/evals-dashboard.test.sh,README.md,RESTORE.md,ci/required-files.txtCurrent-head local proof
bash scripts/test/evals-approvals.test.sh— 9/9 passedevals-framework15/15,evals-events9/9,evals-plans9/9,evals-boundaries9/9,evals-adapters9/9,evals-dashboard7/7Fresh CI and a fresh independent review are required for this head.