Roadmap item 6: Prove the alternative forge and harness against the adapter evals - #244
Merged
Merged
Conversation
One offline, deterministic eval pass over a caller-supplied seed set, returning one canonical run result. - evals/v1/eval-catalog.json names the nine regression families the roadmap requires before any autonomous write, each with its grader kinds, trial policy, and core evidence kinds. Two are seeded here; the other seven are declared and wait for their own seeds. - evals/v1/seed-set.json replays eight canonical core-v2 stage runs through the real portable core (scripts/core-contract.sh validate-stage-run): stale and moved artifacts, and empty, fake, and degraded review evidence. The core is the only judge; the framework records accept/reject plus token and grades that against expectation. - A wrong expectation grades failed. A family only a model or human can grade stays inconclusive. Malformed, moved, or edited inputs fail closed with one token. - Bootstrap, launcher, and driver mirror the state scanner: env -i, pinned jq 1.6 (Rosetta on arm64), private runtime snapshots with exact digests for program, catalog, driver, and the core closure, and an independent re-validation of the delivered document. - Trace events carry the Observability shape (tool, adapter, gate, identity, latency, cost). Latency and cost are absent in this unit: no model runs, so nothing honest is charged. Inactive boundary: no candidate or adapter runs, no model call, no credential or network, no write outside scratch, no qualification, no profile activation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
portable-core-schema.test.sh keeps a closed allowlist of tracked paths that may name the corrective core generation or import the schema module. The framework's program, driver, and launcher now appear there, as every prior component did when it landed. No guard rule is weakened; three paths are added to two closed lists. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review P2: a case in a family without a deterministic grader was graded inconclusive with grader_kind none, while its trace event still said deterministic. The trace now derives grader_kind the same way the case result does, the trace shape accepts either value, and the run-result contract requires every trace event to agree with its case on grader kind and family. The test asserts it on the model-only family. Program and driver digests re-pinned. 13/13 focused checks pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two review findings on the previous head. A seed id could be a valid 128-character id, yet "evals.run." + id then broke the same limit, so the run was built and only refused afterwards. The seed-set shape now requires the prefixed id to fit, so the input is refused before any case runs. validate-run-result accepted any well-formed sha256 in catalog_ref, seed_set_ref, and evaluator. It now takes the exact digests the launcher and driver already hold and requires the refs, the seed id, and the run id to match them. A saved result from another seed set, catalog, or evaluator no longer validates. Program and driver digests re-pinned. Regressions: over-long seed id -> E_SHAPE, longest fitting id runs, and a run result with any moved ref or foreign seed id is rejected. scripts/test/evals-framework.test.sh 15/15, portable-core-schema guard 0 failures, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The new run-result binding check hardcoded the corrective generation id, which the schema activation guard only allows on a closed list of paths. The test now reads the generation from the launcher's own pin, so the guard's allowlist stays unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: the validator only checked shapes and top-level refs, so a candidate with altered verdicts, summary, or trace order still validated. The launcher now hands the driver's recorded observations to the independent re-validation, and validate-run-result rebuilds the run result from the same catalog, evaluator, seed set, and observations and requires the candidate to equal it byte for byte, the same pattern the state scanner uses for its observation check. Program and driver digests re-pinned. Regressions: a flipped verdict with a consistent summary, a reordered trace, and an altered recorded observation are all rejected. scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…y ran Review finding on the previous head: the evaluator shape only checked those refs for form, so an archived result could name a program, driver, or catalog that never ran if its evaluator digest was recomputed. The launcher and driver now pass the exact program, driver, and catalog digests they verified, and evaluator_shape requires the refs to equal them. The driver hashes its own snapshot for the driver digest and carries the program pin in one constant. Program and driver digests re-pinned. Regression: for each of the three refs, a result with the ref changed and the evaluator digest recomputed is rejected. scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Second eval unit. The "repeated, cancelled, and missed events" family
moves from declared to seeded. Its twelve cases in
evals/v1/seed-set-events.json carry canonical orchestrator state
snapshots and replay them through the real inactive state scanner
(orchestrator/v1/scan-state.sh), staged inside the private evals runtime
at pinned digests so the scanner reads its core and jq from the runtime
mirror only.
Cases: a missed attempt deadline strands; a cancelled stage stays
terminal even when the target moves; a failed stage is retryable until
its retry limit and blocked after it; a moved target is stale; a
repeated stage, a live attempt beside a terminal result, and a snapshot
from another revision are refused with one token.
Contracts: a seed set now names one of two active seed sources and its
shape follows the source. Scanner cases carry expected_revision and a
snapshot pair; expectations are observed{class,action,reason_id} or
rejected{error_token}. Observations, case results (subject_ref), trace
tool refs, and the run result's seed_source follow the source, and the
evaluator pins the scanner closure. Grading adds
evals.classification-mismatch. Unit 1 stage-run seeds are unchanged.
Proof: scripts/test/evals-events.test.sh 9/9,
scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Third eval unit. The "repeated, cancelled, and missed events" family gains a second seed source: evals/v1/seed-set-plans.json replays thirteen observation-plus-ledger bundles through the real inactive reconciliation planner (orchestrator/v1/reconciliation-plan.jq), staged in the private runtime at a pinned digest. Cases: a repeated delivery of the same key is a redelivery, not a second effect; an acknowledged delivery is suppressed; a failed stage is planned as the next attempt and refused past the retry limit; a stranded attempt is recovered; deliveries beyond the in-flight limit are deferred with redeliveries first; operator-only classes produce messages and no deliveries; duplicate classifications or ledger entries are refused. Contracts: a catalog family now lists the seed sources that feed it (seed_sources), and a seed set may only feed families the catalog says draw on its source. Planner cases carry an input bundle pair and are graded on a plan summary (deliveries, deferred, suppressed, operator messages). Grading adds evals.plan-mismatch. The unit 1 model-only check now asserts the refusal of a misfiled family and exercises the inconclusive grade through the program with a catalog that does seed that family. Proof: scripts/test/evals-plans.test.sh 9/9, scripts/test/evals-events.test.sh 9/9, scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean, rename gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: the plan summary kept only the initiative id per delivery and reduced deferred, suppressed, and operator-handled items to counts, so a planner that deferred the wrong stage with the same count would still pass. The summary now carries the full delivery key (stage key, request digest, operation, attempt) for every delivery, deferral, and suppression, each deferral's and suppression's reason id, and the stage key, class, and action of every operator message. Expectations were regenerated with those identities and each was checked against the real planner before being written. Program and driver digests re-pinned. Regression: an expectation that defers the other stage with identical counts is graded failed. scripts/test/evals-plans.test.sh 9/9, evals-events 9/9, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…aluator
Fourth eval unit. The "protected-path, credential, network, and
publisher boundaries" family moves from declared to seeded. Its sixteen
cases in evals/v1/seed-set-boundaries.json replay execution-environment
claims through the real inactive sandbox-policy evaluator
(control/v1/evaluate-sandbox.sh), staged with its policy-set validator
and policy in the private evals runtime at pinned digests. The
evaluator finds its jq on PATH and its policy beside itself; both come
from the runtime, nothing from the repo.
Cases: a cleared, allowlisted sandbox is satisfied; a publisher role, an
allowed network or endpoint, a tool that asks for network, an inherited
environment or a secret-looking variable, a credential reference, a
write root outside the fixed sandbox, any target or external write, and
a stale policy-set reference are violated with their exact reason sets;
unknown network or sensitive-material state is inconclusive; a claim
with an unknown field or a wildcard path is refused. Each expectation
was checked against the real evaluator before being written.
Contracts: seed source control.sandbox-policy.v1; sandbox cases carry
policy-set, duty, and claim pairs; expectations are
evaluated{verdict,reason_ids} or rejected{error_token}; the evaluator
pins a control closure of six files; grading adds
evals.verdict-mismatch, and an incomplete reason set fails.
Proof: scripts/test/evals-boundaries.test.sh 9/9,
scripts/test/evals-framework.test.sh 15/15, shellcheck 0.11.0 clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fifth eval unit. The "adapter contract compliance" family moves from
declared to seeded. Its thirty-seven cases in
evals/v1/seed-set-adapters.json replay recorded provider snapshots with
caller bindings through the real inactive default normalizers for the
GitHub forge, GitHub Actions CI, and Codex native reviewer adapters,
each staged in the private evals runtime at a pinned digest. The
normalizers are pure jq with no modules and read nothing else.
Cases: open, blocked, merged, and closed change requests; queued,
running, passed, failed, cancelled, timed-out, and action-required runs;
clean, findings, dismissed, timed-out, and failed reviews; stale
bindings named exactly, singly and in sets; incomplete or unknown
provider state kept inconclusive; provider text that can never decide a
state; and malformed envelopes, bindings, or snapshots refused with the
normalizer's own error id. Each expectation was checked against the real
normalizer before being written.
Contracts: seed source adapters.provider-normalizers.v1; cases name the
one normalizer they run and carry an opaque input pair; expectations are
normalized{state,reason_id,stale_bindings} or rejected{error_token};
each trace event names the exact normalizer that ran, and the evaluator
pins a three-file adapter closure. Grading adds
evals.normalization-mismatch; an incomplete stale-binding set fails.
Proof: scripts/test/evals-adapters.test.sh 9/9, evals-framework 15/15,
evals-events 9/9, evals-plans 9/9, evals-boundaries 9/9, shellcheck
0.11.0 clean, rename gate clean.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ults Sixth eval unit. `run-evals.sh dashboard OBSERVED_AT SEED RESULT [SEED RESULT]...` aggregates one to sixteen run results into one canonical dashboard document. The launcher snapshots each pair and binds the driver to a digest manifest; the driver re-validates every result against its own seed set and this exact program before it counts, then builds the dashboard and re-validates it; the launcher repeats both checks independently after the driver exits. The dashboard reports seeded-family coverage, per-family and overall pass, fail, and inconclusive counts, and the recovery evidence the events family produced (missed attempts recovered, cancellations kept terminal, repeats redelivered once or suppressed, retry limits enforced, malformed or over-limit events refused). Every number is a count over the results handed in. Latency, cost, and token telemetry are recorded absent because no live run exists, and the operating-flow metrics the roadmap names are recorded absent with the reason that there is no operating history yet. Nothing is estimated. Fail closed: a result with an altered verdict, a result paired with the wrong seed set, a duplicate result, a non-canonical or truncated result, and unpaired arguments are each refused with one token. Proof: scripts/test/evals-dashboard.test.sh 6/6, evals-framework 15/15, evals-events 9/9, evals-plans 9/9, evals-boundaries 9/9, evals-adapters 9/9, shellcheck 0.11.0 clean, rename gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First alternative forge. adapters/gitlab-forge/v1/normalize.jq validates one untrusted GitLab merge-request snapshot against caller-supplied project, merge-request iid, head, base, bot-user, time, instruction, and config bindings and returns the same canonical generic observation the GitHub forge returns: open-ready, open-blocked, closed-unmerged, merged, stale, or inconclusive, with the same output keys, effect boundary, and stale-binding shape, so a profile can swap one forge for the other. GitLab vocabulary stays at the edge: a locked request and a checking or unchecked merge status are inconclusive, a merged request is never also closed, a closed or merged request carries no merge status, and the acting identity is the bot user the integration runs as, since GitLab has no app id. Provider metadata stays opaque data; GitHub-shaped states, mergeability values, and trust contexts are refused. Pure jq, offline, unqualified: no GitLab or CLI call, credential, project or merge-request change, authority, qualification, or profile activation. The test proves the contract equals the GitHub forge's output contract key for key. Proof: scripts/test/default-gitlab-forge-adapter.test.sh 54/54, shellcheck 0.11.0 clean, rename gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The test imports the public schema module to check reference shapes, the same way the GitHub forge test does, so it joins that closed allowlist. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… apart Two review findings on the previous head. A run result recorded at a different time than the dashboard was rebuilt with the dashboard's timestamp and refused. The driver and the launcher now rebuild each result at the observed_at the result itself records; the dashboard keeps this invocation's time. Seed and result files were appended byte for byte, so a file without a trailing newline ran into the next document. Each is now re-emitted as one canonical document per line. Driver digest re-pinned. Regression: a dashboard over a result recorded five days earlier, from a seed file with no trailing newline, beside a current result, is accepted and lists both times. scripts/test/evals-dashboard.test.sh 7/7, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: the snapshot used an invented merge-status vocabulary, so an ordinary GitLab merge request could not be normalized. The snapshot now carries detailed_merge_status with GitLab's documented values. mergeable is open-ready; the fourteen blocking values are open-blocked; the four transitional values are inconclusive (gitlab.merge-status-unsettled); not_open belongs only to a closed or merged request. An invented value, the legacy merge_status field, a GitHub-shaped value, or not_open on an open request is refused. scripts/test/default-gitlab-forge-adapter.test.sh 71/71, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: the dashboard's pair validation read the evaluator digest from the result under check, so a result with a rewritten evaluator digest and matching trace refs still counted. The driver and the launcher now hash the embedded evaluator content themselves and validate against that. Driver digest re-pinned. Regression: a result whose evaluator digest and trace refs are rewritten to a claimed value is refused as E_RELATION. scripts/test/evals-dashboard.test.sh 7/7, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: security_policy_violations, a documented GitLab detailed_merge_status value, was refused as an invalid snapshot. The blocking set now carries all twenty documented blocking values, including security_policy_violations, status_checks_must_pass, merge_request_blocked, merge_time, locked_paths, and locked_lfs_files; each normalizes an open request to open-blocked. scripts/test/default-gitlab-forge-adapter.test.sh 76/76, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: ci_still_running was normalized to open-blocked, but it settles on its own once the pipeline finishes, like checking or preparing. It is now transitional and normalizes to inconclusive (gitlab.merge-status-unsettled); ci_must_pass stays blocking because it names a failed requirement. scripts/test/default-gitlab-forge-adapter.test.sh 76/76, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First alternative harness. adapters/codex-cli-producer/v1/normalize.jq is the Claude Code producer normalizer with only the harness identity swapped: the snapshot kind and content id, the manifest id, the recorded snapshot fact, the adapter id, and the model provider the binding must name (openai). Every trust relation, snapshot relation, state, reason, and the generic observation are the same, so a profile can select either harness under one core contract. The test proves the two programs differ only in those six tokens (the Claude program with the tokens substituted is byte-identical to this one), runs the full producer contract suite against the new identity, and refuses a snapshot, recorded fact, or provider from the other harness. Inactive and unqualified: no Codex or model call, credential, network, target write, publish, or profile activation. No manifest ships here. Proof: scripts/test/default-codex-cli-producer-adapter.test.sh 46/46, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Seventh eval unit. The "approval invalidation and no push after approval" family moves from declared to seeded. Its sixteen cases in evals/v1/seed-set-approvals.json each carry a whole decision tuple (policy set, core request, resolved profile, result, duty evaluation, decision claim) and replay it through the real inactive risk-gates evaluator (control/v1/evaluate-risk-gates.sh), which regenerates the duty-separation evaluation and validates the core tuple from a mirror built out of the private evals runtime. The duty-separation and risk-gates files join the pinned control closure (now fourteen files). Cases: a request whose basis moved after the decision is violated (decision.stale); a decision recorded after the request, a missing, rejected, downgraded, wrong-role, wrong-kind, unbound-actor, unbound, ambiguous, or malformed claim is violated with its exact reason set; a duty violation is violated; honest routine, high, and bootstrap accept claims stay inconclusive because no qualified decision-provenance adapter exists; a forged duty evaluation is refused with E_DUTY. Each verdict and primary reason was hand-written and checked against the real evaluator before the full reason set was recorded. Contracts: seed source control.risk-gates.v1 shares the control evaluator expectation and observation shapes with the sandbox source, each bound to its own error-token set. Proof: scripts/test/evals-approvals.test.sh 9/9, and evals-framework 15/15, evals-events 9/9, evals-plans 9/9, evals-boundaries 9/9, evals-adapters 9/9, evals-dashboard 7/7, shellcheck 0.11.0 clean, rename gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The seed set embeds control policy sets that name the selected core generation by id, so the guard must know this tracked path may carry it, as it already knows for the eval program, driver, and launcher. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding on the previous head: the risk-gates shapes reused the sandbox verdict set, which includes satisfied, so an archived risk-gates result claiming a satisfied verdict could pass shape validation. The control evaluation, expectation, and observation shapes now take the evaluator's own verdict set: sandbox keeps inconclusive, satisfied, and violated; risk gates allow only inconclusive and violated, matching the evaluator, which has no qualified decision provenance and so no satisfied result. Program and driver digests re-pinned. Regression: a seed claiming a satisfied risk-gates verdict is refused as E_SHAPE. scripts/test/evals-approvals.test.sh 9/9, evals-boundaries 9/9, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The wrong-expectation check claimed satisfied, which the risk-gates shape now refuses before grading; it claims violated with a plausible reason instead and is graded failed as intended. The tampered-program fixture copies every component directory the launcher stages. evals-approvals 9/9, evals-framework 15/15. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…rmalizer-v1' into ystack/roadmap-6/evals-alternative-adapters-v1
…cer-normalizer-v1' into ystack/roadmap-6/evals-alternative-adapters-v1
…dapter evals The adapter contract compliance family now also replays the GitLab forge normalizer and the Codex CLI producer normalizer through the same contract and safety evals as the GitHub defaults. evals/v1/seed-set- adapters.json grows from 37 to 63 cases: fourteen GitLab merge-request cases (ready, conflict, security-policy block, merged, closed, locked, still-running pipeline, single and multiple stale bindings, provider text that cannot decide, GitHub-shaped state, legacy merge_status, and GitHub-shaped trust context refused) and twelve Codex CLI producer cases (changed, no-change, provider failure, timeout, degraded, stale inputs, incomplete metadata, moved attempt, other-harness provider, unknown state, moved untrusted snapshot, and caller manifest ceiling refused). Each expectation was checked against the real normalizer before it was written. The adapter closure pins five normalizers; every trace event names the one that ran. Producer normalizers report no stale bindings, so their recorded set is empty. The family suites' tampered-program fixtures now stage every component directory, so the edit is the only stale thing. Proof: evals-adapters 9/9, evals-framework 15/15, evals-dashboard 7/7, evals-events 9/9, evals-plans 9/9, evals-boundaries 9/9, evals-approvals 9/9, shellcheck 0.11.0 clean, rename gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deploying ystack with
|
| Latest commit: |
6fda895
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://fc0c6b55.fabrica-6yx.pages.dev |
| Branch Preview URL: | https://ystack-roadmap-6-evals-alter.fabrica-6yx.pages.dev |
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: cbe3862 Posted verbatim by The dashboard path intends to reject non-canonical run-result inputs, but the current check accepts byte variants that differ only in trailing whitespace/newlines. This weakens the evidence integrity of archived eval results. Review comment:
|
Review finding on the alternative-adapters PR, which carries this code: the dashboard's canonical check compared two command substitutions, and those strip trailing newlines, so a supplied result with trailing whitespace or no final newline passed as canonical. The driver now renders the canonical form to a file and compares bytes with cmp; the launcher does the same for its own output. Driver digest re-pinned. Regressions: a result with trailing whitespace and a result with no final newline are each refused as E_CANONICAL. scripts/test/evals-dashboard.test.sh 7/7, evals-framework 15/15, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s-alternative-adapters-v1
Review finding: the dashboard extracted every case's observation value, so a result built from a partial observation set (which build-run-result supports by marking the unobserved case inconclusive) produced a null observation and was refused. The driver and the launcher now hand back only present observations, and the rebuild reproduces the absent one. Driver digest re-pinned. Regression: a result built from a partial observation set yields one inconclusive case, is accepted by the dashboard, and is counted as inconclusive. evals-framework 16/16, evals-dashboard 7/7, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Review finding: the dashboard derived observations from the supplied result itself, so a forged result whose observations, verdicts, and summary agreed with each other validated without any replay. The dashboard now replays every seed set inside the same private runtime at the result's own recorded time, through the same driver path a run uses, and accepts a result only when the replay reproduces it byte for byte. The launcher re-validates each supplied result against the driver's fresh observations and this runtime's evaluator. Nothing embedded in a supplied result is trusted, so a result from another framework version or platform never counts either. The run path is one function now, called once for a run and once per pair for a dashboard. The dashboard loop keeps its own variable names so the replay's globals cannot clobber its comparison targets. Driver digest re-pinned. Regressions: a forged self-consistent result is refused as E_RELATION; a partially observed result, which the program grades inconclusive, is refused because the replay observes every case. evals-framework and evals-dashboard suites pass, shellcheck 0.11.0 clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s-alternative-adapters-v1
…s-alternative-adapters-v1 # Conflicts: # adapters/gitlab-forge/v1/normalize.jq # scripts/test/default-gitlab-forge-adapter.test.sh
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s-alternative-adapters-v1 # Conflicts: # README.md # RESTORE.md # evals/v1/evals-driver.sh # evals/v1/evals-launcher.sh # evals/v1/evals.jq # scripts/test/evals-approvals.test.sh # scripts/test/evals-dashboard.test.sh # scripts/test/evals-framework.test.sh
Owner
Author
Codex reviewer (cross-vendor, read-only)Reviewed-head: 6fda895 Posted verbatim by No discrete correctness issues were found in the diff. The added normalizer coverage, pinned hashes, schema allowances, and test expectation updates appear consistent with the changed seed set size. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evals/v1/seed-set-adapters.jsongrows from 37 to 63 cases: fourteen GitLab merge-request cases and twelve Codex CLI producer cases (listed in the commit). Each expectation was checked against the real normalizer before it was written.main, but this branch carries merges of the Roadmap item 6: Add inactive GitLab forge normalizer payload #239 and Roadmap item 6: Add inactive Codex CLI producer normalizer payload #240 branches (it needs their payloads), so until those two land the diff shows their files too. The merge chain lands them first; this PR then takes a merge ofmain.Honesty of the grade
Unchanged: the normalizer is the only judge; the framework records state, reason, and exact stale-binding set, or the normalizer's own error id, and grades that against the case.
Exact candidate
cbe3862evals/v1/evals.jq,evals/v1/evals-driver.sh,evals/v1/evals-launcher.sh,evals/v1/seed-set-adapters.json,scripts/test/evals-adapters.test.sh,scripts/test/evals-framework.test.sh,scripts/test/evals-dashboard.test.sh,scripts/test/evals-events.test.sh,scripts/test/evals-plans.test.sh,scripts/test/evals-boundaries.test.sh,scripts/test/evals-approvals.test.sh,README.mdCurrent-head local proof
Fresh CI and a fresh independent review are required for this head.