Skip to content

[Experimental] Second semantic-normalization smoke pin - AB#652973 (no product merge/release) - #940

Draft
Wenjie Fan (gggdttt) wants to merge 1 commit into
gggdttt-support-normalization-manifestsfrom
gggdttt-prepare-normalized-smoke-stack
Draft

Wenjie Fan (gggdttt) wants to merge 1 commit into
gggdttt-support-normalization-manifestsfrom
gggdttt-prepare-normalized-smoke-stack

Conversation

@gggdttt

Copy link
Copy Markdown
Collaborator

PREP ONLY: experimental second semantic-normalization smoke pin

AB#652973. No product merge or release intent. NOT_AUTHORIZED for execution. This prepares the third layer above #938; it is not a standalone change against main, a retry of the first combined smoke, or evidence of HTTP A/B causality. No workflow dispatch, reviewer generation, judge invocation, merge, or result publication was performed as part of this preparation.

Base: gggdttt-support-normalization-manifests at exact #938 head 646e2802bc89993e048c410a6f146bad3fc8237e. Initial HEAD, remote branch, and open draft #938 matched that SHA, with a clean worktree. #938 still targets #911 branch gggdttt-http-knowledge-ab-gate at 0cbb709caaf92fcb2017bcfa0b1c77185d37cbc6. Both lower PRs are untouched. Existing native stack 939 was read-only and remained ordered [911, 938]; the coordinator owns any guarded append of this PR. No stack metadata was created, recreated, or mutated here.

Head: 5cacfe09231352d152d8827c307585e85b15c391 on gggdttt-prepare-normalized-smoke-stack.

Exactly two substitutions

Replace da9236a6f43d47c37e0bd59ebced33b6fe73f277 with 7c7c12c2188dd7bee063f925b9d055d05075beff exactly once in each file:

  • .github/actions/install-agent-harnesses/action.yml: the default engine SHA in ENGINE_SHA.
  • tests/test_review_workflows.py: DEFAULT_ENGINE_SHA.

No changes to uploader support, manifest models, dataset/gold, fixtures, runtime, scorer, adapter, package version (0.14.0.dev1), CLI versions, model choices, workflow inputs, or publication gates. #938's strict normalization consumer and the two exact diagnostic selectors **/_review-report.raw.json and **/_review-source-bounds.json are inherited unchanged. All 14 existing diagnostic selectors, their order, seven-day diagnostic retention, and raw-OTel/broad-checkout exclusions are preserved.

Immutable transitive provenance

Layer Exact revision
BC-Bench parent / #938 646e2802bc89993e048c410a6f146bad3fc8237e
BC-Bench smoke head 5cacfe09231352d152d8827c307585e85b15c391
Replacement BC-ALAgents engine 7c7c12c2188dd7bee063f925b9d055d05075beff
Engine's sole direct parent / microsoft/BC-ALAgents#80 head 4a0885e207cf96617b33d54d0146debbcf7d9841
Engine-owned BCQuality / microsoft/BCQuality#220 head 8c9385c16fdb1f21afa6ce957420ead73294d06d
Declared BCQuality version 1.6

Recursive engine tree comparison returned all 44 non-directory entries, without truncation: 43 identical mode/type/blob entries; only agents/ALReviewAgent/bcquality.config.yaml changed. Its bytes are exactly the parent config with the single ref b74967bc5b7a454eae19d6a1250199afd869f064 replaced by 8c9385c16fdb1f21afa6ce957420ead73294d06d; version 1.6 and every other byte are unchanged. Engine tree: 29a070d5a3106b88f7fc919f3d6569f9ad593bad. Config blob: 54239fecf163a2c90003a0a559f6350826cc8aed -> e2e68b42eacbcaa8693e26578a9d9e52bf7a7052.

Recursive BC-Bench tree proof: 937 entries, identical path set and modes, 935 unchanged mode/type/blob entries, and exactly the two one-SHA substitutions above. No other tracked entry differs from #938.

  • Parent tree: 9ba9c56ab8f9d0a28fb1ab563a4a682ac722b361
  • Smoke tree: 382170b3f0ba11c3e05c573abbae272217bdcd06
  • Action blob: c31d48fe99fdd6b7fa1b4be42f46ccc71d5fcec3 -> 83a3b9afe61a59d75e6724c9c6298556679eaf11
  • Test blob: bbaabb76432c78ae3d642353571da2b432adddcd -> 568a1aadbc074e10bdca5ecd46c2fad078b577cc
  • Dataset SHA-256: 027fcf04cdbf506aeada64ef70498b059006d946e6804162a662ccbba45ef1d8
  • Two-pin diff SHA-256: 2e5d94c90fbbbbf98a37b2c0eb636dd6a96b1a1934bdbd7f4068e9a75b05639a (1,413 bytes).

Canonical diff command: git diff --no-ext-diff --no-textconv --no-color --no-renames --full-index --binary --src-prefix=a/ --dst-prefix=b/ 646e2802bc89993e048c410a6f146bad3fc8237e 5cacfe09231352d152d8827c307585e85b15c391 --. Hash raw stdout bytes, without PowerShell re-encoding.

Local validation

  • uv run --no-sync pytest -q tests/test_codereview_fixtures.py tests/test_review_workflows.py tests/test_pr_review_run_manifest.py tests/test_pr_review_agent.py tests/test_pr_review_metrics.py tests/test_pr_review_metrics_reporting.py tests/test_pr_review_output.py tests/test_codereview.py: 446 passed. This includes [Experimental support] Accept strict normalization manifests #938's legacy, ID-only, range-only, combined 4+4, malformed/lifecycle/strictness, omission/roundtrip, mocked adapter/evaluator, and exact diagnostic inclusion/exclusion tests, plus all HTTP fixture/workflow checks (the expanded equivalent of the earlier 95+ focused suite).
  • uv run --no-sync pytest -q: 1234 passed, 2 skipped, 1 deselected.
  • Ruff format/check passed; ty check . --ignore=unresolved-import --exclude notebooks/ passed with the same pre-existing unused-ignore warning in untouched src/bcbench/redteam.py:150.
  • uv run --no-sync pre-commit run --all-files: all 11 hooks passed. Initial Windows child-process propagation of empty core.fsmonitor failed; rerun replaced only that command-local environment value with equivalent false. No repository/global config change, skipped hook, or unrelated code edit.
  • git diff --check and staged diff checks passed. Commit has the required Copilot co-author trailer.

These are deterministic local checks, not a new smoke result, live model validation, or a claim of complete remote CI.

Fresh proposed one-arm authorization — NOT GRANTED

A new explicit authorization would apply only to this exact Bench head, engine, and BCQuality chain, and only to the following ordered seven-entry matrix:

["synthetic__privacy-008","synthetic__privacy-015","synthetic__security-clean-02","synthetic__errh-tryfunction-swallowed-01","synthetic__http-optional-clean-01","synthetic__http-consumed-false-01","synthetic__http-status-body-01"]

One arm, one run: at most 7 entry generations, 140 reviewer processes (19 ordered leaves + 1 root per entry), and 5 gold-judge invocations. There are 8 expected comments across 5 judge-eligible entries. These are invocation bounds, not provider-call/token/credit/dollar caps. No retries, replacements, replications, extra adjudicator, publication, release, or product merge. A failed/missing/invalid result remains a failed observation; stop rather than spending replacement budget or starting a second run.

Freeze CLI 1.0.88, root gpt-5.6-sol, leaf gpt-5.6-luna, gold judge gpt-5.3-codex, serial leaves, configured concurrency 4 / effective 1, 30-minute CLI timeout, and both severity floors Medium. Any separately authorized dispatch must resolve to Bench head 5cacfe09231352d152d8827c307585e85b15c391, use the exact entries array, explicit engine-sha=7c7c12c2188dd7bee063f925b9d055d05075beff, test-run=true, modified-only=false, repeat=1, and preserve the existing concurrency group/matrix cap. Revalidate all heads, clean fetched checkouts, runtime identities, dataset/fixture hashes, and source provenance before spending. This retains the no-tag/no-requeue route and summary mock=true: no Braintrust, Kusto, or leaderboard writes. Summary mock does not mock reviewer generation or the gold judge. Diagnostic artifacts remain allowed for the required audit.

Primary objective: 7/7 valid terminal reports with zero semantic-contract failures, not merely manifest completion or schema validity. Audit every original raw leaf report, its recorded SHA-256, accepted leaf report, normalization manifest, and _review-source-bounds.json; verify every ID/range transformation is allowed, exactly recorded and source-bounded, and that findings/evidence are neither silently dropped nor rewritten. Check root/nested outcomes, all ordered process roles, observed models, complete usage, and executed/declared pin equality. The inherited consumer validates audit metadata shape, not raw-byte integrity or source-bound contents: those require the explicit audit.

Preserve all three positive controls: consumed-TryFunction (synthetic__errh-tryfunction-swallowed-01), swallowed HTTP execution failure (synthetic__http-consumed-false-01), and completed non-2xx body processing (synthetic__http-status-body-01). Inventory unrelated findings and report raw versus accepted findings transparently. This second combined semantic-normalization smoke is not an HTTP knowledge A/B comparison or a causal claim; do not pool it with or reuse authorization from earlier attempts.

PREPARATION ONLY / NO PAID EXECUTION AUTHORIZED.

AB#652973. Preparation only above #938 at 646e280. Replace only the action default engine SHA and matching workflow test constant with 7c7c12c2188dd7bee063f925b9d055d05075beff. No product merge or release intent; no evaluation dispatch authorized.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant