test(msbuild): strengthen eval quality by AbhitejJohn · Pull Request #1118 · dotnet/skills

AbhitejJohn · 2026-09-04T00:07:00Z

Summary

Expand 19 dotnet-msbuild eval suites from 33 to 139 distinct stimuli.
Add realistic failure modes, correct no-op cases, routing boundaries, and customer journeys.
Add replayable golden patches, substantial ATIF references, deterministic graders, and self-contained fixtures.
Remove 18 repaired MSBuild suites from the underpowered-eval allowlist.
Strengthen the shared eval-quality gate, its self-tests and documentation, plus the create-skill-test and improve-skill-quality authoring guidance.

Design principles

These evals use repository eval-review practices, TDD, and Vally guidance as two complementary quality bars.

Test customer outcomes. Build stimuli from real tasks, skill scope, product behavior, and repository history. Do not grade implementation vocabulary when behavior can be executed.
Use distinct decisions, not repeated prompts. Each new stimulus covers another risk, journey, boundary, or valid no-op. Increasing runs measures reliability; it does not add scenario breadth.
Use RED/GREEN evidence. A broken fixture, mutation, or missing rule must fail first; the expected repair must then pass. The shared gate has paired regression tests for each defect class.
Use honest reference evidence. Keep short expected answers inline, use golden patches for text workspace changes, use command graders for builds and behavior, and keep separate ATIF files only for substantial traces or reports.
Prefer deterministic proof. File, patch, regex, and command graders carry the primary assertions. Prompt graders add semantic review where deterministic checks cannot express the whole outcome.
Keep fixtures local and controlled. Fixtures must reproduce the intended condition without network access, machine caches, secrets, staged executables, or answer-bearing comments.
Prove routing as well as execution. Include cases where the skill must defer, stay dormant, or preserve already-correct input so a broadly helpful answer cannot hide over-activation.
Keep binary evidence honest. Do not invent text patches for binlogs or other binary outputs. Use command evidence and record any non-replayable output as explicit debt.
Make quality repeatable. The changed-suite ratchet checks every touched spec, fixture, and reference while leaving unrelated legacy debt visible but non-blocking.

Review map: what changed in each eval

Skill eval	Stimuli	Nature of the new coverage
`agent.msbuild`	2 -> 2	Hardened the two routing cases: controlled offline `NU1101` diagnosis and project-organization advice. The package case is diagnosis-only and cannot resolve from a remote feed or machine fallback cache.
`binlog-failure-analysis`	1 -> 8	Added binlog-only diagnosis, text-replay fallback, warning-versus-failure handling, quiet second-build analysis, resolved framework/package evidence, generated-source tracing, and non-failure/non-MSBuild boundaries.
`binlog-generation`	3 -> 8	Added shell-specific commands, unique names for parallel configurations and CI uploads, history retention, stale-output cleanup, reused-name repair, missing-binlog diagnosis, and a non-MSBuild boundary.
`build-parallelism`	1 -> 7	Added wide-graph parallelism, critical-path optimization, `BuildInParallel`, rejection of unsafe graph mode for run-time project discovery, solution filters, and an incremental-build routing boundary. The original solution fixture was repaired so it builds real projects.
`build-perf-baseline`	1 -> 7	Added measurement-first baselines, routing for broken no-op rebuilds and restore-bound builds, `UseArtifactsOutput`, deterministic cache-safe CI, already-optimized no-op behavior, and a non-MSBuild boundary.
`build-perf-diagnostics`	1 -> 8	Added small-project triage, a real custom-task/RAR topology, Copy bottlenecks, redundant CI restore, dominant custom targets, evaluation overhead, many-small-project overhead, and a runtime-latency boundary.
`check-bin-obj-clash`	1 -> 7	Added shared output/intermediate collisions, collapsed multi-targeting paths, harmful `ProjectReference` metadata, safe shared roots, full mixed-solution repair, audit-only classification, and default-layout no-op behavior.
`directory-build-organization`	1 -> 7	Added repo-wide props/targets organization, evaluation-order and override bugs, an executable centralization repair, lone-project restraint, preserved project exceptions, and package-downgrade/version-management cleanup.
`eval-performance`	1 -> 8	Added deep imports, broad globs, file-I/O property functions, duplicate evaluation under global properties, measurement-first triage, compile/incremental misrouting boundaries, and justified versus excessive `TreatAsLocalProperty`.
`extension-points`	3 -> 8	Added package/repository hook conflicts, safe packed layouts, per-TFM forwarders, package-ID/file-name mismatches, custom SDK extensibility hooks, an executable anti-pattern repair, and an incremental-target boundary.
`including-generated-files`	1 -> 8	Added project globs, target hook timing, clean tracking, non-code output registration, hard-coded `obj` paths, an executable full repair, and a Roslyn source-generator boundary.
`incremental-build`	1 -> 9	Added `Inputs`/`Outputs`, `FileWrites` and Clean behavior, volatile outputs, stale-input log evidence, the limits of `Outputs` alone, compiler-cache explanations, Visual Studio up-to-date behavior, cold-build restraint, and a complete executable repair.
`item-management`	3 -> 7	Added batching and metadata defects, mixed real/claimed code-generation problems, executable repair, valid single-list batching, ineffective `Compile Remove`, already-correct no-op behavior, and a target-ordering boundary.
`msbuild-antipatterns`	4 -> 8	Added F# source/signature ordering, guarded-import judgment, the difference between style-only import separators and behavioral `Exec` path bugs, clean-project restraint, and a modernization routing boundary.
`msbuild-modernization`	1 -> 7	Added SDK-style migration, deterministic language-version choices, multi-target consolidation, restraint around Central Package Management, already-modern no-op behavior, and a non-.NET boundary.
`msbuild-server`	1 -> 8	Added suitability decisions for repeated local and CI builds, IDE and one-off-build boundaries, stale-output diagnosis, persistent configuration, before/after measurement, and background-process memory guidance.
`property-patterns`	3 -> 7	Added property hierarchy and evaluation-order failures, executable shared-property repair, invalid `TargetFramework` conditions in props, cross-platform OS detection, preservation of existing constants, no-op behavior, and a props-versus-targets boundary.
`resolve-project-references`	1 -> 8	Added wait-time versus task-self-time analysis, evidence requests before diagnosis, exact replay commands, single-node graph handoff, Copy-versus-Csc ranking, and dormancy when another performance skill owns the request.
`target-authoring`	3 -> 7	Added cross-file target chains, reliable generated-source ownership, direct-compile hooks, `Returns` versus `Outputs`, executable anti-pattern repair, already-correct no-op behavior, and an incremental-build boundary.

Shared quality tooling

check_eval_quality.py applies 22 fixture, reference, grader, security, and quality checks.
Default mode checks suites changed from HEAD^; --base-ref supports PR-base checks; --all audits the repository.
A change to any file in a suite checks the complete suite, including new and untracked inputs.
107 self-tests cover RED/GREEN mutations, changed-suite behavior, and full-audit behavior.
The authoring skills now document how to create and repair evals using the same evidence rules.

Repository-space impact

The PR adds 290 MSBuild artifact files totaling 176,115 bytes (172 KiB) in the working tree. A standalone ZIP of those files is about 145,041 bytes (142 KiB).

Added artifact type	Files	Working-tree size
Fixtures and support files	238	83 KiB
Substantial ATIF JSON references	31	56 KiB
Golden patches	21	33 KiB

The largest per-suite additions are binlog-failure-analysis (23 KiB), eval-performance (22 KiB), binlog-generation (14 KiB), and msbuild-server (14 KiB). These figures measure this PR's added files, not Git pack-level deduplication.

Reliability and safety

Replaced the live-feed missing-package assumption with Contoso.IntentionallyMissing, an empty local feed, a project-local package cache, and cleared fallback folders.
Replaced the missing sign.exe sentinel with a harmless dotnet --version marker and removed answer-bearing fixture comments.
Removed 98 obsolete JSON references after their short expected answers moved inline; all 31 retained JSON references and all 21 golden patches are wired to evals.
No production plugins/dotnet-msbuild skill or agent behavior changes are included.

Validation

Eval-quality gate: 19 changed suites checked, no errors.
Eval-quality self-tests: 107 passed.
Skill Validator: 19 skills and 3 agents validated.
Vally qualification: 19/19 suites rated FAIR, zero dynamic replay errors.
All staged golden patches apply to their declared fixtures.
Targeted offline NU1101 and path-marker fixture checks passed.
The complete patch applies cleanly to the branch base.
Multi-model review completed with Opus 4.8, Sonnet 5, and Terra; actionable feedback was addressed.

Scope

This draft contains only:

tests/dotnet-msbuild/**
eng/eval-quality/check_eval_quality.py, its self-tests, README, and the MSBuild allowlist removals
.agents/skills/create-skill-test/**
.agents/skills/improve-skill-quality/**

Other plugin eval repairs and unrelated working-tree changes are excluded.

Expand MSBuild eval coverage with realistic fixtures, replayable references, and a changed-suite quality ratchet. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb11c85b-4046-4dbc-94b3-3bdbd4839a0d

Clear NuGet fallback folders so the intentional NU1101 case cannot resolve from machine state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb11c85b-4046-4dbc-94b3-3bdbd4839a0d

AbhitejJohn · 2026-09-04T07:05:55Z

@JanKrivanek @YuliiaKovalova, I would appreciate your thoughts and feedback on this draft, especially on:

whether the 19 eval suites cover useful customer decisions and routing boundaries rather than prompt variations;
whether the balance of deterministic graders, golden patches, and substantial ATIF references is reliable and easy to maintain;
whether the changed-suite quality ratchet is the right way to improve new work without blocking on unrelated legacy debt.

For repository-size context, this PR adds 290 MSBuild artifact files totaling 176,115 bytes (172 KiB) uncompressed, or about 145,041 bytes (142 KiB) as a standalone ZIP: 83 KiB of fixtures/support files, 56 KiB of ATIF JSON, and 33 KiB of golden patches. The PR description now has a row-by-row review map for every changed skill eval and the TDD/Vally-guided design principles used to create them.

AbhitejJohn

/evaluate

AbhitejJohn · 2026-09-04T07:08:47Z

/evaluate 0d63a77

github-actions · 2026-09-04T08:42:12Z

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 0d63a7748d1941554011285e0ad8c5916aa5dae4 to retry this exact commit.

35 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

AbhitejJohn · 2026-09-04T16:58:32Z

/evaluate 0d63a77

github-actions · 2026-09-04T18:07:57Z

📊 Skill Evaluation Results

36 model/skill results across 18 skills and 2 models — ✅ 1 improved, ➖ 12 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 23 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 0d63a7748d1941554011285e0ad8c5916aa5dae4; 2 judge models.

Measurement health: 36 expected / 36 observed / 36 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
binlog-failure-analysis	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 2 dormancy excluded	🟡 0.26	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.
binlog-failure-analysis	gpt-5.6-luna	⛔ Activation contract failed	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 2 dormancy excluded	✅ 0.14	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.
binlog-generation	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded	🔴 0.58	Activation: isolated 5/7; plugin 5/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
binlog-generation	gpt-5.6-luna	➖ Not proven improved	n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded	🟡 0.21	Activation: isolated 7/7; plugin 6/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-parallelism	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded	🟡 0.34	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
build-parallelism	gpt-5.6-luna	➖ Not proven improved	n=6; 3W/0T/3L; d=6; p=0.656; net +0.0%; 1 dormancy excluded	🟡 0.22	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-baseline	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 1 dormancy excluded	🔴 0.52	Dormancy contract: 1 unexpected activation(s); Activation: isolated 4/6; plugin 6/6	Narrow skill routing so the listed off-target scenarios stay dormant.
build-perf-baseline	gpt-5.6-luna	⛔ Activation contract failed	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	🟡 0.23	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 6/6	Narrow skill routing so the listed off-target scenarios stay dormant.
build-perf-diagnostics	claude-sonnet-4.6	➖ Not proven improved	n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded	🟡 0.39	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
build-perf-diagnostics	gpt-5.6-luna	✅ Improved	n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 1 dormancy excluded	✅ 0.16	—	None.
check-bin-obj-clash	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	🟡 0.22	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
check-bin-obj-clash	gpt-5.6-luna	⛔ Activation contract failed	n=6; 0W/4T/2L; d=2; p=0.250; net -33.3%; 1 dormancy excluded	✅ 0.11	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
directory-build-organization	claude-sonnet-4.6	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	🟡 0.33	Activation: isolated 5/6; plugin 5/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	gpt-5.6-luna	⛔ Activation contract failed	n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.07	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.
eval-performance	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 2 dormancy excluded	🟡 0.38	Dormancy contract: 2 unexpected activation(s); Activation: isolated 4/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
eval-performance	gpt-5.6-luna	⛔ Activation contract failed	n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 2 dormancy excluded	✅ 0.09	Dormancy contract: 2 unexpected activation(s); Activation: isolated 6/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
extension-points	claude-sonnet-4.6	➖ Not proven improved	n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%; 1 dormancy excluded	🟡 0.26	Activation: isolated 5/7; plugin 6/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
extension-points	gpt-5.6-luna	➖ Not proven improved	n=7; 2W/3T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded	🟡 0.26	Activation: isolated 7/7; plugin 6/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
including-generated-files	claude-sonnet-4.6	⛔ Activation contract failed	n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%; 1 dormancy excluded	🟡 0.41	Dormancy contract: 1 unexpected activation(s); Activation: isolated 4/7; plugin 6/7	Narrow skill routing so the listed off-target scenarios stay dormant.
including-generated-files	gpt-5.6-luna	➖ Not proven improved	n=7; 2W/2T/3L; d=5; p=0.500; net -14.3%; 1 dormancy excluded	✅ 0.15	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
incremental-build	claude-sonnet-4.6	⛔ Activation contract failed	n=8; 2W/5T/1L; d=3; p=0.500; net +12.5%; 1 dormancy excluded	🟡 0.36	Dormancy contract: 1 unexpected activation(s); Activation: isolated 4/8; plugin 4/8	Narrow skill routing so the listed off-target scenarios stay dormant.
incremental-build	gpt-5.6-luna	⛔ Activation contract failed	n=8; 4W/4T/0L; d=4; p=0.063; net +50.0%; 1 dormancy excluded	✅ 0.10	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/8; plugin 8/8	Narrow skill routing so the listed off-target scenarios stay dormant.
item-management	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 1W/3T/2L; d=3; p=0.500; net -16.7%; 1 dormancy excluded	🟡 0.38	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 4/6	Narrow skill routing so the listed off-target scenarios stay dormant.
item-management	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	✅ 0.11	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	claude-sonnet-4.6	⛔ Activation contract failed	n=7; 1W/6T/0L; d=1; p=0.500; net +14.3%; 1 dormancy excluded	🟡 0.36	Dormancy contract: 1 unexpected activation(s); Activation: isolated 2/7; plugin 2/7	Narrow skill routing so the listed off-target scenarios stay dormant.
msbuild-antipatterns	gpt-5.6-luna	⛔ Activation contract failed	n=7; 0W/4T/3L; d=3; p=0.125; net -42.9%; 1 dormancy excluded	✅ 0.09	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 4/7	Narrow skill routing so the listed off-target scenarios stay dormant.
msbuild-modernization	claude-sonnet-4.6	➖ Not proven improved	n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded	🟡 0.29	Activation: isolated 6/6; plugin 5/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-modernization	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/2T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded	✅ 0.05	Activation: isolated 5/6; plugin 6/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-server	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 3W/1T/2L; d=5; p=0.500; net +16.7%; 2 dormancy excluded	🟡 0.47	Dormancy contract: 2 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.
msbuild-server	gpt-5.6-luna	⛔ Activation contract failed	n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 2 dormancy excluded	🟡 0.22	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 6/6	Narrow skill routing so the listed off-target scenarios stay dormant.
property-patterns	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 1 dormancy excluded	🟡 0.23	Dormancy contract: 1 unexpected activation(s); Activation: isolated 4/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
property-patterns	gpt-5.6-luna	⛔ Activation contract failed	n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded	✅ 0.06	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
resolve-project-references	claude-sonnet-4.6	⛔ Activation contract failed	n=5; 4W/0T/1L; d=5; p=0.188; net +60.0%; 3 dormancy excluded	🟡 0.32	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.
resolve-project-references	gpt-5.6-luna	⛔ Activation contract failed	n=5; 2W/3T/0L; d=2; p=0.250; net +40.0%; 3 dormancy excluded	✅ 0.14	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.
target-authoring	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	🟡 0.29	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 4/6	Narrow skill routing so the listed off-target scenarios stay dormant.
target-authoring	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/2T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded	✅ 0.13	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

ℹ️ How to read this report

✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
⛔ Activation contract failed — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — binlog-failure-analysis (claude-sonnet-4.6)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +35.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 2 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Overfit: Moderate (score 0.26)

Repeated-run reliability (not used by the gate): 8 paired runs (5W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Confirm the actual resolved target framework and package version from a binlog	Eligible	+0.0%	+0.0%	0/1/0
▲ Decline to "diagnose" a build that has not actually failed yet	Excluded (activation contract)	+100.0%	+100.0%	1/0/0
= Diagnose a warning behind a build that actually succeeded	Eligible	+0.0%	+0.0%	0/1/0
▼ Trace why a generated source file is missing at compile time	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Confirm the actual resolved target framework and package version from a binlog: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — binlog-failure-analysis (gpt-5.6-luna)

Why: Net win +16.7% (2W/3T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 2 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Overfit: Low (score 0.14)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Confirm the actual resolved target framework and package version from a binlog	Eligible	+0.0%	+0.0%	0/1/0
= Decline to "diagnose" a build that has not actually failed yet	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose build failures from binlog only (no source files)	Eligible	+0.0%	+0.0%	0/1/0
= Fall back to command-line log replay when the usual binlog tool is unavailable	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant for a non-MSBuild build failure log	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Trace why a generated source file is missing at compile time	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Confirm the actual resolved target framework and package version from a binlog: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — build-parallelism (claude-sonnet-4.6)

Why: Net win +66.7% (4W/2T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +17.1% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 5/6

Overfit: Moderate (score 0.34)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Enable BuildInParallel on a custom MSBuild task	Eligible	+0.0%	+0.0%	0/1/0
= Enable parallel nodes for a wide project graph	Eligible	+0.0%	+0.0%	0/1/0
▼ Non-activation: make one custom target incremental	Excluded (activation contract)	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Enable BuildInParallel on a custom MSBuild task: The resulting edits and final explanations are substantively identical and fully satisfy the task.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — build-perf-baseline (claude-sonnet-4.6)

Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +40.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 4/6; plugin 6/6

Overfit: High (score 0.52)

Repeated-run reliability (not used by the gate): 7 paired runs (5W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Configure deterministic, cache-safe CI builds	Eligible	+100.0%	+100.0%	1/0/0
▲ Decline a non-MSBuild build performance request	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
▼ Leave an already-optimized build unchanged	Eligible	-100.0%	-40.0%	0/0/1
= Route a restore-bound cold build away from architecture changes	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Leave an already-optimized build unchanged: Both fail the central task by changing a repository with no applicable checklist anti-pattern. A at least locates and reviews the requested App/Core project, whereas B works on a different WebApi/BusinessLogic repository and makes unrelated reference edits; A is therefore slig...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — build-perf-baseline (gpt-5.6-luna)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 6/6

Overfit: Moderate (score 0.23)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Configure deterministic, cache-safe CI builds	Eligible	+100.0%	+100.0%	1/0/0
▼ Decline a non-MSBuild build performance request	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Leave an already-optimized build unchanged	Eligible	-100.0%	-40.0%	0/0/1
= Route a broken no-op rebuild away from generic optimization	Eligible	+0.0%	+0.0%	0/1/0
= Route a restore-bound cold build away from architecture changes	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Leave an already-optimized build unchanged: The ideal outcome was to make no changes, as no baseline anti-pattern was present. Both responses failed this and invented a change. However, B committed the specific anti-pattern the rubric explicitly warns against (adding UseArtifactsOutput to a two-project solution), and it...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — check-bin-obj-clash (claude-sonnet-4.6)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +25.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 5/6

Overfit: Moderate (score 0.22)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Audit the mixed solution and separate safe projects from unsafe ones	Eligible	+0.0%	+0.0%	0/1/0
▲ Decline output-clash remediation for separate projects using the default SDK layout	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
= Diagnose multi-targeting outputs that collapse into one path	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix all clash mechanisms in the mixed solution	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Audit the mixed solution and separate safe projects from unsafe ones: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — check-bin-obj-clash (gpt-5.6-luna)

Why: Net win -33.3% (0W/4T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.250), mean preference -17.1% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 0W/4T/2L; d=2; p=0.250; net -33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (0W/4T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Audit the mixed solution and separate safe projects from unsafe ones	Eligible	-100.0%	-40.0%	0/0/1
= Avoid a false clash report when projects share only the top-level artifacts root	Eligible	+0.0%	+0.0%	0/1/0
▼ Decline output-clash remediation for separate projects using the default SDK layout	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Diagnose multi-targeting outputs that collapse into one path	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose redundant project reference metadata that forks a same-path build	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose shared output and intermediate path collision	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix all clash mechanisms in the mixed solution	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Audit the mixed solution and separate safe projects from unsafe ones: Both responses correctly identify the core clashes (LibraryA/B sharing, MultiTargetLib TFM overwrite) and the ToolLib redundant-build mechanism. However, A aligns better with the rubric's intended classification: it flags ConsumerApp as unsafe (the source of the redundant meta...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — directory-build-organization (gpt-5.6-luna)

Why: Net win +16.7% (1W/5T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Overfit: Low (score 0.07)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/6T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Apply repo-level build organization cleanup	Eligible	+0.0%	+0.0%	0/1/0
= Decline adding shared build files to a lone project	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that silently skips in props	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a package downgrade chain and reorganize version management	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose an inner shared-props file that overwrites its own override	Eligible	+0.0%	+0.0%	0/1/0
= Preserve an intentional project-specific exception while centralizing	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Apply repo-level build organization cleanup: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — eval-performance (claude-sonnet-4.6)

Why: Net win +50.0% (3W/3T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +32.5% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — activation contract failed (2 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 2 dormancy excluded

Warnings: Dormancy contract: 2 unexpected activation(s); Activation: isolated 4/6; plugin 5/6

Overfit: Moderate (score 0.38)

Repeated-run reliability (not used by the gate): 8 paired runs (5W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Detect a project evaluated twice under different global properties	Eligible	+0.0%	+0.0%	0/1/0
= Recognize TreatAsLocalProperty overuse versus one justified entry	Eligible	+0.0%	+0.0%	0/1/0
▲ Redirect a compile-time slowdown mistakenly framed as an evaluation problem	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
▲ Redirect an incremental-rebuild complaint mistakenly framed as an evaluation problem	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
= Triage which of two property functions actually costs evaluation time	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Detect a project evaluated twice under different global properties: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — eval-performance (gpt-5.6-luna)

Why: Net win +33.3% (2W/4T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +25.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — activation contract failed (2 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 2 dormancy excluded

Warnings: Dormancy contract: 2 unexpected activation(s); Activation: isolated 6/6; plugin 5/6

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Detect a project evaluated twice under different global properties	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose stacked evaluation-time patterns (deep imports, broad glob, file-I/O property function)	Eligible	+0.0%	+0.0%	0/1/0
= Recognize TreatAsLocalProperty overuse versus one justified entry	Eligible	+0.0%	+0.0%	0/1/0
▼ Redirect a compile-time slowdown mistakenly framed as an evaluation problem	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▲ Redirect an incremental-rebuild complaint mistakenly framed as an evaluation problem	Excluded (activation contract)	+100.0%	+100.0%	1/0/0
= Triage which of two property functions actually costs evaluation time	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Detect a project evaluated twice under different global properties: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — including-generated-files (claude-sonnet-4.6)

Why: Net win +42.9% (4W/2T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 4/7; plugin 6/7

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline manual wiring for Roslyn source generators	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Diagnose hardcoded obj path for generated source	Eligible	+0.0%	+0.0%	0/1/0
▲ Diagnose missing clean tracking for generated source	Eligible	+100.0%	+40.0%	1/0/0
▼ Diagnose missing output registration for generated non-code file	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose wrong hook for generated source files	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose hardcoded obj path for generated source: The outputs are substantively equivalent, accurate, and directly answer the requested root cause and correct path pattern. B is marginally more concise, while A is equally clear; neither has a meaningful quality advantage.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — incremental-build (claude-sonnet-4.6)

Why: Net win +12.5% (2W/5T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +8.9% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=8; 2W/5T/1L; d=3; p=0.500; net +12.5%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 4/8; plugin 4/8

Overfit: Moderate (score 0.36)

Repeated-run reliability (not used by the gate): 9 paired runs (3W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Correct the assumption that Outputs alone enables incremental skipping	Eligible	+0.0%	+0.0%	0/1/0
▲ Decline treating a cold first build as broken incrementality	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
= Diagnose custom targets that always rerun	Eligible	+0.0%	+0.0%	0/1/0
▼ Explain slower builds when MSBuild skipped everything	Eligible	-100.0%	-40.0%	0/0/1
= Explain why Visual Studio keeps rebuilding an up-to-date project	Eligible	+0.0%	+0.0%	0/1/0
= Explain why clean leaves generated hash source behind	Eligible	+0.0%	+0.0%	0/1/0
= Read a diagnostic log to find the stale input	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Correct the assumption that Outputs alone enables incremental skipping: Position-swap inconsistent (forward: A, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — incremental-build (gpt-5.6-luna)

Why: Net win +50.0% (4W/4T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +22.2% across 9 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=8; 4W/4T/0L; d=4; p=0.063; net +50.0%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/8; plugin 8/8

Overfit: Low (score 0.10)

Repeated-run reliability (not used by the gate): 9 paired runs (5W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Correct the assumption that Outputs alone enables incremental skipping	Eligible	+100.0%	+40.0%	1/0/0
▲ Decline treating a cold first build as broken incrementality	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
= Explain slower builds when MSBuild skipped everything	Eligible	+0.0%	+0.0%	0/1/0
= Explain why clean leaves generated hash source behind	Eligible	+0.0%	+0.0%	0/1/0
= Identify a volatile output path that defeats incrementality	Eligible	+0.0%	+0.0%	0/1/0
= Read a diagnostic log to find the stale input	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Explain slower builds when MSBuild skipped everything: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — item-management (claude-sonnet-4.6)

Why: Net win -16.7% (1W/3T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -11.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 1W/3T/2L; d=3; p=0.500; net -16.7%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 4/6

Overfit: Moderate (score 0.38)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/3T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline a target ordering problem with no item-group defect	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Diagnose an ineffective Compile Remove that does not match the glob	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose real and claimed item problems in a code generation pipeline	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix item management anti-patterns	Eligible	-100.0%	-40.0%	0/0/1
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
▼ Leave correct single-list batching unchanged	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Diagnose an ineffective Compile Remove that does not match the glob: The final answers are substantively equivalent, concise, correct, and fully address both the cause and correction.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — msbuild-antipatterns (claude-sonnet-4.6)

Why: Net win +14.3% (1W/6T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=7; 1W/6T/0L; d=1; p=0.500; net +14.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 2/7; plugin 2/7

Overfit: Moderate (score 0.36)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/6T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a module to an F# project	Eligible	+0.0%	+0.0%	0/1/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
= Distinguish a style backslash from a real cross-platform backslash bug	Eligible	+0.0%	+0.0%	0/1/0
= Fix broken file order causing FS0039	Eligible	+0.0%	+0.0%	0/1/0
= Judge an unguarded import inside a NuGet package build folder	Eligible	+0.0%	+0.0%	0/1/0
= Leave a clean project without inventing anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
▲ Non-activation: migrate a legacy project to SDK style	Excluded (activation contract)	+100.0%	+40.0%	1/0/0

Illustrative judge evidence:

Add a module to an F# project: The implementations and demonstrated behavior are substantively identical: both meet the requested validation, project ordering, integration, and successful build/run requirements.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — msbuild-antipatterns (gpt-5.6-luna)

Why: Net win -42.9% (0W/4T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.125), mean preference -20.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=7; 0W/4T/3L; d=3; p=0.125; net -42.9%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 4/7

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 8 paired runs (0W/4T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a module to an F# project	Eligible	+0.0%	+0.0%	0/1/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
= Distinguish a style backslash from a real cross-platform backslash bug	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix broken file order causing FS0039	Eligible	-100.0%	-40.0%	0/0/1
= Judge an unguarded import inside a NuGet package build folder	Eligible	+0.0%	+0.0%	0/1/0
▼ Leave a clean project without inventing anti-patterns	Eligible	-100.0%	-40.0%	0/0/1
▼ Non-activation: migrate a legacy project to SDK style	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Review MSBuild files for anti-patterns and style issues	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Add a module to an F# project: Both responses produced functionally identical results: an equivalent Validation.fs, updated Program.fs, correct compile ordering (evidenced by successful builds requiring Validation before Program), and successful dotnet run. B briefly hit a NuGet restore error but recovered ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — msbuild-server (claude-sonnet-4.6)

Why: Net win +16.7% (3W/1T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +20.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — activation contract failed (2 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/1T/2L; d=5; p=0.500; net +16.7%; 2 dormancy excluded

Warnings: Dormancy contract: 2 unexpected activation(s)

Overfit: Moderate (score 0.47)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/1T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Confirm the MSBuild Server is actually improving build times before declaring success	Eligible	+0.0%	+0.0%	0/1/0
▼ Decline MSBuild Server for a single one-off release build	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Explain a background MSBuild Server process consuming memory	Eligible	-100.0%	-40.0%	0/0/1
▼ Recommend MSBuild Server for slow CLI incremental builds	Eligible	-100.0%	-40.0%	0/0/1
▲ Stay dormant for a purely IDE-side build slowdown	Excluded (activation contract)	+100.0%	+100.0%	1/0/0

Illustrative judge evidence:

Confirm the MSBuild Server is actually improving build times before declaring success: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — msbuild-server (gpt-5.6-luna)

Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +35.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 2 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 6/6

Overfit: Moderate (score 0.22)

Repeated-run reliability (not used by the gate): 8 paired runs (5W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Configure MSBuild Server persistently across new Windows terminals	Eligible	-100.0%	-40.0%	0/0/1
= Decline MSBuild Server for a single one-off release build	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose stale build output after enabling MSBuild Server	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Configure MSBuild Server persistently across new Windows terminals: Both responses correctly recommend a persistent user-level Windows environment variable and instruct closing/reopening terminals. The decisive difference is the variable name: Response A uses DOTNET_CLI_USE_MSBUILD_SERVER, which matches the rubric's designated correct variable...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — property-patterns (claude-sonnet-4.6)

Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +31.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 4/6; plugin 5/6

Overfit: Moderate (score 0.23)

Repeated-run reliability (not used by the gate): 7 paired runs (5W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Choose the right OS detection for a cross-platform property	Eligible	+100.0%	+40.0%	1/0/0
▲ Decline a props-versus-targets placement question	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
= Diagnose a TargetFramework condition that never applies in props	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix shared property configuration	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Diagnose a TargetFramework condition that never applies in props: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — property-patterns (gpt-5.6-luna)

Why: Net win +50.0% (3W/3T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +17.1% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6

Overfit: Low (score 0.06)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Choose the right OS detection for a cross-platform property	Eligible	+100.0%	+40.0%	1/0/0
= Decline a props-versus-targets placement question	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that never applies in props	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose multi-level property hierarchy bugs	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose shared build property issues	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a TargetFramework condition that never applies in props: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — resolve-project-references (claude-sonnet-4.6)

Why: Net win +60.0% (4W/0T/1L over 5 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +27.5% across 8 paired run(s), 3 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=5; 4W/0T/1L; d=5; p=0.188; net +60.0%; 3 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 8 paired runs (6W/0T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Rank Copy ahead of Csc when task self-time is higher	Eligible	-100.0%	-100.0%	0/0/1
▼ Request diagnostic artifact before analysis	Excluded (activation contract)	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Rank Copy ahead of Csc when task self-time is higher: A directly and correctly answers the question using the report's task-level evidence, explains why the target-level leader is not the optimization target, and gives a relevant copy optimization direction. B has the general diagnostic principle right but fails to use the suppli...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — resolve-project-references (gpt-5.6-luna)

Why: Net win +40.0% (2W/3T/0L over 5 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +27.5% across 8 paired run(s), 3 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=5; 2W/3T/0L; d=2; p=0.250; net +40.0%; 3 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Overfit: Low (score 0.14)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Distinguish wait time from a real serial dependency chain	Eligible	+0.0%	+0.0%	0/1/0
= Rank Copy ahead of Csc when task self-time is higher	Eligible	+0.0%	+0.0%	0/1/0
= Redirect from misleading target summary to Csc self-time	Eligible	+0.0%	+0.0%	0/1/0
▲ Request diagnostic artifact before analysis	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
= Stay dormant when Csc is already the obvious bottleneck	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Distinguish wait time from a real serial dependency chain: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — target-authoring (claude-sonnet-4.6)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +20.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 4/6

Overfit: Moderate (score 0.29)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline an incremental-build tuning request	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▲ Diagnose a target hooked to Build that misses direct compile	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose broken SDK target chain across files	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose custom target reliability issues	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix a query target that uses Outputs instead of Returns	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Diagnose broken SDK target chain across files: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — binlog-generation (claude-sonnet-4.6)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +20.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded

Warnings: Activation: isolated 5/7; plugin 5/7

Overfit: High (score 0.58)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Build with /bl in PowerShell	Eligible	+0.0%	+0.0%	0/1/0
= Choose a predictable, non-colliding binlog name for a CI upload step	Eligible	+0.0%	+0.0%	0/1/0
▼ Decline a binlog request for a non-MSBuild Java build	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Fix a CI build script that reuses the same binlog on every run	Eligible	+0.0%	+0.0%	0/1/0
▼ Preserve binlog history while cleaning stale build output	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Build with /bl in PowerShell: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — binlog-generation (gpt-5.6-luna)

Why: Net win +14.3% (3W/2T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +27.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.500 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Moderate (score 0.21)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Build with /bl in PowerShell	Eligible	-100.0%	-40.0%	0/0/1
= Choose a predictable, non-colliding binlog name for a CI upload step	Eligible	+0.0%	+0.0%	0/1/0
= Decline a binlog request for a non-MSBuild Java build	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Fix a CI build script that reuses the same binlog on every run	Eligible	+0.0%	+0.0%	0/1/0
▼ Recognize that a failed build produced no binlog at all	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Build with /bl in PowerShell: Both successfully built the project via PowerShell and produced a valid binlog with 0 errors. A's approach used an explicit quoted filename, which is unambiguous and clean. B relied on the skill's {{}} template syntax which, while it produced a valid unique filename, uses the ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-parallelism (gpt-5.6-luna)

Why: Net win +0.0% (3W/0T/3L over 6 preference-eligible stimulus vote(s), sign test p=0.656), mean preference -5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/0T/3L; d=6; p=0.656; net +0.0%; 1 dormancy excluded

Overfit: Moderate (score 0.22)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/0T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Enable parallel nodes for a wide project graph	Eligible	-100.0%	-40.0%	0/0/1
▼ Non-activation: make one custom target incremental	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Preserve valid dependencies and optimize the slow critical-path project	Eligible	-100.0%	-40.0%	0/0/1
▼ Reduce CI build scope with a solution filter	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Enable parallel nodes for a wide project graph: Both responses are correct and nearly identical in substance, correctly diagnosing the single-node default and recommending -m. Neither suggested confirming via binlog. A slightly edges out by also offering the explicit -m:4 option, better matching the -m:N criterion.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (claude-sonnet-4.6)

Why: Net win -14.3% (1W/4T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -17.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded

Overfit: Moderate (score 0.39)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/4T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	-100.0%	-100.0%	0/0/1
▼ Diagnose NuGet restore running redundantly across CI stages	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose a Copy task dominating build time	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose a single custom target dominating one project's build	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose evaluation overhead before any target runs	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose per-project overhead across many small projects	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose slow build for a small project	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose NuGet restore running redundantly across CI stages: Both reach the central diagnosis and recommended pipeline change, but A gives a more accurate accounting of avoidable work and avoids B's misleading claims about static graph evaluation and post-change stage timing.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

Details for 8 results were omitted to keep this comment under GitHub's limit. Open the workflow summary or Full Results for the complete breakdown.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1118 in dotnet/skills, download eval artifacts with gh run download 33898074788 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/0d63a7748d1941554011285e0ad8c5916aa5dae4/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

JanKrivanek

Do we want to split the infra changes and msbuild eval changes? Both are quite loaded by themselves

Restore dormancy constraint protection, correct in-scope activation expectations, and strengthen MSBuild skill routing from CI evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d0a0a20-fd83-4343-8434-4763f8120503

AbhitejJohn

/evaluate

github-actions · 2026-09-08T20:00:42Z

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 34de668c6602fbf6ec12ec19da961e9f11e6e2e1 to retry this exact commit.

36 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

Reclassify in-scope suitability checks, sharpen true routing boundaries, and isolate binlog fixture builds from shared build-server state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d0a0a20-fd83-4343-8434-4763f8120503

github-actions · 2026-09-08T21:16:57Z

📊 Skill Evaluation Results

36 model/skill results across 18 skills and 2 models — ✅ 3 improved, ➖ 27 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 6 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 6866836a9b096ae2e9aefefe6dbfdab67405f37f; 2 judge models.

Measurement health: 36 expected / 36 observed / 36 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
binlog-failure-analysis	claude-sonnet-4.6	➖ Not proven improved	n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded	🟡 0.24	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
binlog-failure-analysis	gpt-5.6-luna	➖ Not proven improved	n=7; 4W/1T/2L; d=6; p=0.344; net +28.6%; 1 dormancy excluded	✅ 0.10	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
binlog-generation	claude-sonnet-4.6	✅ Improved	n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 1 dormancy excluded	🟡 0.49	—	Review overfit evidence.
binlog-generation	gpt-5.6-luna	➖ Not proven improved	n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%; 1 dormancy excluded	🟡 0.28	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
build-parallelism	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded	🟡 0.43	Dormancy contract: 1 unexpected activation(s); Activation: isolated 3/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
build-parallelism	gpt-5.6-luna	➖ Not proven improved	n=6; 0W/3T/3L; d=3; p=0.125; net -50.0%; 1 dormancy excluded	✅ 0.11	Activation: isolated 5/6; plugin 6/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
build-perf-baseline	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 4W/0T/2L; d=6; p=0.344; net +33.3%; 1 dormancy excluded	🟡 0.43	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
build-perf-baseline	gpt-5.6-luna	➖ Not proven improved	n=6; 4W/0T/2L; d=6; p=0.344; net +33.3%; 1 dormancy excluded	🟡 0.23	Activation: isolated 6/6; plugin 5/6	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-diagnostics	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/1T/3L; d=6; p=0.656; net +0.0%; 1 dormancy excluded	🟡 0.45	Activation: isolated 6/7; plugin 7/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-diagnostics	gpt-5.6-luna	➖ Not proven improved	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded	✅ 0.14	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
check-bin-obj-clash	claude-sonnet-4.6	➖ Not proven improved	n=7; 4W/1T/2L; d=6; p=0.344; net +28.6%	✅ 0.19	Activation: isolated 6/7; plugin 6/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
check-bin-obj-clash	gpt-5.6-luna	➖ Not proven improved	n=7; 0W/6T/1L; d=1; p=0.500; net -14.3%	✅ 0.13	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	claude-sonnet-4.6	➖ Not proven improved	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	🟡 0.33	Activation: isolated 5/6; plugin 5/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	gpt-5.6-luna	⛔ Activation contract failed	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	✅ 0.10	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.
eval-performance	claude-sonnet-4.6	➖ Not proven improved	n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%	🟡 0.27	Activation: isolated 8/8; plugin 7/8	Inspect tied or lost stimuli and fix inconsistent skill behavior.
eval-performance	gpt-5.6-luna	➖ Not proven improved	n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%	✅ 0.09	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
extension-points	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded	🟡 0.26	Activation: isolated 4/7; plugin 5/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
extension-points	gpt-5.6-luna	➖ Not proven improved	n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%; 1 dormancy excluded	🟡 0.24	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
including-generated-files	claude-sonnet-4.6	⛔ Activation contract failed	n=7; 3W/0T/4L; d=7; p=0.500; net -14.3%; 1 dormancy excluded	🟡 0.37	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 4/7	Narrow skill routing so the listed off-target scenarios stay dormant.
including-generated-files	gpt-5.6-luna	➖ Not proven improved	n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%; 1 dormancy excluded	✅ 0.17	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
incremental-build	claude-sonnet-4.6	➖ Not proven improved	n=9; 3W/6T/0L; d=3; p=0.125; net +33.3%	🟡 0.33	Activation: isolated 8/9; plugin 2/9	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
incremental-build	gpt-5.6-luna	➖ Not proven improved	n=9; 4W/4T/1L; d=5; p=0.188; net +33.3%	✅ 0.18	Activation: isolated 9/9; plugin 8/9	Inspect tied or lost stimuli and fix inconsistent skill behavior.
item-management	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded	🟡 0.34	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 3/6	Narrow skill routing so the listed off-target scenarios stay dormant.
item-management	gpt-5.6-luna	➖ Not proven improved	n=6; 0W/5T/1L; d=1; p=0.500; net -16.7%; 1 dormancy excluded	✅ 0.11	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	claude-sonnet-4.6	➖ Not proven improved	n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded	🟡 0.29	Activation: isolated 3/7; plugin 2/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	gpt-5.6-luna	➖ Not proven improved	n=7; 2W/2T/3L; d=5; p=0.500; net -14.3%; 1 dormancy excluded	✅ 0.09	Activation: isolated 6/7; plugin 4/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
msbuild-modernization	claude-sonnet-4.6	➖ Not proven improved	n=6; 1W/3T/2L; d=3; p=0.500; net -16.7%; 1 dormancy excluded	🟡 0.32	Activation: isolated 6/6; plugin 5/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-modernization	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.05	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-server	claude-sonnet-4.6	✅ Improved	n=8; 8W/0T/0L; d=8; p=0.004; net +100.0%	🔴 0.52	Activation: isolated 8/8; plugin 7/8	Fix activation gaps; Review overfit evidence.
msbuild-server	gpt-5.6-luna	➖ Not proven improved	n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%	✅ 0.15	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
property-patterns	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	🟡 0.26	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 4/6	Narrow skill routing so the listed off-target scenarios stay dormant.
property-patterns	gpt-5.6-luna	➖ Not proven improved	n=6; 3W/1T/2L; d=5; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.06	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
resolve-project-references	claude-sonnet-4.6	✅ Improved	n=6; 6W/0T/0L; d=6; p=0.016; net +100.0%; 2 dormancy excluded	🟡 0.36	Activation: isolated 6/6; plugin 5/6	Fix activation gaps; Review overfit evidence.
resolve-project-references	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 2 dormancy excluded	✅ 0.15	Activation: isolated 4/6; plugin 5/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
target-authoring	claude-sonnet-4.6	➖ Not proven improved	n=6; 2W/1T/3L; d=5; p=0.500; net -16.7%; 1 dormancy excluded	🟡 0.31	Activation: isolated 6/6; plugin 5/6	Inspect tied or lost stimuli and fix inconsistent skill behavior.
target-authoring	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.10	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

ℹ️ How to read this report

✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
⛔ Activation contract failed — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — build-parallelism (claude-sonnet-4.6)

Why: Net win +50.0% (3W/3T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +22.9% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 3/6; plugin 5/6

Overfit: Moderate (score 0.43)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Enable BuildInParallel on a custom MSBuild task	Eligible	+0.0%	+0.0%	0/1/0
= Enable parallel nodes for a wide project graph	Eligible	+0.0%	+0.0%	0/1/0
▲ Non-activation: make one custom target incremental	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
▲ Preserve valid dependencies and optimize the slow critical-path project	Eligible	+100.0%	+40.0%	1/0/0
= Reduce CI build scope with a solution filter	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Enable BuildInParallel on a custom MSBuild task: The responses make the same correct change and communicate it with essentially equivalent clarity.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — build-perf-baseline (claude-sonnet-4.6)

Why: Net win +33.3% (4W/0T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 4W/0T/2L; d=6; p=0.344; net +33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 5/6

Overfit: Moderate (score 0.43)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/0T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Configure deterministic, cache-safe CI builds	Eligible	-100.0%	-100.0%	0/0/1
▼ Decline a non-MSBuild build performance request	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Leave an already-optimized build unchanged	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Configure deterministic, cache-safe CI builds: A makes the essential requested, CI-guarded ContinuousIntegrationBuild change in the relevant Directory.Build.props and leaves valid configuration. Its unnecessary Deterministic addition and inaccurate default claim are flaws, but B misses the central required property and edi...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — directory-build-organization (gpt-5.6-luna)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +11.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Overfit: Low (score 0.10)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline adding shared build files to a lone project	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Diagnose a package downgrade chain and reorganize version management	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose an inner shared-props file that overwrites its own override	Eligible	+0.0%	+0.0%	0/1/0
= Preserve an intentional project-specific exception while centralizing	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a package downgrade chain and reorganize version management: Both responses reach the same correct conclusion and implementation. However, Response A verified its fix more thoroughly by running both dotnet restore AND dotnet test successfully, and normalized the project XML. Response B ran a build that surfaced unrelated CS1591 errors a...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — including-generated-files (claude-sonnet-4.6)

Why: Net win -14.3% (3W/0T/4L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -12.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=7; 3W/0T/4L; d=7; p=0.500; net -14.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 4/7

Overfit: Moderate (score 0.37)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/1T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline manual wiring for Roslyn source generators	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Diagnose hardcoded obj path for generated source	Eligible	-100.0%	-100.0%	0/0/1
▲ Diagnose missing clean tracking for generated source	Eligible	+100.0%	+40.0%	1/0/0
▼ Diagnose missing output registration for generated non-code file	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose project-level glob for generated source	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose wrong hook for generated source files	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Diagnose hardcoded obj path for generated source: A is precise, consistent with the stated behavior, and supplies the correct reusable pattern. B's recommendation is broadly right, but its root-cause narrative is internally inconsistent and reverses the key path relationship.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — item-management (claude-sonnet-4.6)

Why: Net win +66.7% (4W/2T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +37.1% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 3/6

Overfit: Moderate (score 0.34)

Repeated-run reliability (not used by the gate): 7 paired runs (5W/2T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Decline a target ordering problem with no item-group defect	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
▲ Diagnose an ineffective Compile Remove that does not match the glob	Eligible	+100.0%	+40.0%	1/0/0
▲ Diagnose real and claimed item problems in a code generation pipeline	Eligible	+100.0%	+40.0%	1/0/0
= Fix item management anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
▲ Leave correct single-list batching unchanged	Eligible	+100.0%	+40.0%	1/0/0

Illustrative judge evidence:

Fix item management anti-patterns: The two final project edits implement the same four substantive corrections with no material correctness difference evident in the recorded outputs.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — property-patterns (claude-sonnet-4.6)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 4/6

Overfit: Moderate (score 0.26)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Choose the right OS detection for a cross-platform property	Eligible	-100.0%	-40.0%	0/0/1
▼ Decline a props-versus-targets placement question	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▲ Diagnose a TargetFramework condition that never applies in props	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose multi-level property hierarchy bugs	Eligible	+0.0%	+0.0%	0/1/0
= Fix shared property configuration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Choose the right OS detection for a cross-platform property: Both responses correctly identify the defect and propose explicit Windows/macOS/Linux detection. A is slightly clearer and more directly actionable by naming the MSBuild intrinsic and its required platform names.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — binlog-failure-analysis (claude-sonnet-4.6)

Why: Net win +57.1% (5W/1T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +42.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded

Overfit: Moderate (score 0.24)

Repeated-run reliability (not used by the gate): 8 paired runs (5W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Determine whether a quiet second build actually failed	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose a warning behind a build that actually succeeded	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant for a non-MSBuild build failure log	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Determine whether a quiet second build actually failed: Both answers are correct, concise, and address the user's concern. A is marginally stronger because it presents the key skip/up-to-date evidence directly and adds practical ways to force recompilation.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — binlog-failure-analysis (gpt-5.6-luna)

Why: Net win +28.6% (4W/1T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +2.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/1T/2L; d=6; p=0.344; net +28.6%; 1 dormancy excluded

Overfit: Low (score 0.10)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Assess a requested binlog investigation when the current build is healthy	Eligible	-100.0%	-100.0%	0/0/1
= Confirm the actual resolved target framework and package version from a binlog	Eligible	+0.0%	+0.0%	0/1/0
▼ Fall back to command-line log replay when the usual binlog tool is unavailable	Eligible	-100.0%	-40.0%	0/0/1
= Stay dormant for a non-MSBuild build failure log	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Assess a requested binlog investigation when the current build is healthy: The staged project builds successfully; there is no genuine failure to diagnose. Response A captured a binlog and correctly concluded the build succeeds with 0 warnings/errors, noting only benign SDK-environment diagnostics. Response B, despite using the dedicated skill, manuf...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — binlog-generation (gpt-5.6-luna)

Why: Net win +57.1% (4W/3T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +37.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%; 1 dormancy excluded

Overfit: Moderate (score 0.28)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Build with /bl in PowerShell	Eligible	+0.0%	+0.0%	0/1/0
= Choose a predictable, non-colliding binlog name for a CI upload step	Eligible	+0.0%	+0.0%	0/1/0
▼ Decline a binlog request for a non-MSBuild Java build	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Recognize that a failed build produced no binlog at all	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Build with /bl in PowerShell: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-parallelism (gpt-5.6-luna)

Why: Net win -50.0% (0W/3T/3L over 6 preference-eligible stimulus vote(s), sign test p=0.125), mean preference -25.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 0W/3T/3L; d=3; p=0.125; net -50.0%; 1 dormancy excluded

Warnings: Activation: isolated 5/6; plugin 6/6

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (0W/4T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Analyze build parallelism bottlenecks	Eligible	-100.0%	-40.0%	0/0/1
= Decline graph build for runtime-discovered projects	Eligible	+0.0%	+0.0%	0/1/0
= Enable BuildInParallel on a custom MSBuild task	Eligible	+0.0%	+0.0%	0/1/0
= Enable parallel nodes for a wide project graph	Eligible	+0.0%	+0.0%	0/1/0
= Non-activation: make one custom target incremental	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Preserve valid dependencies and optimize the slow critical-path project	Eligible	-100.0%	-100.0%	0/0/1
▼ Reduce CI build scope with a solution filter	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Analyze build parallelism bottlenecks: Both responses reach the same correct conclusions on all substantive rubric points (serial chain, minimum build time, redundant Tests->Api reference, graph-hygiene-only). They are essentially tied. A edges ahead slightly because its evidence (per-project performance summary wi...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-baseline (gpt-5.6-luna)

Why: Net win +33.3% (4W/0T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +25.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 4W/0T/2L; d=6; p=0.344; net +33.3%; 1 dormancy excluded

Warnings: Activation: isolated 6/6; plugin 5/6

Overfit: Moderate (score 0.23)

Repeated-run reliability (not used by the gate): 7 paired runs (5W/0T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Configure deterministic, cache-safe CI builds	Eligible	+100.0%	+100.0%	1/0/0
▼ Leave an already-optimized build unchanged	Eligible	-100.0%	-40.0%	0/0/1
▼ Route a broken no-op rebuild away from generic optimization	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Leave an already-optimized build unchanged: The ideal answer was to conclude no changes were needed since the project is already optimized (minimal refs, CI-conditioned docs). Both responses failed this by inventing a change. However, B committed exactly the anti-pattern the rubric explicitly calls out (adding UseArtifa...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (claude-sonnet-4.6)

Why: Net win +0.0% (3W/1T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.656), mean preference -20.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/1T/3L; d=6; p=0.656; net +0.0%; 1 dormancy excluded

Warnings: Activation: isolated 6/7; plugin 7/7

Overfit: Moderate (score 0.45)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/1T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	-100.0%	-100.0%	0/0/1
▼ Diagnose NuGet restore running redundantly across CI stages	Eligible	-100.0%	-100.0%	0/0/1
▼ Diagnose a single custom target dominating one project's build	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose evaluation overhead before any target runs	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose slow build for a small project	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose NuGet restore running redundantly across CI stages: A correctly diagnoses the repeated restore cost and supplies all requested concrete CI and MSBuild changes; B fails to locate the available file and produces no substantive answer.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (gpt-5.6-luna)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded

Overfit: Low (score 0.14)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose NuGet restore running redundantly across CI stages	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose a pathological ResolveAssemblyReference time	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose per-project overhead across many small projects	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose slow build for a small project	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose NuGet restore running redundantly across CI stages: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — check-bin-obj-clash (claude-sonnet-4.6)

Why: Net win +28.6% (4W/1T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +20.0% across 7 paired run(s) — not credible (sign test p=0.344 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/1T/2L; d=6; p=0.344; net +28.6%

Warnings: Activation: isolated 6/7; plugin 6/7

Overfit: Low (score 0.19)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/1T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Audit the mixed solution and separate safe projects from unsafe ones	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose multi-targeting outputs that collapse into one path	Eligible	-100.0%	-40.0%	0/0/1
= Fix all clash mechanisms in the mixed solution	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Audit the mixed solution and separate safe projects from unsafe ones: The audits are substantively very similar and correctly classify the risks. A is slightly stronger overall because its LibraryA/LibraryB remediation actually separates both colliding output and intermediate directories; B's fix addresses only obj and its "Unsafe Projects (3)" ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — check-bin-obj-clash (gpt-5.6-luna)

Why: Net win -14.3% (0W/6T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -5.7% across 7 paired run(s) — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 0W/6T/1L; d=1; p=0.500; net -14.3%

Overfit: Low (score 0.13)

Repeated-run reliability (not used by the gate): 7 paired runs (0W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Audit the mixed solution and separate safe projects from unsafe ones	Eligible	-100.0%	-40.0%	0/0/1
= Avoid a false clash report when projects share only the top-level artifacts root	Eligible	+0.0%	+0.0%	0/1/0
= Decline output-clash remediation for separate projects using the default SDK layout	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose multi-targeting outputs that collapse into one path	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose redundant project reference metadata that forks a same-path build	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose shared output and intermediate path collision	Eligible	+0.0%	+0.0%	0/1/0
= Fix all clash mechanisms in the mixed solution	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Audit the mixed solution and separate safe projects from unsafe ones: Both responses reach substantively correct and thorough conclusions with the same technical verification (msbuild property inspection, multi-target checks). The key differentiator is causal attribution: the rubric wants ConsumerApp flagged as unsafe (its reference metadata is ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — directory-build-organization (claude-sonnet-4.6)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded

Warnings: Activation: isolated 5/6; plugin 5/6

Overfit: Moderate (score 0.33)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Apply repo-level build organization cleanup	Eligible	-100.0%	-100.0%	0/0/1
▼ Decline adding shared build files to a lone project	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Diagnose an inner shared-props file that overwrites its own override	Eligible	+0.0%	+0.0%	0/1/0
= Preserve an intentional project-specific exception while centralizing	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Apply repo-level build organization cleanup: A completed the requested build-layout refactor and verified the created shared files, while B only reported an unavailable path and delivered no task result.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — eval-performance (claude-sonnet-4.6)

Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +40.0% across 8 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%

Warnings: Activation: isolated 8/8; plugin 7/8

Overfit: Moderate (score 0.27)

Repeated-run reliability (not used by the gate): 8 paired runs (6W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Redirect a compile-time slowdown mistakenly framed as an evaluation problem	Eligible	+0.0%	+0.0%	0/1/0
▼ Triage which of two property functions actually costs evaluation time	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Redirect a compile-time slowdown mistakenly framed as an evaluation problem: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — eval-performance (gpt-5.6-luna)

Why: Net win +50.0% (5W/2T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +35.0% across 8 paired run(s) — not credible (sign test p=0.109 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 8 paired runs (5W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Detect a project evaluated twice under different global properties	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose stacked evaluation-time patterns (deep imports, broad glob, file-I/O property function)	Eligible	-100.0%	-40.0%	0/0/1
= Triage which of two property functions actually costs evaluation time	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Detect a project evaluated twice under different global properties: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — extension-points (claude-sonnet-4.6)

Why: Net win +14.3% (3W/2T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.500 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded

Warnings: Activation: isolated 4/7; plugin 5/7

Overfit: Moderate (score 0.26)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Diagnose a broken per-TFM forwarder	Eligible	+100.0%	+40.0%	1/0/0
▼ Diagnose a package ID and file-name mismatch	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose build extension point failures	Eligible	-100.0%	-100.0%	0/0/1
= Fix extension point anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
= Non-activation: repair an incremental custom target	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Review packed layout without a false missing-file bug	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a package ID and file-name mismatch: Both are correct, concise, and directly address the silent convention-based import failure. A is marginally stronger as a standalone final result because it includes exact corrected filenames, nuspec entries, and the necessary repackaging step; B's claimed applied fix is usefu...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — extension-points (gpt-5.6-luna)

Why: Net win +57.1% (4W/3T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +22.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%; 1 dormancy excluded

Overfit: Moderate (score 0.24)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose build extension point failures	Eligible	+0.0%	+0.0%	0/1/0
= Fix extension point anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
▼ Non-activation: repair an incremental custom target	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Review packed layout without a false missing-file bug	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose build extension point failures: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — including-generated-files (gpt-5.6-luna)

Why: Net win +42.9% (4W/2T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +15.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%; 1 dormancy excluded

Overfit: Low (score 0.17)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline manual wiring for Roslyn source generators	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose missing clean tracking for generated source	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose missing generated source inclusion	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose missing output registration for generated non-code file	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Diagnose missing clean tracking for generated source: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — incremental-build (claude-sonnet-4.6)

Why: Net win +33.3% (3W/6T/0L over 9 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +13.3% across 9 paired run(s) — not credible — 6 of 9 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 3W/6T/0L; d=3; p=0.125; net +33.3%

Warnings: Activation: isolated 8/9; plugin 2/9

Overfit: Moderate (score 0.33)

Repeated-run reliability (not used by the gate): 9 paired runs (3W/6T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Correct the assumption that Outputs alone enables incremental skipping	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose custom targets that always rerun	Eligible	+0.0%	+0.0%	0/1/0
▲ Explain slower builds when MSBuild skipped everything	Eligible	+100.0%	+40.0%	1/0/0
= Explain why Visual Studio keeps rebuilding an up-to-date project	Eligible	+0.0%	+0.0%	0/1/0
= Explain why clean leaves generated hash source behind	Eligible	+0.0%	+0.0%	0/1/0
= Fix broken incremental targets and clean tracking	Eligible	+0.0%	+0.0%	0/1/0
= Identify a volatile output path that defeats incrementality	Eligible	+0.0%	+0.0%	0/1/0
= Read a diagnostic log to find the stale input	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose custom targets that always rerun: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — incremental-build (gpt-5.6-luna)

Why: Net win +33.3% (4W/4T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +13.3% across 9 paired run(s) — not credible (sign test p=0.188 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 4W/4T/1L; d=5; p=0.188; net +33.3%

Warnings: Activation: isolated 9/9; plugin 8/9

Overfit: Low (score 0.18)

Repeated-run reliability (not used by the gate): 9 paired runs (4W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Diagnose custom targets that always rerun	Eligible	-100.0%	-40.0%	0/0/1
= Distinguish a cold first build from broken incrementality	Eligible	+0.0%	+0.0%	0/1/0
▲ Explain slower builds when MSBuild skipped everything	Eligible	+100.0%	+40.0%	1/0/0
= Fix broken incremental targets and clean tracking	Eligible	+0.0%	+0.0%	0/1/0
= Identify a volatile output path that defeats incrementality	Eligible	+0.0%	+0.0%	0/1/0
= Read a diagnostic log to find the stale input	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose custom targets that always rerun: Both responses correctly diagnose the problem and identify the two targets. A is slightly better because it provides concrete corrected XML with explicit Inputs/Outputs, making the fix actionable, while B stays at a prose description.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — item-management (gpt-5.6-luna)

Why: Net win -16.7% (0W/5T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 0W/5T/1L; d=1; p=0.500; net -16.7%; 1 dormancy excluded

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (0W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a target ordering problem with no item-group defect	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose an ineffective Compile Remove that does not match the glob	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose item group and batching issues	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose real and claimed item problems in a code generation pipeline	Eligible	-100.0%	-40.0%	0/0/1
= Fix item management anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
= Leave correct single-list batching unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose an ineffective Compile Remove that does not match the glob: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-antipatterns (claude-sonnet-4.6)

Why: Net win -14.3% (1W/4T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -17.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded

Warnings: Activation: isolated 3/7; plugin 2/7

Overfit: Moderate (score 0.29)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/4T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a module to an F# project	Eligible	+0.0%	+0.0%	0/1/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
▼ Distinguish a style backslash from a real cross-platform backslash bug	Eligible	-100.0%	-100.0%	0/0/1
▼ Fix broken file order causing FS0039	Eligible	-100.0%	-40.0%	0/0/1
= Judge an unguarded import inside a NuGet package build folder	Eligible	+0.0%	+0.0%	0/1/0
= Leave a clean project without inventing anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
▼ Non-activation: migrate a legacy project to SDK style	Excluded (activation contract)	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Add a module to an F# project: Both runs implement the requested functionality correctly, include the new file in valid F# compilation order, update processing flow for validation failures, and successfully build/run the application. B's placement directly after Domain.fs is marginally tidier, but A's place...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-antipatterns (gpt-5.6-luna)

Why: Net win -14.3% (2W/2T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -2.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 2W/2T/3L; d=5; p=0.500; net -14.3%; 1 dormancy excluded

Warnings: Activation: isolated 6/7; plugin 4/7

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/2T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Add a module to an F# project	Eligible	+100.0%	+40.0%	1/0/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
= Distinguish a style backslash from a real cross-platform backslash bug	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix broken file order causing FS0039	Eligible	-100.0%	-40.0%	0/0/1
▼ Judge an unguarded import inside a NuGet package build folder	Eligible	-100.0%	-40.0%	0/0/1
▼ Non-activation: migrate a legacy project to SDK style	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Review MSBuild files for anti-patterns and style issues	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Add a signature file to define public API: Both responses produced functionally identical, correct results: an appropriate Domain.fsi with all types, placed before Domain.fs in the project file, with a verified clean build. Their approaches, recovery from the apply_patch failure, and final verification were essentially...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-modernization (claude-sonnet-4.6)

Why: Net win -16.7% (1W/3T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +2.9% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/3T/2L; d=3; p=0.500; net -16.7%; 1 dormancy excluded

Warnings: Activation: isolated 6/6; plugin 5/6

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/4T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Apply the migration to SDK-style	Eligible	-100.0%	-40.0%	0/0/1
= Consolidate duplicated projects into a multi-targeting SDK-style project	Eligible	+0.0%	+0.0%	0/1/0
= Decline modernizing a non-.NET build	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Identify legacy patterns for SDK-style migration	Eligible	-100.0%	-40.0%	0/0/1
= Modernize a single project without introducing Central Package Management	Eligible	+0.0%	+0.0%	0/1/0
▲ Modernize further without introducing a nondeterministic language version	Eligible	+100.0%	+100.0%	1/0/0
= Recognize an already-modern project needs no migration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Apply the migration to SDK-style: Both conversions satisfy the core SDK-style migration and should avoid duplicate assembly attributes. A is marginally safer and less behavior-changing because it preserves the complete existing AssemblyInfo source rather than relying on a manual migration of its attributes.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-modernization (gpt-5.6-luna)

Why: Net win +16.7% (1W/5T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +0.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 6 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 1 dormancy excluded

Overfit: Low (score 0.05)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Apply the migration to SDK-style	Eligible	+0.0%	+0.0%	0/1/0
= Consolidate duplicated projects into a multi-targeting SDK-style project	Eligible	+0.0%	+0.0%	0/1/0
▼ Decline modernizing a non-.NET build	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Identify legacy patterns for SDK-style migration	Eligible	+0.0%	+0.0%	0/1/0
= Modernize a single project without introducing Central Package Management	Eligible	+0.0%	+0.0%	0/1/0
= Recognize an already-modern project needs no migration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Apply the migration to SDK-style: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Details for 8 results were omitted to keep this comment under GitHub's limit. Open the workflow summary or Full Results for the complete breakdown.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1118 in dotnet/skills, download eval artifacts with gh run download 34273037202 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/6866836a9b096ae2e9aefefe6dbfdab67405f37f/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

Put required input constraints before overlapping performance, generated-file, item, and property vocabulary identified by the second CI evaluation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d0a0a20-fd83-4343-8434-4763f8120503

github-actions · 2026-09-08T22:25:59Z

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 26b6a729a5c819dc4dd3fb45a8beb43ee14c3ba1 to retry this exact commit.

36 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

Preserve the off-target contracts with leading exclusion rules and retry transient binlog setup failures that otherwise drop one comparison arm. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d0a0a20-fd83-4343-8434-4763f8120503

github-actions · 2026-09-08T23:39:04Z

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate b37f36513cac14d8fd738759cfee7678f3b65483 to retry this exact commit.

36 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

Describe the positive applicability test for the three remaining Sonnet dormancy misses without leading with overlapping off-target vocabulary. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d0a0a20-fd83-4343-8434-4763f8120503

github-actions · 2026-09-09T00:45:30Z

📊 Skill Evaluation Results

36 model/skill results across 18 skills and 2 models — ✅ 6 improved, ➖ 27 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 3 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 8bc2f2078f7640a8ab78eff8e439d1370c816923; 2 judge models.

Measurement health: 36 expected / 36 observed / 36 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
binlog-failure-analysis	claude-sonnet-4.6	✅ Improved	n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 1 dormancy excluded	✅ 0.19	—	None.
binlog-failure-analysis	gpt-5.6-luna	✅ Improved	n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%; 1 dormancy excluded	✅ 0.11	—	None.
binlog-generation	claude-sonnet-4.6	✅ Improved	n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%; 1 dormancy excluded	🔴 0.56	Activation: isolated 7/7; plugin 6/7	Fix activation gaps; Review overfit evidence.
binlog-generation	gpt-5.6-luna	✅ Improved	n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 1 dormancy excluded	🟡 0.26	—	Review overfit evidence.
build-parallelism	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	🟡 0.32	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
build-parallelism	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.11	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
build-perf-baseline	claude-sonnet-4.6	➖ Not proven improved	n=6; 2W/0T/4L; d=6; p=0.344; net -33.3%; 1 dormancy excluded	🟡 0.35	Activation: isolated 5/6; plugin 5/6	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-baseline	gpt-5.6-luna	➖ Not proven improved	n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 1 dormancy excluded	🟡 0.25	Activation: isolated 6/6; plugin 5/6	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-diagnostics	claude-sonnet-4.6	➖ Not proven improved	n=7; 4W/1T/2L; d=6; p=0.344; net +28.6%; 1 dormancy excluded	🔴 0.50	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-diagnostics	gpt-5.6-luna	➖ Not proven improved	n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded	✅ 0.16	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
check-bin-obj-clash	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%	✅ 0.18	Activation: isolated 6/7; plugin 6/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
check-bin-obj-clash	gpt-5.6-luna	➖ Not proven improved	n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%	✅ 0.11	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	claude-sonnet-4.6	➖ Not proven improved	n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 1 dormancy excluded	🟡 0.36	Activation: isolated 4/6; plugin 5/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.07	Activation: isolated 6/6; plugin 5/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
eval-performance	claude-sonnet-4.6	➖ Not proven improved	n=8; 5W/1T/2L; d=7; p=0.227; net +37.5%	🟡 0.36	Activation: isolated 8/8; plugin 7/8	Inspect tied or lost stimuli and fix inconsistent skill behavior.
eval-performance	gpt-5.6-luna	✅ Improved	n=8; 7W/1T/0L; d=7; p=0.008; net +87.5%	✅ 0.09	—	None.
extension-points	claude-sonnet-4.6	➖ Not proven improved	n=7; 2W/3T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded	🟡 0.41	Activation: isolated 4/7; plugin 4/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
extension-points	gpt-5.6-luna	➖ Not proven improved	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded	🟡 0.29	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
including-generated-files	claude-sonnet-4.6	⛔ Activation contract failed	n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded	🟡 0.46	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 6/7	Narrow skill routing so the listed off-target scenarios stay dormant.
including-generated-files	gpt-5.6-luna	➖ Not proven improved	n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	✅ 0.17	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
incremental-build	claude-sonnet-4.6	➖ Not proven improved	n=9; 4W/5T/0L; d=4; p=0.063; net +44.4%	🟡 0.45	Activation: isolated 8/9; plugin 4/9	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
incremental-build	gpt-5.6-luna	➖ Not proven improved	n=9; 3W/5T/1L; d=4; p=0.312; net +22.2%	✅ 0.14	Activation: isolated 9/9; plugin 7/9	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
item-management	claude-sonnet-4.6	➖ Not proven improved	n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	🟡 0.32	Activation: isolated 4/6; plugin 3/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
item-management	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.11	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	claude-sonnet-4.6	➖ Not proven improved	n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	🟡 0.32	Activation: isolated 2/7; plugin 2/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	gpt-5.6-luna	➖ Not proven improved	n=7; 0W/6T/1L; d=1; p=0.500; net -14.3%; 1 dormancy excluded	✅ 0.08	Activation: isolated 6/7; plugin 5/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-modernization	claude-sonnet-4.6	➖ Not proven improved	n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	🟡 0.29	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-modernization	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 1 dormancy excluded	✅ 0.06	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-server	claude-sonnet-4.6	✅ Improved	n=8; 7W/1T/0L; d=7; p=0.008; net +87.5%	🟡 0.43	—	Review overfit evidence.
msbuild-server	gpt-5.6-luna	➖ Not proven improved	n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%	✅ 0.19	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
property-patterns	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded	🟡 0.24	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 4/6	Narrow skill routing so the listed off-target scenarios stay dormant.
property-patterns	gpt-5.6-luna	➖ Not proven improved	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	✅ 0.08	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
resolve-project-references	claude-sonnet-4.6	➖ Not proven improved	n=6; 5W/0T/1L; d=6; p=0.109; net +66.7%; 2 dormancy excluded	🟡 0.36	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
resolve-project-references	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 2 dormancy excluded	✅ 0.15	Activation: isolated 4/6; plugin 6/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
target-authoring	claude-sonnet-4.6	➖ Not proven improved	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	🟡 0.35	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
target-authoring	gpt-5.6-luna	➖ Not proven improved	n=6; 3W/1T/2L; d=5; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.12	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.

ℹ️ How to read this report

✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
⛔ Activation contract failed — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — build-parallelism (claude-sonnet-4.6)

Why: Net win +16.7% (2W/3T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +20.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline graph build for runtime-discovered projects	Eligible	+0.0%	+0.0%	0/1/0
= Enable BuildInParallel on a custom MSBuild task	Eligible	+0.0%	+0.0%	0/1/0
= Enable parallel nodes for a wide project graph	Eligible	+0.0%	+0.0%	0/1/0
▲ Non-activation: make one custom target incremental	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
▼ Reduce CI build scope with a solution filter	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Decline graph build for runtime-discovered projects: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — including-generated-files (claude-sonnet-4.6)

Why: Net win -14.3% (1W/4T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -17.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 6/7

Overfit: Moderate (score 0.46)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/4T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline manual wiring for Roslyn source generators	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Diagnose missing clean tracking for generated source	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose missing generated source inclusion	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose missing output registration for generated non-code file	Eligible	-100.0%	-100.0%	0/0/1
= Diagnose project-level glob for generated source	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose wrong hook for generated source files	Eligible	-100.0%	-40.0%	0/0/1
= Fix generated source inclusion and clean tracking	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose missing clean tracking for generated source: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — property-patterns (claude-sonnet-4.6)

Why: Net win +66.7% (4W/2T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +45.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 4/6

Overfit: Moderate (score 0.24)

Repeated-run reliability (not used by the gate): 7 paired runs (5W/2T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Choose the right OS detection for a cross-platform property	Eligible	+0.0%	+0.0%	0/1/0
▲ Decline a props-versus-targets placement question	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
▲ Diagnose a TargetFramework condition that never applies in props	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose multi-level property hierarchy bugs	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Choose the right OS detection for a cross-platform property: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-parallelism (gpt-5.6-luna)

Why: Net win +16.7% (2W/3T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Analyze build parallelism bottlenecks	Eligible	+0.0%	+0.0%	0/1/0
▼ Decline graph build for runtime-discovered projects	Eligible	-100.0%	-40.0%	0/0/1
= Enable BuildInParallel on a custom MSBuild task	Eligible	+0.0%	+0.0%	0/1/0
= Enable parallel nodes for a wide project graph	Eligible	+0.0%	+0.0%	0/1/0
= Non-activation: make one custom target incremental	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Analyze build parallelism bottlenecks: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-baseline (claude-sonnet-4.6)

Why: Net win -33.3% (2W/0T/4L over 6 preference-eligible stimulus vote(s), sign test p=0.344), mean preference -25.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/0T/4L; d=6; p=0.344; net -33.3%; 1 dormancy excluded

Warnings: Activation: isolated 5/6; plugin 5/6

Overfit: Moderate (score 0.35)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/0T/5L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Configure deterministic, cache-safe CI builds	Eligible	-100.0%	-40.0%	0/0/1
▼ Decline a non-MSBuild build performance request	Excluded (activation contract)	-100.0%	-100.0%	0/0/1
▼ Leave an already-optimized build unchanged	Eligible	-100.0%	-100.0%	0/0/1
▼ Route a broken no-op rebuild away from generic optimization	Eligible	-100.0%	-40.0%	0/0/1
▼ Route a restore-bound cold build away from architecture changes	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Configure deterministic, cache-safe CI builds: Both miss the required CI-guarded ContinuousIntegrationBuild change and give materially misleading root-cause explanations. A is only marginally better because it acknowledges the SDK default for Deterministic, even though its chosen fix remains unnecessary and insufficient.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-baseline (gpt-5.6-luna)

Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +31.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 1 dormancy excluded

Warnings: Activation: isolated 6/6; plugin 5/6

Overfit: Moderate (score 0.25)

Repeated-run reliability (not used by the gate): 7 paired runs (5W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Configure deterministic, cache-safe CI builds	Eligible	-100.0%	-40.0%	0/0/1
= Route a broken no-op rebuild away from generic optimization	Eligible	+0.0%	+0.0%	0/1/0
▲ Route a restore-bound cold build away from architecture changes	Eligible	+100.0%	+40.0%	1/0/0

Illustrative judge evidence:

Configure deterministic, cache-safe CI builds: The responses are nearly identical in approach, change, and outcome. Both added ContinuousIntegrationBuild unconditionally (missing the CI-only guard), both verified the build, both explained the rationale briefly. A slightly edges out by additionally verifying DeterministicSo...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (claude-sonnet-4.6)

Why: Net win +28.6% (4W/1T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +5.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/1T/2L; d=6; p=0.344; net +28.6%; 1 dormancy excluded

Overfit: High (score 0.50)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/1T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Diagnose NuGet restore running redundantly across CI stages	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose evaluation overhead before any target runs	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose per-project overhead across many small projects	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose NuGet restore running redundantly across CI stages: The core diagnosis and prescribed changes are essentially the same, but A is more concise and avoids B's notably misleading after-time table: a dedicated restore still costs about 9.7s, and test/publish have work beyond compilation, so a 4–5s total CI estimate is unsupported. ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (gpt-5.6-luna)

Why: Net win +57.1% (5W/1T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +20.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded

Overfit: Low (score 0.16)

Repeated-run reliability (not used by the gate): 8 paired runs (5W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Diagnose NuGet restore running redundantly across CI stages	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose evaluation overhead before any target runs	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose NuGet restore running redundantly across CI stages: Both responses are essentially correct and equivalent on the core diagnosis and main recommendations, covering all three rubric points. A edges ahead slightly by adding --locked-mode (appropriate given the committed lock file) and explicitly noting the need to preserve restore...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — check-bin-obj-clash (claude-sonnet-4.6)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +20.0% across 7 paired run(s) — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%

Warnings: Activation: isolated 6/7; plugin 6/7

Overfit: Low (score 0.18)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Audit the mixed solution and separate safe projects from unsafe ones	Eligible	+0.0%	+0.0%	0/1/0
= Decline output-clash remediation for separate projects using the default SDK layout	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose multi-targeting outputs that collapse into one path	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix all clash mechanisms in the mixed solution	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Audit the mixed solution and separate safe projects from unsafe ones: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — check-bin-obj-clash (gpt-5.6-luna)

Why: Net win +57.1% (4W/3T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +22.9% across 7 paired run(s) — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Audit the mixed solution and separate safe projects from unsafe ones	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose multi-targeting outputs that collapse into one path	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose shared output and intermediate path collision	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Audit the mixed solution and separate safe projects from unsafe ones: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — directory-build-organization (claude-sonnet-4.6)

Why: Net win +33.3% (2W/4T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +20.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 6 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 1 dormancy excluded

Warnings: Activation: isolated 4/6; plugin 5/6

Overfit: Moderate (score 0.36)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline adding shared build files to a lone project	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that silently skips in props	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a package downgrade chain and reorganize version management	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose an inner shared-props file that overwrites its own override	Eligible	+0.0%	+0.0%	0/1/0
= Preserve an intentional project-specific exception while centralizing	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a TargetFramework condition that silently skips in props: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — directory-build-organization (gpt-5.6-luna)

Why: Net win +16.7% (1W/5T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 6 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 1 dormancy excluded

Warnings: Activation: isolated 6/6; plugin 5/6

Overfit: Low (score 0.07)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/6T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Apply repo-level build organization cleanup	Eligible	+0.0%	+0.0%	0/1/0
= Decline adding shared build files to a lone project	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that silently skips in props	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a package downgrade chain and reorganize version management	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose an inner shared-props file that overwrites its own override	Eligible	+0.0%	+0.0%	0/1/0
= Preserve an intentional project-specific exception while centralizing	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Apply repo-level build organization cleanup: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — eval-performance (claude-sonnet-4.6)

Why: Net win +37.5% (5W/1T/2L over 8 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +30.0% across 8 paired run(s) — not credible (sign test p=0.227 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 5W/1T/2L; d=7; p=0.227; net +37.5%

Warnings: Activation: isolated 8/8; plugin 7/8

Overfit: Moderate (score 0.36)

Repeated-run reliability (not used by the gate): 8 paired runs (5W/1T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Recognize TreatAsLocalProperty overuse versus one justified entry	Eligible	-100.0%	-40.0%	0/0/1
▼ Redirect a compile-time slowdown mistakenly framed as an evaluation problem	Eligible	-100.0%	-40.0%	0/0/1
= Triage which of two property functions actually costs evaluation time	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Recognize TreatAsLocalProperty overuse versus one justified entry: Both satisfy every requested rubric item and give the right change. A is marginally stronger because it avoids B's questionable claims about blocking parent overrides and inheritance behavior.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — extension-points (claude-sonnet-4.6)

Why: Net win +0.0% (2W/3T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.687), mean preference -12.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 2W/3T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded

Warnings: Activation: isolated 4/7; plugin 4/7

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/3T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Create extensibility hooks for a custom SDK target file	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose a broken per-TFM forwarder	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose a package ID and file-name mismatch	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix extension point anti-patterns	Eligible	-100.0%	-100.0%	0/0/1
▼ Non-activation: repair an incremental custom target	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Review packed layout without a false missing-file bug	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Create extensibility hooks for a custom SDK target file: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — extension-points (gpt-5.6-luna)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +22.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded

Overfit: Moderate (score 0.29)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose NuGet package and repo extension conflicts	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose build extension point failures	Eligible	-100.0%	-40.0%	0/0/1
= Fix extension point anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
= Review packed layout without a false missing-file bug	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose NuGet package and repo extension conflicts: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — including-generated-files (gpt-5.6-luna)

Why: Net win +0.0% (1W/5T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Overfit: Low (score 0.17)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline manual wiring for Roslyn source generators	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose hardcoded obj path for generated source	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose missing clean tracking for generated source	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose missing generated source inclusion	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose missing output registration for generated non-code file	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose project-level glob for generated source	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose wrong hook for generated source files	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose hardcoded obj path for generated source: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — incremental-build (claude-sonnet-4.6)

Why: Net win +44.4% (4W/5T/0L over 9 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +17.8% across 9 paired run(s) — not credible — 5 of 9 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 4W/5T/0L; d=4; p=0.063; net +44.4%

Warnings: Activation: isolated 8/9; plugin 4/9

Overfit: Moderate (score 0.45)

Repeated-run reliability (not used by the gate): 9 paired runs (4W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Correct the assumption that Outputs alone enables incremental skipping	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose custom targets that always rerun	Eligible	+0.0%	+0.0%	0/1/0
= Explain slower builds when MSBuild skipped everything	Eligible	+0.0%	+0.0%	0/1/0
= Explain why Visual Studio keeps rebuilding an up-to-date project	Eligible	+0.0%	+0.0%	0/1/0
▲ Explain why clean leaves generated hash source behind	Eligible	+100.0%	+40.0%	1/0/0
= Fix broken incremental targets and clean tracking	Eligible	+0.0%	+0.0%	0/1/0
= Identify a volatile output path that defeats incrementality	Eligible	+0.0%	+0.0%	0/1/0
▲ Read a diagnostic log to find the stale input	Eligible	+100.0%	+40.0%	1/0/0

Illustrative judge evidence:

Diagnose custom targets that always rerun: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — incremental-build (gpt-5.6-luna)

Why: Net win +22.2% (3W/5T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +15.6% across 9 paired run(s) — not credible — 5 of 9 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 3W/5T/1L; d=4; p=0.312; net +22.2%

Warnings: Activation: isolated 9/9; plugin 7/9

Overfit: Low (score 0.14)

Repeated-run reliability (not used by the gate): 9 paired runs (3W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose custom targets that always rerun	Eligible	+0.0%	+0.0%	0/1/0
= Explain slower builds when MSBuild skipped everything	Eligible	+0.0%	+0.0%	0/1/0
▼ Explain why Visual Studio keeps rebuilding an up-to-date project	Eligible	-100.0%	-40.0%	0/0/1
= Fix broken incremental targets and clean tracking	Eligible	+0.0%	+0.0%	0/1/0
= Identify a volatile output path that defeats incrementality	Eligible	+0.0%	+0.0%	0/1/0
= Read a diagnostic log to find the stale input	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose custom targets that always rerun: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — item-management (claude-sonnet-4.6)

Why: Net win +0.0% (1W/4T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.750), mean preference -8.6% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Warnings: Activation: isolated 4/6; plugin 3/6

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a target ordering problem with no item-group defect	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Diagnose an ineffective Compile Remove that does not match the glob	Eligible	-100.0%	-100.0%	0/0/1
= Diagnose real and claimed item problems in a code generation pipeline	Eligible	+0.0%	+0.0%	0/1/0
= Fix item management anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
= Leave correct single-list batching unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose an ineffective Compile Remove that does not match the glob: A directly and accurately answers the actual cause and fix. B supplies a misleading, technically incorrect separator-based root cause, failing the core requested diagnosis despite ending with a usable-looking glob.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — item-management (gpt-5.6-luna)

Why: Net win +16.7% (2W/3T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a target ordering problem with no item-group defect	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Diagnose an ineffective Compile Remove that does not match the glob	Eligible	-100.0%	-40.0%	0/0/1
= Fix item management anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
= Leave correct single-list batching unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose an ineffective Compile Remove that does not match the glob: Both reach the same correct diagnosis and avoid recommending Exclude. However, Response A actually applied the fix to the file (recovering from the failed apply_patch by using sed) and verified via dotnet msbuild -getItem:Compile that Api.g.cs is now excluded while Service.c...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-antipatterns (claude-sonnet-4.6)

Why: Net win +0.0% (1W/5T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Warnings: Activation: isolated 2/7; plugin 2/7

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a module to an F# project	Eligible	+0.0%	+0.0%	0/1/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
= Distinguish a style backslash from a real cross-platform backslash bug	Eligible	+0.0%	+0.0%	0/1/0
= Fix broken file order causing FS0039	Eligible	+0.0%	+0.0%	0/1/0
▼ Judge an unguarded import inside a NuGet package build folder	Eligible	-100.0%	-40.0%	0/0/1
= Non-activation: migrate a legacy project to SDK style	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Review MSBuild files for anti-patterns and style issues	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Add a module to an F# project: The implementations and verification outcomes are materially identical, fully addressing the requested validation, compilation ordering, and guarded processing behavior.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-antipatterns (gpt-5.6-luna)

Why: Net win -14.3% (0W/6T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 0W/6T/1L; d=1; p=0.500; net -14.3%; 1 dormancy excluded

Warnings: Activation: isolated 6/7; plugin 5/7

Overfit: Low (score 0.08)

Repeated-run reliability (not used by the gate): 8 paired runs (0W/6T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a module to an F# project	Eligible	+0.0%	+0.0%	0/1/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
= Distinguish a style backslash from a real cross-platform backslash bug	Eligible	+0.0%	+0.0%	0/1/0
= Fix broken file order causing FS0039	Eligible	+0.0%	+0.0%	0/1/0
▼ Judge an unguarded import inside a NuGet package build folder	Eligible	-100.0%	-40.0%	0/0/1
= Leave a clean project without inventing anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
▼ Non-activation: migrate a legacy project to SDK style	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Review MSBuild files for anti-patterns and style issues	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Add a module to an F# project: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-modernization (claude-sonnet-4.6)

Why: Net win +0.0% (1W/4T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +22.9% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Overfit: Moderate (score 0.29)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Apply the migration to SDK-style	Eligible	-100.0%	-40.0%	0/0/1
= Consolidate duplicated projects into a multi-targeting SDK-style project	Eligible	+0.0%	+0.0%	0/1/0
= Identify legacy patterns for SDK-style migration	Eligible	+0.0%	+0.0%	0/1/0
= Modernize a single project without introducing Central Package Management	Eligible	+0.0%	+0.0%	0/1/0
= Recognize an already-modern project needs no migration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Apply the migration to SDK-style: Both conversions satisfy the core SDK-style migration requirements. A is marginally safer for fidelity because it retains all preexisting assembly metadata directly rather than requiring every attribute to be accurately translated before deleting AssemblyInfo.cs.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-modernization (gpt-5.6-luna)

Why: Net win +33.3% (2W/4T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +11.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 6 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 1 dormancy excluded

Overfit: Low (score 0.06)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Consolidate duplicated projects into a multi-targeting SDK-style project	Eligible	+0.0%	+0.0%	0/1/0
= Decline modernizing a non-.NET build	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Identify legacy patterns for SDK-style migration	Eligible	+0.0%	+0.0%	0/1/0
= Modernize a single project without introducing Central Package Management	Eligible	+0.0%	+0.0%	0/1/0
= Recognize an already-modern project needs no migration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Consolidate duplicated projects into a multi-targeting SDK-style project: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-server (gpt-5.6-luna)

Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +40.0% across 8 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%

Overfit: Low (score 0.19)

Repeated-run reliability (not used by the gate): 8 paired runs (6W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Confirm the MSBuild Server is actually improving build times before declaring success	Eligible	-100.0%	-40.0%	0/0/1
= Decline MSBuild Server for a single one-off release build	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Confirm the MSBuild Server is actually improving build times before declaring success: Both responses give solid, empirically-grounded advice with cold/warm comparison and A/B testing. A adds value by investigating the actual project (discovering it's a tiny LoopApp), warning that the trivial project gives no meaningful signal, and providing a diagnostic log gre...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — property-patterns (gpt-5.6-luna)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +11.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded

Overfit: Low (score 0.08)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a props-versus-targets placement question	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that never applies in props	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose shared build property issues	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix shared property configuration	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Diagnose a TargetFramework condition that never applies in props: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — resolve-project-references (claude-sonnet-4.6)

Why: Net win +66.7% (5W/0T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +37.5% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 5W/0T/1L; d=6; p=0.109; net +66.7%; 2 dormancy excluded

Overfit: Moderate (score 0.36)

Repeated-run reliability (not used by the gate): 8 paired runs (7W/0T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Distinguish wait time from a real serial dependency chain	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Distinguish wait time from a real serial dependency chain: Both are strong and satisfy every requested point. A is marginally more technically precise and avoids conflating the inflated target accounting with a direct bottleneck, while still clearly identifying the serial graph and remedy.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — resolve-project-references (gpt-5.6-luna)

Why: Net win +16.7% (2W/3T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +17.5% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 2 dormancy excluded

Warnings: Activation: isolated 4/6; plugin 6/6

Overfit: Low (score 0.15)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Distinguish wait time from a real serial dependency chain	Eligible	+0.0%	+0.0%	0/1/0
▲ Give the exact command to replay task self-time	Eligible	+100.0%	+100.0%	1/0/0
= Rank Copy ahead of Csc when task self-time is higher	Eligible	+0.0%	+0.0%	0/1/0
= Redirect from misleading target summary to Csc self-time	Eligible	+0.0%	+0.0%	0/1/0
▼ Require diagnostic evidence before analyzing project-reference time	Eligible	-100.0%	-40.0%	0/0/1
= Stay dormant when Csc is already the obvious bottleneck	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Distinguish wait time from a real serial dependency chain: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 3 results are in Full Results.

Details for 5 results were omitted to keep this comment under GitHub's limit. Open the workflow summary or Full Results for the complete breakdown.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1118 in dotnet/skills, download eval artifacts with gh run download 34291893629 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/8bc2f2078f7640a8ab78eff8e439d1370c816923/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

Remove off-target vocabulary from the three descriptions that Sonnet selected before any workspace inspection, while retaining full boundary guidance in the skill bodies. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d0a0a20-fd83-4343-8434-4763f8120503

github-actions · 2026-09-09T01:58:56Z

📊 Skill Evaluation Results

36 model/skill results across 18 skills and 2 models — ✅ 5 improved, ➖ 26 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 5 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 532f862a7cac6f521a7cc0b621c5379fd6e81bf9; 2 judge models.

Measurement health: 36 expected / 36 observed / 36 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
binlog-failure-analysis	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded	🟡 0.24	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
binlog-failure-analysis	gpt-5.6-luna	✅ Improved	n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%; 1 dormancy excluded	✅ 0.11	—	None.
binlog-generation	claude-sonnet-4.6	✅ Improved	n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 1 dormancy excluded	🟡 0.47	—	Review overfit evidence.
binlog-generation	gpt-5.6-luna	➖ Not proven improved	n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded	🟡 0.25	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-parallelism	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 3W/1T/2L; d=5; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.17	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
build-parallelism	gpt-5.6-luna	➖ Not proven improved	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	✅ 0.14	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
build-perf-baseline	claude-sonnet-4.6	➖ Not proven improved	n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 1 dormancy excluded	🟡 0.38	Activation: isolated 5/6; plugin 5/6	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-baseline	gpt-5.6-luna	➖ Not proven improved	n=6; 4W/0T/2L; d=6; p=0.344; net +33.3%; 1 dormancy excluded	🟡 0.24	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-diagnostics	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/1T/3L; d=6; p=0.656; net +0.0%; 1 dormancy excluded	🟡 0.49	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-diagnostics	gpt-5.6-luna	➖ Not proven improved	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded	✅ 0.17	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
check-bin-obj-clash	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%	🟡 0.20	Activation: isolated 6/7; plugin 5/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
check-bin-obj-clash	gpt-5.6-luna	➖ Not proven improved	n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%	✅ 0.10	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	claude-sonnet-4.6	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	🟡 0.23	Activation: isolated 4/6; plugin 4/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	gpt-5.6-luna	➖ Not proven improved	n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded	✅ 0.07	Activation: isolated 5/6; plugin 6/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
eval-performance	claude-sonnet-4.6	➖ Not proven improved	n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%	🟡 0.37	Activation: isolated 8/8; plugin 7/8	Inspect tied or lost stimuli and fix inconsistent skill behavior.
eval-performance	gpt-5.6-luna	➖ Not proven improved	n=8; 3W/5T/0L; d=3; p=0.125; net +37.5%	✅ 0.10	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
extension-points	claude-sonnet-4.6	➖ Not proven improved	n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded	🟡 0.39	Activation: isolated 4/7; plugin 5/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
extension-points	gpt-5.6-luna	➖ Not proven improved	n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%; 1 dormancy excluded	🟡 0.25	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
including-generated-files	claude-sonnet-4.6	⛔ Activation contract failed	n=7; 2W/2T/3L; d=5; p=0.500; net -14.3%; 1 dormancy excluded	🟡 0.37	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 3/7	Narrow skill routing so the listed off-target scenarios stay dormant.
including-generated-files	gpt-5.6-luna	➖ Not proven improved	n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded	✅ 0.17	Activation: isolated 6/7; plugin 6/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
incremental-build	claude-sonnet-4.6	✅ Improved	n=9; 5W/4T/0L; d=5; p=0.031; net +55.6%	🟡 0.25	Activation: isolated 7/9; plugin 6/9	Fix activation gaps; Review overfit evidence.
incremental-build	gpt-5.6-luna	✅ Improved	n=9; 5W/4T/0L; d=5; p=0.031; net +55.6%	✅ 0.09	Activation: isolated 9/9; plugin 8/9	Fix activation gaps.
item-management	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	🟡 0.33	Dormancy contract: 1 unexpected activation(s); Activation: isolated 4/6; plugin 2/6	Narrow skill routing so the listed off-target scenarios stay dormant.
item-management	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.11	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	claude-sonnet-4.6	➖ Not proven improved	n=7; 2W/5T/0L; d=2; p=0.250; net +28.6%; 1 dormancy excluded	🟡 0.35	Activation: isolated 2/7; plugin 2/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	gpt-5.6-luna	➖ Not proven improved	n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	✅ 0.09	Activation: isolated 6/7; plugin 6/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-modernization	claude-sonnet-4.6	➖ Not proven improved	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	🟡 0.35	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-modernization	gpt-5.6-luna	➖ Not proven improved	n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded	✅ 0.05	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-server	claude-sonnet-4.6	➖ Not proven improved	n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%	🟡 0.50	Activation: isolated 8/8; plugin 7/8	Inspect tied or lost stimuli and fix inconsistent skill behavior.
msbuild-server	gpt-5.6-luna	➖ Not proven improved	n=8; 4W/4T/0L; d=4; p=0.063; net +50.0%	✅ 0.18	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
property-patterns	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	🟡 0.27	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 4/6	Narrow skill routing so the listed off-target scenarios stay dormant.
property-patterns	gpt-5.6-luna	⛔ Activation contract failed	n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded	✅ 0.05	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
resolve-project-references	claude-sonnet-4.6	✅ Improved	n=6; 6W/0T/0L; d=6; p=0.016; net +100.0%; 2 dormancy excluded	🟡 0.34	—	Review overfit evidence.
resolve-project-references	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/2T/3L; d=4; p=0.312; net -33.3%; 2 dormancy excluded	✅ 0.14	Activation: isolated 3/6; plugin 6/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
target-authoring	claude-sonnet-4.6	➖ Not proven improved	n=6; 3W/1T/2L; d=5; p=0.500; net +16.7%; 1 dormancy excluded	🟡 0.31	Activation: isolated 6/6; plugin 5/6	Inspect tied or lost stimuli and fix inconsistent skill behavior.
target-authoring	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.14	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

ℹ️ How to read this report

✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
⛔ Activation contract failed — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — build-parallelism (claude-sonnet-4.6)

Why: Net win +16.7% (3W/1T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +8.6% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/1T/2L; d=5; p=0.500; net +16.7%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6

Overfit: Low (score 0.17)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/1T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Enable BuildInParallel on a custom MSBuild task	Eligible	-100.0%	-40.0%	0/0/1
= Enable parallel nodes for a wide project graph	Eligible	+0.0%	+0.0%	0/1/0
▼ Non-activation: make one custom target incremental	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Reduce CI build scope with a solution filter	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Enable BuildInParallel on a custom MSBuild task: The implemented repository change is equivalent and correct in both responses. A's final explanation is marginally more precise, whereas B makes an unnecessary guarantee about simultaneous separate worker nodes.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — including-generated-files (claude-sonnet-4.6)

Why: Net win -14.3% (2W/2T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=7; 2W/2T/3L; d=5; p=0.500; net -14.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 3/7

Overfit: Moderate (score 0.37)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/2T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline manual wiring for Roslyn source generators	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▲ Diagnose hardcoded obj path for generated source	Eligible	+100.0%	+40.0%	1/0/0
▼ Diagnose missing clean tracking for generated source	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose missing generated source inclusion	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose missing output registration for generated non-code file	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose project-level glob for generated source	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose wrong hook for generated source files	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Diagnose missing clean tracking for generated source: Both give the core correct diagnosis and fix. A is marginally stronger because its account of what a lingering file does after clean is more technically precise, whereas B makes an unsupported 'alongside it' compilation claim.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — item-management (claude-sonnet-4.6)

Why: Net win +16.7% (2W/3T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 4/6; plugin 2/6

Overfit: Moderate (score 0.33)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a target ordering problem with no item-group defect	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose an ineffective Compile Remove that does not match the glob	Eligible	+0.0%	+0.0%	0/1/0
▲ Diagnose real and claimed item problems in a code generation pipeline	Eligible	+100.0%	+40.0%	1/0/0
= Fix item management anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
▼ Leave correct single-list batching unchanged	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Diagnose an ineffective Compile Remove that does not match the glob: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — property-patterns (claude-sonnet-4.6)

Why: Net win +0.0% (1W/4T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 4/6

Overfit: Moderate (score 0.27)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Choose the right OS detection for a cross-platform property	Eligible	+0.0%	+0.0%	0/1/0
= Decline a props-versus-targets placement question	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that never applies in props	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose multi-level property hierarchy bugs	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix shared property configuration	Eligible	-100.0%	-40.0%	0/0/1
= Leave already-correct shared properties unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Choose the right OS detection for a cross-platform property: Position-swap inconsistent (forward: A, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — property-patterns (gpt-5.6-luna)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6

Overfit: Low (score 0.05)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Choose the right OS detection for a cross-platform property	Eligible	+100.0%	+40.0%	1/0/0
▼ Decline a props-versus-targets placement question	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Diagnose a TargetFramework condition that never applies in props	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose multi-level property hierarchy bugs	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose shared build property issues	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a TargetFramework condition that never applies in props: Both responses reach identical, correct conclusions: the condition fails because TargetFramework is empty when Directory.Build.props evaluates, and both move the settings to Directory.Build.targets, verifying the fix with dotnet msbuild showing correct evaluated properties. Th...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — binlog-failure-analysis (claude-sonnet-4.6)

Why: Net win +14.3% (3W/2T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +12.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.500 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded

Overfit: Moderate (score 0.24)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Confirm the actual resolved target framework and package version from a binlog	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose a warning behind a build that actually succeeded	Eligible	+0.0%	+0.0%	0/1/0
= Fall back to command-line log replay when the usual binlog tool is unavailable	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant for a non-MSBuild build failure log	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Trace why a generated source file is missing at compile time	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Confirm the actual resolved target framework and package version from a binlog: Both deliver the two requested resolved values correctly, but A more accurately reports and contextualizes the package-versus-assembly version distinction. B's erroneous claim about the assembly version is a material avoidable inaccuracy in a compliance-oriented response.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — binlog-generation (gpt-5.6-luna)

Why: Net win +57.1% (5W/1T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +55.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded

Overfit: Moderate (score 0.25)

Repeated-run reliability (not used by the gate): 8 paired runs (6W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Choose a predictable, non-colliding binlog name for a CI upload step	Eligible	+0.0%	+0.0%	0/1/0
▼ Recognize that a failed build produced no binlog at all	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Choose a predictable, non-colliding binlog name for a CI upload step: Both responses achieved identical correct outcomes: read the manifest, chose explicit non-colliding debug-4.binlog and release-4.binlog names, recovered from the same initial cwd error, and built successfully. A added explicit test ! -e existence guards which is a minor robust...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-parallelism (gpt-5.6-luna)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +11.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded

Overfit: Low (score 0.14)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Enable BuildInParallel on a custom MSBuild task	Eligible	+0.0%	+0.0%	0/1/0
= Enable parallel nodes for a wide project graph	Eligible	+0.0%	+0.0%	0/1/0
= Non-activation: make one custom target incremental	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Reduce CI build scope with a solution filter	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Enable BuildInParallel on a custom MSBuild task: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-baseline (claude-sonnet-4.6)

Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +2.9% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 1 dormancy excluded

Warnings: Activation: isolated 5/6; plugin 5/6

Overfit: Moderate (score 0.38)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/1T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Configure deterministic, cache-safe CI builds	Eligible	-100.0%	-100.0%	0/0/1
▼ Decline a non-MSBuild build performance request	Excluded (activation contract)	-100.0%	-100.0%	0/0/1
= Route a restore-bound cold build away from architecture changes	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Configure deterministic, cache-safe CI builds: A makes the requested, reference-aligned CI-only ContinuousIntegrationBuild change and addresses the actual cross-agent path-normalization issue. Its explanation contains a determinism misconception, but B misses the required change entirely and adds an unnecessary property ba...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-baseline (gpt-5.6-luna)

Why: Net win +33.3% (4W/0T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 4W/0T/2L; d=6; p=0.344; net +33.3%; 1 dormancy excluded

Overfit: Moderate (score 0.24)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/0T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline a non-MSBuild build performance request	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Leave an already-optimized build unchanged	Eligible	-100.0%	-40.0%	0/0/1
▼ Route a broken no-op rebuild away from generic optimization	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Leave an already-optimized build unchanged: Both responses failed the core task, which was to recognize the config was already optimized and make no changes. Both invented improvements. However, B committed the exact invented-problem the rubric warns against (adding UseArtifactsOutput to a two-project solution), which i...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (claude-sonnet-4.6)

Why: Net win +0.0% (3W/1T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.656), mean preference -5.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/1T/3L; d=6; p=0.656; net +0.0%; 1 dormancy excluded

Overfit: Moderate (score 0.49)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/1T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Diagnose a Copy task dominating build time	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose a pathological ResolveAssemblyReference time	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose a single custom target dominating one project's build	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose evaluation overhead before any target runs	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Diagnose a Copy task dominating build time: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (gpt-5.6-luna)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded

Overfit: Low (score 0.17)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Diagnose a pathological ResolveAssemblyReference time	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose a single custom target dominating one project's build	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose evaluation overhead before any target runs	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose per-project overhead across many small projects	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a pathological ResolveAssemblyReference time: Both responses arrive at the identical correct diagnosis (RAR ~18.9s) and the identical correct fix (ReferenceOutputAssembly="false"), and both verify with a successful build. They tie on all four rubric criteria, each missing the DisableTransitiveProjectReferences and Packa...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — check-bin-obj-clash (claude-sonnet-4.6)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +20.0% across 7 paired run(s) — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%

Warnings: Activation: isolated 6/7; plugin 5/7

Overfit: Moderate (score 0.20)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Audit the mixed solution and separate safe projects from unsafe ones	Eligible	+0.0%	+0.0%	0/1/0
▲ Avoid a false clash report when projects share only the top-level artifacts root	Eligible	+100.0%	+100.0%	1/0/0
= Diagnose multi-targeting outputs that collapse into one path	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose redundant project reference metadata that forks a same-path build	Eligible	-100.0%	-40.0%	0/0/1
= Fix all clash mechanisms in the mixed solution	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Audit the mixed solution and separate safe projects from unsafe ones: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — check-bin-obj-clash (gpt-5.6-luna)

Why: Net win -14.3% (1W/4T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -5.7% across 7 paired run(s) — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%

Overfit: Low (score 0.10)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/4T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Audit the mixed solution and separate safe projects from unsafe ones	Eligible	+0.0%	+0.0%	0/1/0
= Avoid a false clash report when projects share only the top-level artifacts root	Eligible	+0.0%	+0.0%	0/1/0
▼ Decline output-clash remediation for separate projects using the default SDK layout	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose redundant project reference metadata that forks a same-path build	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose shared output and intermediate path collision	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix all clash mechanisms in the mixed solution	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Audit the mixed solution and separate safe projects from unsafe ones: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — directory-build-organization (claude-sonnet-4.6)

Why: Net win +16.7% (2W/3T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded

Warnings: Activation: isolated 4/6; plugin 4/6

Overfit: Moderate (score 0.23)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline adding shared build files to a lone project	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that silently skips in props	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose a package downgrade chain and reorganize version management	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose an inner shared-props file that overwrites its own override	Eligible	+0.0%	+0.0%	0/1/0
= Preserve an intentional project-specific exception while centralizing	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a TargetFramework condition that silently skips in props: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — directory-build-organization (gpt-5.6-luna)

Why: Net win +50.0% (3W/3T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +17.1% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded

Warnings: Activation: isolated 5/6; plugin 6/6

Overfit: Low (score 0.07)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline adding shared build files to a lone project	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that silently skips in props	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a package downgrade chain and reorganize version management	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose an inner shared-props file that overwrites its own override	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a TargetFramework condition that silently skips in props: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — eval-performance (claude-sonnet-4.6)

Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +25.0% across 8 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%

Warnings: Activation: isolated 8/8; plugin 7/8

Overfit: Moderate (score 0.37)

Repeated-run reliability (not used by the gate): 8 paired runs (6W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Detect a project evaluated twice under different global properties	Eligible	-100.0%	-100.0%	0/0/1
= Recognize TreatAsLocalProperty overuse versus one justified entry	Eligible	+0.0%	+0.0%	0/1/0
▲ Triage which of two property functions actually costs evaluation time	Eligible	+100.0%	+40.0%	1/0/0

Illustrative judge evidence:

Detect a project evaluated twice under different global properties: A reaches the required binlog-backed diagnosis and identifies the exact Build.proj calls. Some peripheral claims and recommendations are overbroad (for example, evaluation itself does not run restore or RAR), but they do not overturn the central correct finding. B's final answ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — eval-performance (gpt-5.6-luna)

Why: Net win +37.5% (3W/5T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +30.0% across 8 paired run(s) — not credible — 5 of 8 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 3W/5T/0L; d=3; p=0.125; net +37.5%

Overfit: Low (score 0.10)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Detect a project evaluated twice under different global properties	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose stacked evaluation-time patterns (deep imports, broad glob, file-I/O property function)	Eligible	+0.0%	+0.0%	0/1/0
= Recognize TreatAsLocalProperty overuse versus one justified entry	Eligible	+0.0%	+0.0%	0/1/0
= Redirect a compile-time slowdown mistakenly framed as an evaluation problem	Eligible	+0.0%	+0.0%	0/1/0
= Triage which of two property functions actually costs evaluation time	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Detect a project evaluated twice under different global properties: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — extension-points (claude-sonnet-4.6)

Why: Net win -14.3% (1W/4T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -25.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded

Warnings: Activation: isolated 4/7; plugin 5/7

Overfit: Moderate (score 0.39)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/4T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Create extensibility hooks for a custom SDK target file	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose NuGet package and repo extension conflicts	Eligible	+0.0%	+0.0%	0/1/0
▲ Diagnose a broken per-TFM forwarder	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose a package ID and file-name mismatch	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose build extension point failures	Eligible	-100.0%	-100.0%	0/0/1
= Fix extension point anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
▼ Non-activation: repair an incremental custom target	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Review packed layout without a false missing-file bug	Eligible	-100.0%	-100.0%	0/0/1

Illustrative judge evidence:

Create extensibility hooks for a custom SDK target file: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — extension-points (gpt-5.6-luna)

Why: Net win +42.9% (3W/4T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +15.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%; 1 dormancy excluded

Overfit: Moderate (score 0.25)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Create extensibility hooks for a custom SDK target file	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose NuGet package and repo extension conflicts	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a package ID and file-name mismatch	Eligible	+0.0%	+0.0%	0/1/0
= Non-activation: repair an incremental custom target	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Review packed layout without a false missing-file bug	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Create extensibility hooks for a custom SDK target file: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — including-generated-files (gpt-5.6-luna)

Why: Net win +14.3% (3W/2T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.500 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded

Warnings: Activation: isolated 6/7; plugin 6/7

Overfit: Low (score 0.17)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline manual wiring for Roslyn source generators	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Diagnose hardcoded obj path for generated source	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose missing clean tracking for generated source	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose project-level glob for generated source	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose wrong hook for generated source files	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose hardcoded obj path for generated source: Both responses correctly diagnose the root cause (hardcoded obj path diverging from redirected IntermediateOutputPath) and recommend $(IntermediateOutputPath). They are nearly equivalent in quality. Response A is marginally more thorough by explicitly noting both Compile and F...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — item-management (gpt-5.6-luna)

Why: Net win +16.7% (2W/3T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a target ordering problem with no item-group defect	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Diagnose real and claimed item problems in a code generation pipeline	Eligible	-100.0%	-40.0%	0/0/1
= Fix item management anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
= Leave correct single-list batching unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose real and claimed item problems in a code generation pipeline: Both responses achieved the core outcome: six complete validation pairs, no duplicate-source warning, FileWrites moved to generation, custom clean removed, and standard Clean removing generated sources. Both verified via build/build/clean. Response A provides stronger verifica...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-antipatterns (claude-sonnet-4.6)

Why: Net win +28.6% (2W/5T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 7 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 2W/5T/0L; d=2; p=0.250; net +28.6%; 1 dormancy excluded

Warnings: Activation: isolated 2/7; plugin 2/7

Overfit: Moderate (score 0.35)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/6T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a module to an F# project	Eligible	+0.0%	+0.0%	0/1/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
= Distinguish a style backslash from a real cross-platform backslash bug	Eligible	+0.0%	+0.0%	0/1/0
= Fix broken file order causing FS0039	Eligible	+0.0%	+0.0%	0/1/0
= Judge an unguarded import inside a NuGet package build folder	Eligible	+0.0%	+0.0%	0/1/0
= Non-activation: migrate a legacy project to SDK style	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Add a module to an F# project: Both deliver the requested validation module, correct F# compile ordering, validation gate in Program.fs, and successful build-through-run evidence. No material final-result difference is shown.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-antipatterns (gpt-5.6-luna)

Why: Net win +0.0% (1W/5T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.750), mean preference -5.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Warnings: Activation: isolated 6/7; plugin 6/7

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/5T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a module to an F# project	Eligible	+0.0%	+0.0%	0/1/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
= Distinguish a style backslash from a real cross-platform backslash bug	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix broken file order causing FS0039	Eligible	-100.0%	-40.0%	0/0/1
= Judge an unguarded import inside a NuGet package build folder	Eligible	+0.0%	+0.0%	0/1/0
= Leave a clean project without inventing anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
▼ Non-activation: migrate a legacy project to SDK style	Excluded (activation contract)	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Add a module to an F# project: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-modernization (claude-sonnet-4.6)

Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded

Overfit: Moderate (score 0.35)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Consolidate duplicated projects into a multi-targeting SDK-style project	Eligible	+0.0%	+0.0%	0/1/0
▼ Decline modernizing a non-.NET build	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
▼ Modernize a single project without introducing Central Package Management	Eligible	-100.0%	-40.0%	0/0/1
= Recognize an already-modern project needs no migration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Consolidate duplicated projects into a multi-targeting SDK-style project: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-modernization (gpt-5.6-luna)

Why: Net win +50.0% (3W/3T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +22.9% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded

Overfit: Low (score 0.05)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Identify legacy patterns for SDK-style migration	Eligible	+0.0%	+0.0%	0/1/0
= Modernize a single project without introducing Central Package Management	Eligible	+0.0%	+0.0%	0/1/0
= Recognize an already-modern project needs no migration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Identify legacy patterns for SDK-style migration: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-server (claude-sonnet-4.6)

Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +40.0% across 8 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%

Warnings: Activation: isolated 8/8; plugin 7/8

Overfit: Moderate (score 0.50)

Repeated-run reliability (not used by the gate): 8 paired runs (6W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline MSBuild Server for a single one-off release build	Eligible	+0.0%	+0.0%	0/1/0
▼ Explain a background MSBuild Server process consuming memory	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Decline MSBuild Server for a single one-off release build: Both responses are concise, correct, directly answer no, and give the same essential rationale without irrelevant caveats or confusion.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-server (gpt-5.6-luna)

Why: Net win +50.0% (4W/4T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +35.0% across 8 paired run(s) — not credible — 4 of 8 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 4W/4T/0L; d=4; p=0.063; net +50.0%

Overfit: Low (score 0.18)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Confirm the MSBuild Server is actually improving build times before declaring success	Eligible	+0.0%	+0.0%	0/1/0
= Decline MSBuild Server for a single one-off release build	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose stale build output after enabling MSBuild Server	Eligible	+0.0%	+0.0%	0/1/0
= Explain a background MSBuild Server process consuming memory	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Confirm the MSBuild Server is actually improving build times before declaring success: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

Details for 7 results were omitted to keep this comment under GitHub's limit. Open the workflow summary or Full Results for the complete breakdown.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1118 in dotnet/skills, download eval artifacts with gh run download 34296652785 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/532f862a7cac6f521a7cc0b621c5379fd6e81bf9/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

Remove ambiguity from the four off-target prompts while preserving expect_activation false, and restore the prior routing descriptions after positive-only metadata regressed both model families. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d0a0a20-fd83-4343-8434-4763f8120503

github-actions · 2026-09-09T03:06:43Z

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 0567225fcdc1476ca03551702f59a44bde46a64d to retry this exact commit.

36 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

Retry transient warm-up failures, narrow the two remaining routing descriptions to concrete inputs, and reclassify property placement after repeated cross-family activation and quality evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d0a0a20-fd83-4343-8434-4763f8120503

github-actions · 2026-09-09T04:11:10Z

📊 Skill Evaluation Results

36 model/skill results across 18 skills and 2 models — ✅ 6 improved, ➖ 27 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 3 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit e27a3bffc8aabc847288dc1fcfc683e44b48e7be; 2 judge models.

Measurement health: 36 expected / 36 observed / 36 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
binlog-failure-analysis	claude-sonnet-4.6	➖ Not proven improved	n=7; 4W/0T/3L; d=7; p=0.500; net +14.3%; 1 dormancy excluded	🟡 0.26	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
binlog-failure-analysis	gpt-5.6-luna	✅ Improved	n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 1 dormancy excluded	✅ 0.11	—	None.
binlog-generation	claude-sonnet-4.6	✅ Improved	n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%; 1 dormancy excluded	🟡 0.50	—	Review overfit evidence.
binlog-generation	gpt-5.6-luna	✅ Improved	n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 1 dormancy excluded	🟡 0.26	—	Review overfit evidence.
build-parallelism	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 1W/2T/3L; d=4; p=0.312; net -33.3%; 1 dormancy excluded	🟡 0.35	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
build-parallelism	gpt-5.6-luna	➖ Not proven improved	n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded	✅ 0.12	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
build-perf-baseline	claude-sonnet-4.6	➖ Not proven improved	n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded	🟡 0.28	Activation: isolated 5/6; plugin 5/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
build-perf-baseline	gpt-5.6-luna	➖ Not proven improved	n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 1 dormancy excluded	🟡 0.23	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-diagnostics	claude-sonnet-4.6	➖ Not proven improved	n=7; 1W/2T/4L; d=5; p=0.188; net -42.9%; 1 dormancy excluded	🟡 0.42	Activation: isolated 7/7; plugin 6/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-diagnostics	gpt-5.6-luna	➖ Not proven improved	n=7; 2W/5T/0L; d=2; p=0.250; net +28.6%; 1 dormancy excluded	✅ 0.17	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
check-bin-obj-clash	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/1T/3L; d=6; p=0.656; net +0.0%	✅ 0.18	Activation: isolated 6/7; plugin 6/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
check-bin-obj-clash	gpt-5.6-luna	➖ Not proven improved	n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%	✅ 0.11	Activation: isolated 7/7; plugin 6/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	claude-sonnet-4.6	➖ Not proven improved	n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded	🟡 0.32	Activation: isolated 4/6; plugin 4/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.08	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
eval-performance	claude-sonnet-4.6	✅ Improved	n=8; 7W/1T/0L; d=7; p=0.008; net +87.5%	🟡 0.28	Activation: isolated 8/8; plugin 7/8	Fix activation gaps; Review overfit evidence.
eval-performance	gpt-5.6-luna	➖ Not proven improved	n=8; 4W/2T/2L; d=6; p=0.344; net +25.0%	✅ 0.09	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
extension-points	claude-sonnet-4.6	➖ Not proven improved	n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	🟡 0.26	Activation: isolated 4/7; plugin 4/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
extension-points	gpt-5.6-luna	➖ Not proven improved	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded	🟡 0.26	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
including-generated-files	claude-sonnet-4.6	⛔ Activation contract failed	n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded	🟡 0.40	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 3/7	Narrow skill routing so the listed off-target scenarios stay dormant.
including-generated-files	gpt-5.6-luna	➖ Not proven improved	n=7; 1W/6T/0L; d=1; p=0.500; net +14.3%; 1 dormancy excluded	✅ 0.16	Activation: isolated 7/7; plugin 6/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
incremental-build	claude-sonnet-4.6	✅ Improved	n=9; 5W/4T/0L; d=5; p=0.031; net +55.6%	🟡 0.28	Activation: isolated 9/9; plugin 6/9	Fix activation gaps; Review overfit evidence.
incremental-build	gpt-5.6-luna	➖ Not proven improved	n=9; 4W/4T/1L; d=5; p=0.188; net +33.3%	✅ 0.09	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
item-management	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 1 dormancy excluded	🟡 0.34	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 2/6	Narrow skill routing so the listed off-target scenarios stay dormant.
item-management	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	✅ 0.11	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded	🟡 0.29	Activation: isolated 2/7; plugin 2/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	gpt-5.6-luna	➖ Not proven improved	n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded	✅ 0.09	Activation: isolated 6/7; plugin 4/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-modernization	claude-sonnet-4.6	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	🟡 0.29	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-modernization	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.08	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-server	claude-sonnet-4.6	➖ Not proven improved	n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%	🔴 0.53	Activation: isolated 8/8; plugin 7/8	Inspect tied or lost stimuli and fix inconsistent skill behavior.
msbuild-server	gpt-5.6-luna	✅ Improved	n=8; 6W/2T/0L; d=6; p=0.016; net +75.0%	🟡 0.20	—	Review overfit evidence.
property-patterns	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%	🟡 0.37	Activation: isolated 6/7; plugin 6/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
property-patterns	gpt-5.6-luna	➖ Not proven improved	n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%	✅ 0.09	Activation: isolated 6/7; plugin 6/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
resolve-project-references	claude-sonnet-4.6	➖ Not proven improved	n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 2 dormancy excluded	🟡 0.38	Activation: isolated 6/6; plugin 5/6	Inspect tied or lost stimuli and fix inconsistent skill behavior.
resolve-project-references	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/2T/3L; d=4; p=0.312; net -33.3%; 2 dormancy excluded	✅ 0.15	Activation: isolated 4/6; plugin 6/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
target-authoring	claude-sonnet-4.6	➖ Not proven improved	n=6; 1W/3T/2L; d=3; p=0.500; net -16.7%; 1 dormancy excluded	🟡 0.35	Activation: isolated 6/6; plugin 5/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
target-authoring	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	✅ 0.11	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

ℹ️ How to read this report

✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
⛔ Activation contract failed — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — build-parallelism (claude-sonnet-4.6)

Why: Net win -33.3% (1W/2T/3L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference -5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 1W/2T/3L; d=4; p=0.312; net -33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6

Overfit: Moderate (score 0.35)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/2T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Analyze build parallelism bottlenecks	Eligible	-100.0%	-40.0%	0/0/1
▼ Decline graph build for runtime-discovered projects	Eligible	-100.0%	-40.0%	0/0/1
= Enable BuildInParallel on a custom MSBuild task	Eligible	+0.0%	+0.0%	0/1/0
= Enable parallel nodes for a wide project graph	Eligible	+0.0%	+0.0%	0/1/0
▲ Non-activation: make one custom target incremental	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
▼ Reduce CI build scope with a solution filter	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Analyze build parallelism bottlenecks: Both answer the substantive questions correctly and B validates removal by rebuilding. A is slightly stronger overall because its binlog evidence is more concrete and its conclusion about the redundant edge is more precise, whereas B adds speculative/overstated scheduler and o...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — including-generated-files (claude-sonnet-4.6)

Why: Net win -14.3% (1W/4T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 3/7

Overfit: Moderate (score 0.40)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/4T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline manual wiring for Roslyn source generators	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Diagnose hardcoded obj path for generated source	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose missing clean tracking for generated source	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose missing output registration for generated non-code file	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose project-level glob for generated source	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose wrong hook for generated source files	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix generated source inclusion and clean tracking	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Diagnose hardcoded obj path for generated source: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — item-management (claude-sonnet-4.6)

Why: Net win +33.3% (2W/4T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +11.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/6; plugin 2/6

Overfit: Moderate (score 0.34)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a target ordering problem with no item-group defect	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose an ineffective Compile Remove that does not match the glob	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose item group and batching issues	Eligible	+0.0%	+0.0%	0/1/0
▲ Diagnose real and claimed item problems in a code generation pipeline	Eligible	+100.0%	+40.0%	1/0/0
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
= Leave correct single-list batching unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose an ineffective Compile Remove that does not match the glob: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — binlog-failure-analysis (claude-sonnet-4.6)

Why: Net win +14.3% (4W/0T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +12.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.500 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/0T/3L; d=7; p=0.500; net +14.3%; 1 dormancy excluded

Overfit: Moderate (score 0.26)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/1T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Determine whether a quiet second build actually failed	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose build failures from binlog only (no source files)	Eligible	-100.0%	-40.0%	0/0/1
= Stay dormant for a non-MSBuild build failure log	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Trace why a generated source file is missing at compile time	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Determine whether a quiet second build actually failed: Both answers are correct, concise, and well-supported. A is marginally stronger because it supplies more of the concrete skipped-target/up-to-date evidence requested, while B's conclusion and explanation are otherwise equally sound.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-parallelism (gpt-5.6-luna)

Why: Net win +50.0% (3W/3T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +22.9% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded

Overfit: Low (score 0.12)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline graph build for runtime-discovered projects	Eligible	+0.0%	+0.0%	0/1/0
= Enable BuildInParallel on a custom MSBuild task	Eligible	+0.0%	+0.0%	0/1/0
= Reduce CI build scope with a solution filter	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Decline graph build for runtime-discovered projects: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-baseline (claude-sonnet-4.6)

Why: Net win +66.7% (4W/2T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +25.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded

Warnings: Activation: isolated 5/6; plugin 5/6

Overfit: Moderate (score 0.28)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Configure deterministic, cache-safe CI builds	Eligible	+0.0%	+0.0%	0/1/0
▼ Decline a non-MSBuild build performance request	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Leave an already-optimized build unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Configure deterministic, cache-safe CI builds: Both make the same fundamental incorrect change: explicitly setting an already-default Deterministic property while omitting the required CI-only ContinuousIntegrationBuild configuration. Their final states are comparably buildable but do not satisfy the task.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-baseline (gpt-5.6-luna)

Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 1 dormancy excluded

Overfit: Moderate (score 0.23)

Repeated-run reliability (not used by the gate): 7 paired runs (5W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Leave an already-optimized build unchanged	Eligible	-100.0%	-100.0%	0/0/1
= Route a broken no-op rebuild away from generic optimization	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Leave an already-optimized build unchanged: The ideal answer was to determine no checklist anti-pattern was present and make no changes. Neither response achieved that ideal, but B is clearly worse: it invented UseArtifactsOutput (the specific fabricated recommendation warned against in the rubric), removed TargetFramew...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (claude-sonnet-4.6)

Why: Net win -42.9% (1W/2T/4L over 7 preference-eligible stimulus vote(s), sign test p=0.188), mean preference -17.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/2T/4L; d=5; p=0.188; net -42.9%; 1 dormancy excluded

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Moderate (score 0.42)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/2T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose NuGet restore running redundantly across CI stages	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose a pathological ResolveAssemblyReference time	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose a single custom target dominating one project's build	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose evaluation overhead before any target runs	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose per-project overhead across many small projects	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose slow build for a small project	Eligible	-100.0%	-100.0%	0/0/1

Illustrative judge evidence:

Diagnose NuGet restore running redundantly across CI stages: The responses are substantively equivalent and fully satisfy all rubric requirements. Each includes minor overstatements about the precise mechanism/effect of static graph restore, but neither has a meaningful quality advantage.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (gpt-5.6-luna)

Why: Net win +28.6% (2W/5T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 7 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 2W/5T/0L; d=2; p=0.250; net +28.6%; 1 dormancy excluded

Overfit: Low (score 0.17)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/6T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose NuGet restore running redundantly across CI stages	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a pathological ResolveAssemblyReference time	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose evaluation overhead before any target runs	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose per-project overhead across many small projects	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose slow build for a small project	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose NuGet restore running redundantly across CI stages: Both responses reach essentially the same correct conclusions and satisfy all three rubric criteria with nearly identical recommendations. B provides a slightly more precise quantification (89%) and cleaner phrasing, while A also notes the cross-agent caching concern. The diff...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — check-bin-obj-clash (claude-sonnet-4.6)

Why: Net win +0.0% (3W/1T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.656), mean preference -8.6% across 7 paired run(s) — no improvement

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/1T/3L; d=6; p=0.656; net +0.0%

Warnings: Activation: isolated 6/7; plugin 6/7

Overfit: Low (score 0.18)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/1T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Audit the mixed solution and separate safe projects from unsafe ones	Eligible	-100.0%	-40.0%	0/0/1
▼ Avoid a false clash report when projects share only the top-level artifacts root	Eligible	-100.0%	-100.0%	0/0/1
▼ Diagnose multi-targeting outputs that collapse into one path	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose shared output and intermediate path collision	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Audit the mixed solution and separate safe projects from unsafe ones: Both are substantively correct and satisfy the core audit. A is marginally clearer and more complete in its project-by-project summary, especially its explicit nuance that ConsumerApp's outputs are safe while its reference configuration is not.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — check-bin-obj-clash (gpt-5.6-luna)

Why: Net win +0.0% (1W/5T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 7 paired run(s) — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Avoid a false clash report when projects share only the top-level artifacts root	Eligible	+0.0%	+0.0%	0/1/0
= Decline output-clash remediation for separate projects using the default SDK layout	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose multi-targeting outputs that collapse into one path	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose redundant project reference metadata that forks a same-path build	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose shared output and intermediate path collision	Eligible	-100.0%	-40.0%	0/0/1
= Fix all clash mechanisms in the mixed solution	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Avoid a false clash report when projects share only the top-level artifacts root: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — directory-build-organization (claude-sonnet-4.6)

Why: Net win +50.0% (3W/3T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +25.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded

Warnings: Activation: isolated 4/6; plugin 4/6

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline adding shared build files to a lone project	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▲ Diagnose a TargetFramework condition that silently skips in props	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose a package downgrade chain and reorganize version management	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose an inner shared-props file that overwrites its own override	Eligible	+0.0%	+0.0%	0/1/0
= Preserve an intentional project-specific exception while centralizing	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a package downgrade chain and reorganize version management: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — directory-build-organization (gpt-5.6-luna)

Why: Net win +16.7% (1W/5T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 6 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 1 dormancy excluded

Overfit: Low (score 0.08)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/6T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Apply repo-level build organization cleanup	Eligible	+0.0%	+0.0%	0/1/0
= Decline adding shared build files to a lone project	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that silently skips in props	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a package downgrade chain and reorganize version management	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose an inner shared-props file that overwrites its own override	Eligible	+0.0%	+0.0%	0/1/0
= Preserve an intentional project-specific exception while centralizing	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Apply repo-level build organization cleanup: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — eval-performance (gpt-5.6-luna)

Why: Net win +25.0% (4W/2T/2L over 8 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +25.0% across 8 paired run(s) — not credible (sign test p=0.344 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 4W/2T/2L; d=6; p=0.344; net +25.0%

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Detect a project evaluated twice under different global properties	Eligible	-100.0%	-40.0%	0/0/1
= Recognize TreatAsLocalProperty overuse versus one justified entry	Eligible	+0.0%	+0.0%	0/1/0
▼ Redirect a compile-time slowdown mistakenly framed as an evaluation problem	Eligible	-100.0%	-40.0%	0/0/1
= Triage which of two property functions actually costs evaluation time	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Detect a project evaluated twice under different global properties: Both responses correctly parsed the binlog, identified the double evaluation of Shared.csproj with MyFlavor=Alpha/Beta, and avoided misattributing the cost to globs/imports/property functions. A gives a slightly clearer causal explanation of the global-property mechanism (sepa...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — extension-points (claude-sonnet-4.6)

Why: Net win +0.0% (1W/5T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.750), mean preference -7.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Warnings: Activation: isolated 4/7; plugin 4/7

Overfit: Moderate (score 0.26)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Create extensibility hooks for a custom SDK target file	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose NuGet package and repo extension conflicts	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a broken per-TFM forwarder	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a package ID and file-name mismatch	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix extension point anti-patterns	Eligible	-100.0%	-100.0%	0/0/1
= Non-activation: repair an incremental custom target	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Review packed layout without a false missing-file bug	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Create extensibility hooks for a custom SDK target file: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — extension-points (gpt-5.6-luna)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +22.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded

Overfit: Moderate (score 0.26)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Diagnose NuGet package and repo extension conflicts	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose a package ID and file-name mismatch	Eligible	+0.0%	+0.0%	0/1/0
= Fix extension point anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
= Review packed layout without a false missing-file bug	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose NuGet package and repo extension conflicts: Both responses correctly diagnosed and fixed the same core issues (inverted Common.props condition, late CustomBeforeMicrosoftCommonTargets assignment, unguarded optional imports, and GenerateAssemblyInfo collision) with very similar approaches and quality. The key differentia...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — including-generated-files (gpt-5.6-luna)

Why: Net win +14.3% (1W/6T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 6 of 7 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/6T/0L; d=1; p=0.500; net +14.3%; 1 dormancy excluded

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Low (score 0.16)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/6T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose hardcoded obj path for generated source	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose missing clean tracking for generated source	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose missing generated source inclusion	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose missing output registration for generated non-code file	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose project-level glob for generated source	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose wrong hook for generated source files	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose hardcoded obj path for generated source: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — incremental-build (gpt-5.6-luna)

Why: Net win +33.3% (4W/4T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +20.0% across 9 paired run(s) — not credible (sign test p=0.188 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 4W/4T/1L; d=5; p=0.188; net +33.3%

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 9 paired runs (4W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Diagnose custom targets that always rerun	Eligible	-100.0%	-40.0%	0/0/1
= Explain slower builds when MSBuild skipped everything	Eligible	+0.0%	+0.0%	0/1/0
= Fix broken incremental targets and clean tracking	Eligible	+0.0%	+0.0%	0/1/0
= Identify a volatile output path that defeats incrementality	Eligible	+0.0%	+0.0%	0/1/0
= Read a diagnostic log to find the stale input	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose custom targets that always rerun: Both reach the correct diagnosis and identify the same targets and missing metadata. A is slightly better by providing concrete, ready-to-apply XML with specific input/output paths, while B stays more descriptive (though B adds a useful nuance about FileWrites). The gap is small.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — item-management (gpt-5.6-luna)

Why: Net win +0.0% (1W/4T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +2.9% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/4T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline a target ordering problem with no item-group defect	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Diagnose an ineffective Compile Remove that does not match the glob	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose item group and batching issues	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix item management anti-patterns	Eligible	-100.0%	-40.0%	0/0/1
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
= Leave correct single-list batching unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose an ineffective Compile Remove that does not match the glob: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-antipatterns (claude-sonnet-4.6)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +5.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%; 1 dormancy excluded

Warnings: Activation: isolated 2/7; plugin 2/7

Overfit: Moderate (score 0.29)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a module to an F# project	Eligible	+0.0%	+0.0%	0/1/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
▲ Distinguish a style backslash from a real cross-platform backslash bug	Eligible	+100.0%	+40.0%	1/0/0
▲ Fix broken file order causing FS0039	Eligible	+100.0%	+40.0%	1/0/0
▼ Judge an unguarded import inside a NuGet package build folder	Eligible	-100.0%	-40.0%	0/0/1
= Leave a clean project without inventing anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
▼ Non-activation: migrate a legacy project to SDK style	Excluded (activation contract)	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Add a module to an F# project: The implementations are substantively identical, satisfy the requested validation and integration behavior, and both successfully run the resulting project. Neither has a meaningful quality advantage.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-antipatterns (gpt-5.6-luna)

Why: Net win -14.3% (1W/4T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +7.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded

Warnings: Activation: isolated 6/7; plugin 4/7

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/4T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Add a module to an F# project	Eligible	+0.0%	+0.0%	0/1/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
= Distinguish a style backslash from a real cross-platform backslash bug	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix broken file order causing FS0039	Eligible	-100.0%	-40.0%	0/0/1
▼ Judge an unguarded import inside a NuGet package build folder	Eligible	-100.0%	-40.0%	0/0/1
= Review MSBuild files for anti-patterns and style issues	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Add a module to an F# project: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-modernization (claude-sonnet-4.6)

Why: Net win +16.7% (2W/3T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +28.6% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded

Overfit: Moderate (score 0.29)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Apply the migration to SDK-style	Eligible	+0.0%	+0.0%	0/1/0
= Consolidate duplicated projects into a multi-targeting SDK-style project	Eligible	+0.0%	+0.0%	0/1/0
▼ Modernize a single project without introducing Central Package Management	Eligible	-100.0%	-40.0%	0/0/1
= Recognize an already-modern project needs no migration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Apply the migration to SDK-style: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-modernization (gpt-5.6-luna)

Why: Net win +16.7% (2W/3T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +0.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/3T/1L; d=3; p=0.500; net +16.7%; 1 dormancy excluded

Overfit: Low (score 0.08)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Apply the migration to SDK-style	Eligible	-100.0%	-40.0%	0/0/1
= Consolidate duplicated projects into a multi-targeting SDK-style project	Eligible	+0.0%	+0.0%	0/1/0
▼ Decline modernizing a non-.NET build	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Modernize a single project without introducing Central Package Management	Eligible	+0.0%	+0.0%	0/1/0
= Recognize an already-modern project needs no migration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Apply the migration to SDK-style: The task explicitly required the project to be 'buildable after the change.' Response A recognized the missing net472 reference assemblies and added the Microsoft.NETFramework.ReferenceAssemblies package, achieving an actual successful build ('Build succeeded', produces Legacy...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-server (claude-sonnet-4.6)

Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +55.0% across 8 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%

Warnings: Activation: isolated 8/8; plugin 7/8

Overfit: High (score 0.53)

Repeated-run reliability (not used by the gate): 8 paired runs (6W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Confirm the MSBuild Server is actually improving build times before declaring success	Eligible	-100.0%	-40.0%	0/0/1
= Decline MSBuild Server for a single one-off release build	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Confirm the MSBuild Server is actually improving build times before declaring success: A satisfies every requested check and gives the essential diagnostic interpretation for no speedup. B covers the main timing workflow but omits that diagnostic and additionally suggests dotnet build-server status, which is not a standard supported subcommand and can mislead ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — property-patterns (claude-sonnet-4.6)

Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +20.0% across 7 paired run(s) — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%

Warnings: Activation: isolated 6/7; plugin 6/7

Overfit: Moderate (score 0.37)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Choose the correct shared build file for a post-build target	Eligible	+0.0%	+0.0%	0/1/0
▼ Choose the right OS detection for a cross-platform property	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose multi-level property hierarchy bugs	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose shared build property issues	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Choose the correct shared build file for a post-build target: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — property-patterns (gpt-5.6-luna)

Why: Net win +14.3% (3W/2T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.7% across 7 paired run(s) — not credible (sign test p=0.500 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%

Warnings: Activation: isolated 6/7; plugin 6/7

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Choose the correct shared build file for a post-build target	Eligible	+0.0%	+0.0%	0/1/0
▲ Choose the right OS detection for a cross-platform property	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose a TargetFramework condition that never applies in props	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose multi-level property hierarchy bugs	Eligible	-100.0%	-40.0%	0/0/1
▼ Fix shared property configuration	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Choose the correct shared build file for a post-build target: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — resolve-project-references (claude-sonnet-4.6)

Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +27.5% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 4W/1T/1L; d=5; p=0.188; net +50.0%; 2 dormancy excluded

Warnings: Activation: isolated 6/6; plugin 5/6

Overfit: Moderate (score 0.38)

Repeated-run reliability (not used by the gate): 8 paired runs (5W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Rank Copy ahead of Csc when task self-time is higher	Eligible	-100.0%	-40.0%	0/0/1
= Redirect from misleading target summary to Csc self-time	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant on a general build performance review	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Rank Copy ahead of Csc when task self-time is higher: Both reach the required conclusion with the key report figures. A is modestly stronger because its explanation of nested child-build work and its hardlink recommendation are more precise, while B includes distracting and questionable implementation detail.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — resolve-project-references (gpt-5.6-luna)

Why: Net win -33.3% (1W/2T/3L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference -10.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/2T/3L; d=4; p=0.312; net -33.3%; 2 dormancy excluded

Warnings: Activation: isolated 4/6; plugin 6/6

Overfit: Low (score 0.15)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/4T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Distinguish wait time from a real serial dependency chain	Eligible	-100.0%	-40.0%	0/0/1
▲ Give the exact command to replay task self-time	Eligible	+100.0%	+40.0%	1/0/0
= Hand off a wide graph blocked by single-node scheduling	Eligible	+0.0%	+0.0%	0/1/0
▼ Rank Copy ahead of Csc when task self-time is higher	Eligible	-100.0%	-40.0%	0/0/1
▼ Redirect from misleading target summary to Csc self-time	Eligible	-100.0%	-40.0%	0/0/1
= Require diagnostic evidence before analyzing project-reference time	Eligible	+0.0%	+0.0%	0/1/0
= Stay dormant on a general build performance review	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Stay dormant when Csc is already the obvious bottleneck	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Distinguish wait time from a real serial dependency chain: Both responses are strong, correct, and avoid the trap of dismissing the delay. A more clearly identifies the serial dependency chain as the structural bottleneck and explains the -m parallelism limitation, plus warns against summing inclusive target times. B has a slightly be...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

Details for 7 results were omitted to keep this comment under GitHub's limit. Open the workflow summary or Full Results for the complete breakdown.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1118 in dotnet/skills, download eval artifacts with gh run download 34306127683 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/e27a3bffc8aabc847288dc1fcfc683e44b48e7be/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

Keep the three off-target contracts while removing negative lexical matches from prompts and requiring concrete positive artifacts in routing metadata. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d0a0a20-fd83-4343-8434-4763f8120503

github-actions · 2026-09-09T05:25:14Z

📊 Skill Evaluation Results

36 model/skill results across 18 skills and 2 models — ✅ 7 improved, ➖ 25 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 4 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 37ceee80d21a676da1e07bc7cd67629d20609070; 2 judge models.

Measurement health: 36 expected / 36 observed / 36 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
binlog-failure-analysis	claude-sonnet-4.6	➖ Not proven improved	n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded	✅ 0.18	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
binlog-failure-analysis	gpt-5.6-luna	✅ Improved	n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%; 1 dormancy excluded	✅ 0.10	—	None.
binlog-generation	claude-sonnet-4.6	✅ Improved	n=7; 7W/0T/0L; d=7; p=0.008; net +100.0%; 1 dormancy excluded	🔴 0.50	Activation: isolated 7/7; plugin 6/7	Fix activation gaps; Review overfit evidence.
binlog-generation	gpt-5.6-luna	✅ Improved	n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 1 dormancy excluded	🟡 0.28	Activation: isolated 7/7; plugin 6/7	Fix activation gaps; Review overfit evidence.
build-parallelism	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded	🟡 0.44	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6	Narrow skill routing so the listed off-target scenarios stay dormant.
build-parallelism	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/2T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded	✅ 0.11	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
build-perf-baseline	claude-sonnet-4.6	➖ Not proven improved	n=6; 3W/1T/2L; d=5; p=0.500; net +16.7%; 1 dormancy excluded	🟡 0.32	Activation: isolated 5/6; plugin 5/6	Inspect tied or lost stimuli and fix inconsistent skill behavior.
build-perf-baseline	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/2T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded	🟡 0.23	Activation: isolated 5/6; plugin 6/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
build-perf-diagnostics	claude-sonnet-4.6	➖ Not proven improved	n=7; 2W/4T/1L; d=3; p=0.500; net +14.3%; 1 dormancy excluded	🟡 0.40	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
build-perf-diagnostics	gpt-5.6-luna	➖ Not proven improved	n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%; 1 dormancy excluded	✅ 0.15	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
check-bin-obj-clash	claude-sonnet-4.6	✅ Improved	n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%	🟡 0.23	Activation: isolated 6/7; plugin 6/7	Fix activation gaps; Review overfit evidence.
check-bin-obj-clash	gpt-5.6-luna	➖ Not proven improved	n=7; 2W/4T/1L; d=3; p=0.500; net +14.3%	✅ 0.10	Activation: isolated 7/7; plugin 6/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	claude-sonnet-4.6	➖ Not proven improved	n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded	🟡 0.37	Activation: isolated 4/6; plugin 4/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
directory-build-organization	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	✅ 0.09	Activation: isolated 5/6; plugin 6/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
eval-performance	claude-sonnet-4.6	➖ Not proven improved	n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%	🟡 0.41	Activation: isolated 8/8; plugin 7/8	Inspect tied or lost stimuli and fix inconsistent skill behavior.
eval-performance	gpt-5.6-luna	➖ Not proven improved	n=8; 3W/4T/1L; d=4; p=0.312; net +25.0%	✅ 0.10	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
extension-points	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded	✅ 0.19	Activation: isolated 4/7; plugin 5/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
extension-points	gpt-5.6-luna	➖ Not proven improved	n=7; 2W/4T/1L; d=3; p=0.500; net +14.3%; 1 dormancy excluded	🟡 0.24	Activation: isolated 7/7; plugin 6/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
including-generated-files	claude-sonnet-4.6	⛔ Activation contract failed	n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded	🟡 0.41	Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 5/7	Narrow skill routing so the listed off-target scenarios stay dormant.
including-generated-files	gpt-5.6-luna	➖ Not proven improved	n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%; 1 dormancy excluded	🟡 0.20	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
incremental-build	claude-sonnet-4.6	➖ Not proven improved	n=9; 4W/4T/1L; d=5; p=0.188; net +33.3%	🟡 0.31	Activation: isolated 7/9; plugin 5/9	Inspect tied or lost stimuli and fix inconsistent skill behavior.
incremental-build	gpt-5.6-luna	➖ Not proven improved	n=9; 4W/5T/0L; d=4; p=0.063; net +44.4%	✅ 0.16	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
item-management	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 1W/3T/2L; d=3; p=0.500; net -16.7%; 1 dormancy excluded	🟡 0.28	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 2/6	Narrow skill routing so the listed off-target scenarios stay dormant.
item-management	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 1 dormancy excluded	✅ 0.11	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	claude-sonnet-4.6	➖ Not proven improved	n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded	🟡 0.24	Activation: isolated 2/7; plugin 2/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-antipatterns	gpt-5.6-luna	➖ Not proven improved	n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded	✅ 0.09	Activation: isolated 6/7; plugin 5/7	Inspect tied or lost stimuli and fix inconsistent skill behavior.
msbuild-modernization	claude-sonnet-4.6	➖ Not proven improved	n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded	✅ 0.19	Activation: isolated 6/6; plugin 5/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-modernization	gpt-5.6-luna	➖ Not proven improved	n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 1 dormancy excluded	✅ 0.06	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
msbuild-server	claude-sonnet-4.6	✅ Improved	n=8; 7W/1T/0L; d=7; p=0.008; net +87.5%	🟡 0.47	Activation: isolated 8/8; plugin 7/8	Fix activation gaps; Review overfit evidence.
msbuild-server	gpt-5.6-luna	✅ Improved	n=8; 6W/2T/0L; d=6; p=0.016; net +75.0%	✅ 0.14	—	None.
property-patterns	claude-sonnet-4.6	➖ Not proven improved	n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%	🟡 0.20	Activation: isolated 7/7; plugin 6/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
property-patterns	gpt-5.6-luna	➖ Not proven improved	n=7; 0W/6T/1L; d=1; p=0.500; net -14.3%	✅ 0.07	Activation: isolated 6/7; plugin 7/7	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
resolve-project-references	claude-sonnet-4.6	✅ Improved	n=6; 5W/1T/0L; d=5; p=0.031; net +83.3%; 2 dormancy excluded	🟡 0.25	—	Review overfit evidence.
resolve-project-references	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 2 dormancy excluded	✅ 0.14	Activation: isolated 4/6; plugin 6/6	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
target-authoring	claude-sonnet-4.6	⛔ Activation contract failed	n=6; 1W/2T/3L; d=4; p=0.312; net -33.3%; 1 dormancy excluded	🟡 0.38	Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 4/6	Narrow skill routing so the listed off-target scenarios stay dormant.
target-authoring	gpt-5.6-luna	➖ Not proven improved	n=6; 1W/3T/2L; d=3; p=0.500; net -16.7%; 1 dormancy excluded	✅ 0.14	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

ℹ️ How to read this report

✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
⛔ Activation contract failed — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — build-parallelism (claude-sonnet-4.6)

Why: Net win +66.7% (4W/2T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +25.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 5/6

Overfit: Moderate (score 0.44)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline graph build for runtime-discovered projects	Eligible	+0.0%	+0.0%	0/1/0
▲ Enable BuildInParallel on a custom MSBuild task	Eligible	+100.0%	+40.0%	1/0/0
= Enable parallel nodes for a wide project graph	Eligible	+0.0%	+0.0%	0/1/0
▼ Non-activation: make one custom target incremental	Excluded (activation contract)	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Decline graph build for runtime-discovered projects: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — including-generated-files (claude-sonnet-4.6)

Why: Net win +14.3% (3W/2T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 5/7; plugin 5/7

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline manual wiring for Roslyn source generators	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose hardcoded obj path for generated source	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose missing clean tracking for generated source	Eligible	-100.0%	-40.0%	0/0/1
▼ Diagnose missing output registration for generated non-code file	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose project-level glob for generated source	Eligible	+0.0%	+0.0%	0/1/0
▲ Diagnose wrong hook for generated source files	Eligible	+100.0%	+40.0%	1/0/0

Illustrative judge evidence:

Diagnose hardcoded obj path for generated source: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — item-management (claude-sonnet-4.6)

Why: Net win -16.7% (1W/3T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 1W/3T/2L; d=3; p=0.500; net -16.7%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 2/6

Overfit: Moderate (score 0.28)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/4T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a target ordering problem with no item-group defect	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▲ Diagnose an ineffective Compile Remove that does not match the glob	Eligible	+100.0%	+40.0%	1/0/0
▼ Diagnose item group and batching issues	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose real and claimed item problems in a code generation pipeline	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix item management anti-patterns	Eligible	-100.0%	-100.0%	0/0/1
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
= Leave correct single-list batching unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose item group and batching issues: Both give correct concise diagnoses for validation batching and generated-file cleanup, but A correctly centers SDK default source globbing. B's explanation and apparent fix for CS2002 omit that fundamental duplication and risk leaving it unresolved.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

⛔ Activation contract failed — target-authoring (claude-sonnet-4.6)

Why: Net win -33.3% (1W/2T/3L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference -14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=6; 1W/2T/3L; d=4; p=0.312; net -33.3%; 1 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s); Activation: isolated 6/6; plugin 4/6

Overfit: Moderate (score 0.38)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/2T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Decline an incremental-build tuning request	Excluded (activation contract)	+100.0%	+40.0%	1/0/0
▼ Diagnose a target hooked to Build that misses direct compile	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose broken SDK target chain across files	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose custom target reliability issues	Eligible	-100.0%	-40.0%	0/0/1
= Fix a query target that uses Outputs instead of Returns	Eligible	+0.0%	+0.0%	0/1/0
▼ Leave an already-correct custom target unchanged	Eligible	-100.0%	-100.0%	0/0/1

Illustrative judge evidence:

Diagnose a target hooked to Build that misses direct compile: A is concise and fully correct. B reaches the same correct primary conclusion, but its unnecessary alternative involving CoreCompileDependsOn and its associated chain description reduce technical precision.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — binlog-failure-analysis (claude-sonnet-4.6)

Why: Net win +57.1% (5W/1T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +22.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded

Overfit: Low (score 0.18)

Repeated-run reliability (not used by the gate): 8 paired runs (5W/1T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Determine whether a quiet second build actually failed	Eligible	-100.0%	-40.0%	0/0/1
= Fall back to command-line log replay when the usual binlog tool is unavailable	Eligible	+0.0%	+0.0%	0/1/0
▼ Stay dormant for a non-MSBuild build failure log	Excluded (activation contract)	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Determine whether a quiet second build actually failed: Both answers are correct and concise, but A more precisely cites and explains the requested up-to-date skip evidence.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-parallelism (gpt-5.6-luna)

Why: Net win +0.0% (2W/2T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.687), mean preference +0.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/2T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/3T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Analyze build parallelism bottlenecks	Eligible	-100.0%	-40.0%	0/0/1
= Decline graph build for runtime-discovered projects	Eligible	+0.0%	+0.0%	0/1/0
= Enable parallel nodes for a wide project graph	Eligible	+0.0%	+0.0%	0/1/0
= Non-activation: make one custom target incremental	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▼ Reduce CI build scope with a solution filter	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Analyze build parallelism bottlenecks: Both responses reached the same correct conclusions across all analytical criteria: the serial dependency chain, the parallelism limitation, the redundant Tests->Api reference, and that removal is hygiene only. They are essentially tied on substance. Response A provides slight...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-baseline (claude-sonnet-4.6)

Why: Net win +16.7% (3W/1T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -17.1% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.500 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/1T/2L; d=5; p=0.500; net +16.7%; 1 dormancy excluded

Warnings: Activation: isolated 5/6; plugin 5/6

Overfit: Moderate (score 0.32)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/1T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Configure deterministic, cache-safe CI builds	Eligible	-100.0%	-40.0%	0/0/1
▼ Decline a non-MSBuild build performance request	Excluded (activation contract)	-100.0%	-100.0%	0/0/1
▼ Leave an already-optimized build unchanged	Eligible	-100.0%	-100.0%	0/0/1
= Route a broken no-op rebuild away from generic optimization	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Configure deterministic, cache-safe CI builds: Both miss the required CI-only ContinuousIntegrationBuild fix and wrongly treat deterministic compilation as absent, so both are substantially inadequate. A is marginally better because it acknowledges the relevant SDK default, even though its resulting change and justificatio...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-baseline (gpt-5.6-luna)

Why: Net win +0.0% (2W/2T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.687), mean preference -5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/2T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded

Warnings: Activation: isolated 5/6; plugin 6/6

Overfit: Moderate (score 0.23)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/2T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline a non-MSBuild build performance request	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Establish build performance baseline and recommend optimizations	Eligible	+0.0%	+0.0%	0/1/0
▼ Leave an already-optimized build unchanged	Eligible	-100.0%	-40.0%	0/0/1
▼ Route a broken no-op rebuild away from generic optimization	Eligible	-100.0%	-40.0%	0/0/1
= Route a restore-bound cold build away from architecture changes	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Establish build performance baseline and recommend optimizations: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (claude-sonnet-4.6)

Why: Net win +14.3% (2W/4T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 2W/4T/1L; d=3; p=0.500; net +14.3%; 1 dormancy excluded

Overfit: Moderate (score 0.40)

Repeated-run reliability (not used by the gate): 8 paired runs (2W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a Copy task dominating build time	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a pathological ResolveAssemblyReference time	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a single custom target dominating one project's build	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose evaluation overhead before any target runs	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose per-project overhead across many small projects	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a Copy task dominating build time: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — build-perf-diagnostics (gpt-5.6-luna)

Why: Net win +42.9% (3W/4T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +15.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%; 1 dormancy excluded

Overfit: Low (score 0.15)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline a runtime latency request that is not a build performance issue	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose NuGet restore running redundantly across CI stages	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a pathological ResolveAssemblyReference time	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose evaluation overhead before any target runs	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose per-project overhead across many small projects	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose NuGet restore running redundantly across CI stages: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — check-bin-obj-clash (gpt-5.6-luna)

Why: Net win +14.3% (2W/4T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +5.7% across 7 paired run(s) — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 2W/4T/1L; d=3; p=0.500; net +14.3%

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Low (score 0.10)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Audit the mixed solution and separate safe projects from unsafe ones	Eligible	-100.0%	-40.0%	0/0/1
= Avoid a false clash report when projects share only the top-level artifacts root	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose multi-targeting outputs that collapse into one path	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose redundant project reference metadata that forks a same-path build	Eligible	+0.0%	+0.0%	0/1/0
= Fix all clash mechanisms in the mixed solution	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Audit the mixed solution and separate safe projects from unsafe ones: Both responses correctly identify LibraryA/LibraryB, MultiTargetLib clashes and understand ToolLib's own file is safe. B did more rigorous empirical verification via msbuild evaluation, which is commendable. However, on the framing dimension the rubric explicitly wants Consume...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — directory-build-organization (claude-sonnet-4.6)

Why: Net win +50.0% (3W/3T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +25.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded

Warnings: Activation: isolated 4/6; plugin 4/6

Overfit: Moderate (score 0.37)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline adding shared build files to a lone project	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
▲ Diagnose a TargetFramework condition that silently skips in props	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose a package downgrade chain and reorganize version management	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose an inner shared-props file that overwrites its own override	Eligible	+0.0%	+0.0%	0/1/0
= Preserve an intentional project-specific exception while centralizing	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a package downgrade chain and reorganize version management: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — directory-build-organization (gpt-5.6-luna)

Why: Net win +0.0% (1W/4T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Warnings: Activation: isolated 5/6; plugin 6/6

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 7 paired runs (1W/5T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Apply repo-level build organization cleanup	Eligible	+0.0%	+0.0%	0/1/0
= Audit a chaotic multi-project repo	Eligible	+0.0%	+0.0%	0/1/0
= Decline adding shared build files to a lone project	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that silently skips in props	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a package downgrade chain and reorganize version management	Eligible	+0.0%	+0.0%	0/1/0
▲ Diagnose an inner shared-props file that overwrites its own override	Eligible	+100.0%	+40.0%	1/0/0
▼ Preserve an intentional project-specific exception while centralizing	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Apply repo-level build organization cleanup: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — eval-performance (claude-sonnet-4.6)

Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +47.5% across 8 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%

Warnings: Activation: isolated 8/8; plugin 7/8

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 8 paired runs (6W/1T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Recognize TreatAsLocalProperty overuse versus one justified entry	Eligible	+0.0%	+0.0%	0/1/0
▼ Triage which of two property functions actually costs evaluation time	Eligible	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Recognize TreatAsLocalProperty overuse versus one justified entry: The responses reach the same fully correct conclusion, substantiate each required point, and provide the same actionable edit. Their minor wording differences do not create a meaningful quality difference.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — eval-performance (gpt-5.6-luna)

Why: Net win +25.0% (3W/4T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +25.0% across 8 paired run(s) — not credible — 4 of 8 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=8; 3W/4T/1L; d=4; p=0.312; net +25.0%

Overfit: Low (score 0.10)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Detect a project evaluated twice under different global properties	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose stacked evaluation-time patterns (deep imports, broad glob, file-I/O property function)	Eligible	+0.0%	+0.0%	0/1/0
▼ Recognize TreatAsLocalProperty overuse versus one justified entry	Eligible	-100.0%	-40.0%	0/0/1
= Redirect a compile-time slowdown mistakenly framed as an evaluation problem	Eligible	+0.0%	+0.0%	0/1/0
= Triage which of two property functions actually costs evaluation time	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Detect a project evaluated twice under different global properties: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — extension-points (claude-sonnet-4.6)

Why: Net win +14.3% (3W/2T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +2.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.500 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded

Warnings: Activation: isolated 4/7; plugin 5/7

Overfit: Low (score 0.19)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Diagnose a broken per-TFM forwarder	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose a package ID and file-name mismatch	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose build extension point failures	Eligible	-100.0%	-100.0%	0/0/1
= Review packed layout without a false missing-file bug	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose a broken per-TFM forwarder: A accurately diagnoses the actual broken relative path and supplies a direct working correction for the shown package. It is incomplete as a general per-TFM-forwarder recommendation and has a briefly confusing discarded path, but B adds materially misleading claims that root b...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — extension-points (gpt-5.6-luna)

Why: Net win +14.3% (2W/4T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 2W/4T/1L; d=3; p=0.500; net +14.3%; 1 dormancy excluded

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Moderate (score 0.24)

Repeated-run reliability (not used by the gate): 8 paired runs (3W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Create extensibility hooks for a custom SDK target file	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose NuGet package and repo extension conflicts	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a package ID and file-name mismatch	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix extension point anti-patterns	Eligible	-100.0%	-40.0%	0/0/1
= Review packed layout without a false missing-file bug	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Create extensibility hooks for a custom SDK target file: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — including-generated-files (gpt-5.6-luna)

Why: Net win +42.9% (4W/2T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +20.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%; 1 dormancy excluded

Overfit: Moderate (score 0.20)

Repeated-run reliability (not used by the gate): 8 paired runs (5W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Diagnose missing clean tracking for generated source	Eligible	-100.0%	-40.0%	0/0/1
= Diagnose missing output registration for generated non-code file	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose wrong hook for generated source files	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose missing clean tracking for generated source: Both responses arrive at the same correct diagnosis and fix. A's explanation is marginally more precise about why clean depends on FileWrites (file-list tracking under obj) and explicitly reinforces that compilation isn't the missing piece. B is slightly more terse. The differ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — incremental-build (claude-sonnet-4.6)

Why: Net win +33.3% (4W/4T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +13.3% across 9 paired run(s) — not credible (sign test p=0.188 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 4W/4T/1L; d=5; p=0.188; net +33.3%

Warnings: Activation: isolated 7/9; plugin 5/9

Overfit: Moderate (score 0.31)

Repeated-run reliability (not used by the gate): 9 paired runs (4W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Correct the assumption that Outputs alone enables incremental skipping	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose custom targets that always rerun	Eligible	+0.0%	+0.0%	0/1/0
= Distinguish a cold first build from broken incrementality	Eligible	+0.0%	+0.0%	0/1/0
▲ Explain slower builds when MSBuild skipped everything	Eligible	+100.0%	+40.0%	1/0/0
▼ Explain why clean leaves generated hash source behind	Eligible	-100.0%	-40.0%	0/0/1
= Fix broken incremental targets and clean tracking	Eligible	+0.0%	+0.0%	0/1/0
= Identify a volatile output path that defeats incrementality	Eligible	+0.0%	+0.0%	0/1/0
▲ Read a diagnostic log to find the stale input	Eligible	+100.0%	+40.0%	1/0/0

Illustrative judge evidence:

Diagnose custom targets that always rerun: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — incremental-build (gpt-5.6-luna)

Why: Net win +44.4% (4W/5T/0L over 9 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +17.8% across 9 paired run(s) — not credible — 5 of 9 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 4W/5T/0L; d=4; p=0.063; net +44.4%

Overfit: Low (score 0.16)

Repeated-run reliability (not used by the gate): 9 paired runs (4W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose custom targets that always rerun	Eligible	+0.0%	+0.0%	0/1/0
= Explain slower builds when MSBuild skipped everything	Eligible	+0.0%	+0.0%	0/1/0
= Explain why clean leaves generated hash source behind	Eligible	+0.0%	+0.0%	0/1/0
= Identify a volatile output path that defeats incrementality	Eligible	+0.0%	+0.0%	0/1/0
= Read a diagnostic log to find the stale input	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose custom targets that always rerun: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — item-management (gpt-5.6-luna)

Why: Net win +16.7% (1W/5T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +11.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 6 preference-eligible stimulus vote(s) tied, leaving only 1 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/5T/0L; d=1; p=0.500; net +16.7%; 1 dormancy excluded

Overfit: Low (score 0.11)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/5T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Diagnose an ineffective Compile Remove that does not match the glob	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose real and claimed item problems in a code generation pipeline	Eligible	+0.0%	+0.0%	0/1/0
= Fix item management anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-correct item management unchanged	Eligible	+0.0%	+0.0%	0/1/0
= Leave correct single-list batching unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Diagnose an ineffective Compile Remove that does not match the glob: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-antipatterns (claude-sonnet-4.6)

Why: Net win +0.0% (1W/5T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.750), mean preference -5.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 1W/5T/1L; d=2; p=0.750; net +0.0%; 1 dormancy excluded

Warnings: Activation: isolated 2/7; plugin 2/7

Overfit: Moderate (score 0.24)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/5T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Add a module to an F# project	Eligible	-100.0%	-40.0%	0/0/1
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
= Distinguish a style backslash from a real cross-platform backslash bug	Eligible	+0.0%	+0.0%	0/1/0
= Fix broken file order causing FS0039	Eligible	+0.0%	+0.0%	0/1/0
= Judge an unguarded import inside a NuGet package build folder	Eligible	+0.0%	+0.0%	0/1/0
= Leave a clean project without inventing anti-patterns	Eligible	+0.0%	+0.0%	0/1/0
▼ Non-activation: migrate a legacy project to SDK style	Excluded (activation contract)	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

Add a module to an F# project: The implemented changes appear functionally equivalent and correct in both runs. A has a modest verification advantage because it explicitly performed and confirmed a successful dotnet build, in addition to running the application.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-antipatterns (gpt-5.6-luna)

Why: Net win +14.3% (3W/2T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +25.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.500 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%; 1 dormancy excluded

Warnings: Activation: isolated 6/7; plugin 5/7

Overfit: Low (score 0.09)

Repeated-run reliability (not used by the gate): 8 paired runs (4W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Add a module to an F# project	Eligible	+100.0%	+40.0%	1/0/0
= Add a signature file to define public API	Eligible	+0.0%	+0.0%	0/1/0
▼ Fix broken file order causing FS0039	Eligible	-100.0%	-40.0%	0/0/1
▼ Judge an unguarded import inside a NuGet package build folder	Eligible	-100.0%	-40.0%	0/0/1
= Review MSBuild files for anti-patterns and style issues	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Add a signature file to define public API: Both responses produced identical, correct results: a valid Domain.fsi with matching type signatures, inserted before Domain.fs, and a successful build. B briefly encountered a --no-restore error but recovered immediately, so the final outcomes are equivalent in quality.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-modernization (claude-sonnet-4.6)

Why: Net win +50.0% (3W/3T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +25.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 6 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 3W/3T/0L; d=3; p=0.125; net +50.0%; 1 dormancy excluded

Warnings: Activation: isolated 6/6; plugin 5/6

Overfit: Low (score 0.19)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/4T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Apply the migration to SDK-style	Eligible	+100.0%	+40.0%	1/0/0
= Consolidate duplicated projects into a multi-targeting SDK-style project	Eligible	+0.0%	+0.0%	0/1/0
= Decline modernizing a non-.NET build	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Modernize a single project without introducing Central Package Management	Eligible	+0.0%	+0.0%	0/1/0
= Recognize an already-modern project needs no migration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Consolidate duplicated projects into a multi-targeting SDK-style project: Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — msbuild-modernization (gpt-5.6-luna)

Why: Net win +33.3% (2W/4T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 6 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 1 dormancy excluded

Overfit: Low (score 0.06)

Repeated-run reliability (not used by the gate): 7 paired runs (2W/4T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Decline modernizing a non-.NET build	Excluded (activation contract)	-100.0%	-40.0%	0/0/1
= Identify legacy patterns for SDK-style migration	Eligible	+0.0%	+0.0%	0/1/0
= Modernize a single project without introducing Central Package Management	Eligible	+0.0%	+0.0%	0/1/0
= Modernize further without introducing a nondeterministic language version	Eligible	+0.0%	+0.0%	0/1/0
= Recognize an already-modern project needs no migration	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Identify legacy patterns for SDK-style migration: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — property-patterns (claude-sonnet-4.6)

Why: Net win +57.1% (4W/3T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +31.4% across 7 paired run(s) — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%

Warnings: Activation: isolated 7/7; plugin 6/7

Overfit: Moderate (score 0.20)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/3T/0L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Choose the correct shared build file for a post-build target	Eligible	+0.0%	+0.0%	0/1/0
▲ Choose the right OS detection for a cross-platform property	Eligible	+100.0%	+40.0%	1/0/0
= Diagnose multi-level property hierarchy bugs	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose shared build property issues	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Choose the correct shared build file for a post-build target: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — property-patterns (gpt-5.6-luna)

Why: Net win -14.3% (0W/6T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -5.7% across 7 paired run(s) — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 0W/6T/1L; d=1; p=0.500; net -14.3%

Warnings: Activation: isolated 6/7; plugin 7/7

Overfit: Low (score 0.07)

Repeated-run reliability (not used by the gate): 7 paired runs (0W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Choose the correct shared build file for a post-build target	Eligible	+0.0%	+0.0%	0/1/0
= Choose the right OS detection for a cross-platform property	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose a TargetFramework condition that never applies in props	Eligible	+0.0%	+0.0%	0/1/0
= Diagnose multi-level property hierarchy bugs	Eligible	+0.0%	+0.0%	0/1/0
▼ Diagnose shared build property issues	Eligible	-100.0%	-40.0%	0/0/1
= Fix shared property configuration	Eligible	+0.0%	+0.0%	0/1/0
= Leave already-correct shared properties unchanged	Eligible	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Choose the correct shared build file for a post-build target: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — resolve-project-references (gpt-5.6-luna)

Why: Net win +0.0% (1W/4T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +0.0% across 8 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=6; 1W/4T/1L; d=2; p=0.750; net +0.0%; 2 dormancy excluded

Warnings: Activation: isolated 4/6; plugin 6/6

Overfit: Low (score 0.14)

Repeated-run reliability (not used by the gate): 8 paired runs (1W/6T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Distinguish wait time from a real serial dependency chain	Eligible	+0.0%	+0.0%	0/1/0
▲ Give the exact command to replay task self-time	Eligible	+100.0%	+40.0%	1/0/0
= Hand off a wide graph blocked by single-node scheduling	Eligible	+0.0%	+0.0%	0/1/0
= Rank Copy ahead of Csc when task self-time is higher	Eligible	+0.0%	+0.0%	0/1/0
= Redirect from misleading target summary to Csc self-time	Eligible	+0.0%	+0.0%	0/1/0
▼ Require diagnostic evidence before analyzing project-reference time	Eligible	-100.0%	-40.0%	0/0/1
= Stay dormant on a general build performance review	Excluded (activation contract)	+0.0%	+0.0%	0/1/0
= Stay dormant when Csc is already the obvious bottleneck	Excluded (activation contract)	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

Distinguish wait time from a real serial dependency chain: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 2 results are in Full Results.

Details for 6 results were omitted to keep this comment under GitHub's limit. Open the workflow summary or Full Results for the complete breakdown.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1118 in dotnet/skills, download eval artifacts with gh run download 34310235930 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/37ceee80d21a676da1e07bc7cd67629d20609070/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

Use off-target boundaries that remain meaningful when the target is the only loaded skill, while retaining anti-hijack coverage for compilation, T4, property, and compiler-performance workflows. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2d0a0a20-fd83-4343-8434-4763f8120503

github-actions · 2026-09-09T06:20:24Z

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 6a0ca12d2af2520642beadde77116d339e4d83a2 to retry this exact commit.

github-actions · 2026-09-09T06:26:47Z

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 6a0ca12d2af2520642beadde77116d339e4d83a2 to retry this exact commit.

github-actions · 2026-09-09T22:07:26Z

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 6a0ca12d2af2520642beadde77116d339e4d83a2 to retry this exact commit.

36 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

github-actions · 2026-09-09T23:13:20Z

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 6a0ca12d2af2520642beadde77116d339e4d83a2 to retry this exact commit.

36 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

github-actions · 2026-09-10T00:17:55Z

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 6a0ca12d2af2520642beadde77116d339e4d83a2 to retry this exact commit.

36 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

AbhitejJohn added 2 commits September 3, 2026 17:06

test(msbuild): strengthen eval quality

0d177e5

Expand MSBuild eval coverage with realistic fixtures, replayable references, and a changed-suite quality ratchet. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb11c85b-4046-4dbc-94b3-3bdbd4839a0d

test(msbuild): isolate missing-package fixture

0d63a77

Clear NuGet fallback folders so the intentional NU1101 case cannot resolve from machine state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb11c85b-4046-4dbc-94b3-3bdbd4839a0d

AbhitejJohn mentioned this pull request Sep 4, 2026

Move heavy eval fixtures off the skill-consumption critical path #1121

Open

AbhitejJohn commented Sep 4, 2026

View reviewed changes

github-actions Bot added a commit that referenced this pull request Sep 4, 2026

Update PR token usage data (PR #1118)

395c6b6

github-actions Bot added a commit that referenced this pull request Sep 4, 2026

Update PR token usage data (PR #1118)

1fa06d5

JanKrivanek reviewed Sep 7, 2026

View reviewed changes

Comment thread eng/eval-quality/check_eval_quality.py

AbhitejJohn commented Sep 8, 2026

View reviewed changes

github-actions Bot added a commit that referenced this pull request Sep 8, 2026

Update PR token usage data (PR #1118)

2681715

github-actions Bot added a commit that referenced this pull request Sep 8, 2026

Update PR token usage data (PR #1118)

d2bbab0

github-actions Bot added a commit that referenced this pull request Sep 8, 2026

Update PR token usage data (PR #1118)

95e62d7

github-actions Bot added a commit that referenced this pull request Sep 8, 2026

Update PR token usage data (PR #1118)

e0c22f7

github-actions Bot added a commit that referenced this pull request Sep 9, 2026

Update PR token usage data (PR #1118)

7ba15c1

github-actions Bot added a commit that referenced this pull request Sep 9, 2026

Update PR token usage data (PR #1118)

2dfb4ef

github-actions Bot added a commit that referenced this pull request Sep 9, 2026

Update PR token usage data (PR #1118)

05ea98a

github-actions Bot added a commit that referenced this pull request Sep 9, 2026

Update PR token usage data (PR #1118)

26783da

github-actions Bot added a commit that referenced this pull request Sep 9, 2026

Update PR token usage data (PR #1118)

77b693f

github-actions Bot added a commit that referenced this pull request Sep 9, 2026

Update PR token usage data (PR #1118)

9754d86

github-actions Bot added a commit that referenced this pull request Sep 9, 2026

Update PR token usage data (PR #1118)

2bf5376

github-actions Bot added a commit that referenced this pull request Sep 10, 2026

Update PR token usage data (PR #1118)

8b44625

Uh oh!

Conversation

AbhitejJohn commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design principles

Review map: what changed in each eval

Shared quality tooling

Repository-space impact

Reliability and safety

Validation

Scope

Uh oh!

AbhitejJohn commented Sep 4, 2026

Uh oh!

AbhitejJohn left a comment

Choose a reason for hiding this comment

Uh oh!

AbhitejJohn commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

AbhitejJohn commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

📊 Skill Evaluation Results

Uh oh!

JanKrivanek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AbhitejJohn left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

📊 Skill Evaluation Results

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

📊 Skill Evaluation Results

Uh oh!

github-actions Bot commented Sep 9, 2026

📊 Skill Evaluation Results

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

📊 Skill Evaluation Results

Uh oh!

github-actions Bot commented Sep 9, 2026

📊 Skill Evaluation Results

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

AbhitejJohn commented Sep 4, 2026 •

edited

Loading