feat(evalforge-yaml-gate): scenario lint, confirm-on-fail, and quarantine list

[danshw]

Adds a decision layer to the wix-manage eval gate so a single noisy eval run no longer decides a PR's fate, scenario-authoring standards are enforced instead of documented-only, and known-flaky scenarios stop blocking unrelated PRs.

Scenario lint

Every scenario a PR adds or modifies is linted before any EvalForge call, and violations fail the check with a per-rule table comment:

rule	requirement
three-assertions	≥3 assertions (doc tool-call, correctness judge, quality judge)
coverage-assertion	a tool-call assertion with params.articleUrl on the skill doc
two-llm-judges	≥2 llm_judge assertions (outcome + tool-call path)
min-score-floor	numeric judges declare minScore ≥ 6
judge-fail-criteria	judge prompts state explicit fail criteria
task-shaped-prompt	triggerPrompt is a task with concrete values, not "how do I…"
max-tokens	top-level maxTokens set

Scenarios untouched by the PR are never linted, so existing content is ratcheted up only when edited.

Confirm-on-fail

Eval runs are not deterministic, so the gate no longer blocks on a single failed attempt. Scenarios that fail the comparison (LLM-judge or token budget) are rerun — up to two targeted retries of just the failing scenario IDs — and the gate blocks only when a majority of attempts fail. The PR comment separates confirmed failures from recovered flaky runs. Guardrails:

• more than 10 first-attempt failures skips retries entirely (broad failure is signal, not noise)
• a retry-infrastructure error falls back to the first attempt's verdict and still blocks — no path to a silent green
• auto-approve now requires a clean first attempt
• job timeout raised 60 → 120 min to cover the retry worst case

Quarantine list

yaml/wix-manage-evals/quarantine.yaml lists flaky scenarios awaiting rewrite. A quarantined scenario is skipped when a PR pulls it in only via doc coverage, but still runs when the PR edits the scenario's own YAML. Entries are loaded from the base branch, so a PR cannot quarantine scenarios for its own run — entries take effect only after their own reviewed PR merges.

Also

• gate ordering: all local checks (including the lint) now run before the EvalForge MCP-version mutation
• the gate's PR comment now composes comparison, confirm-on-fail, and quarantine sections into one body, so run links survive follow-up updates
• docs: lint rules in docs/eval-scenarios.md, confirm-on-fail + quarantine in docs/skill-evaluation.md, CONTRIBUTING checklist pointer

Testing

• 111 vitest tests (24 new across lint/quarantine/confirm/comment composition), tsc --noEmit clean, dist rebuilt
• follow-up: throwaway PR exercising a lint violation end-to-end against the live gate

🤖 Generated with Claude Code

[github-actions]

✅ EvalForge YAML Gate: No Gated Changes

Nothing under evals/ or sibling .md changed.