Skip to content

grader(pj-s-question): batch-form citations were wrongly rejected - #61

Merged
im-tyler merged 2 commits into
mainfrom
w5/grader-question-ruling
Oct 4, 2026
Merged

im-tyler merged 2 commits into
mainfrom
w5/grader-question-ruling

Conversation

@im-tyler

@im-tyler im-tyler commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Ruling: grader extraction defect (a), not designed strictness

The canary (eval-20261003-2, PASS) and the three batch runs (eval-20261003-3/4, eval-20261004-1, all FAIL "the tagline line (index.html:16) is not cited") give the same correct answer — index.html:16, <p class="tagline">Furniture made on the coast</p>. The difference is citation layout only: the canary used the inline `index.html:16` — `string` form; the batch runs put the word "line" between file and number ("in index.html line 16 — the element <exact string>") and introduced the exact string as a fenced or indented block under a colon-ended heading ("(index.html:16, full line verbatim):").

Grounds for (a):

  • The task's stated demand is "Answer with file, line number and the exact current string" — the batch answers supply exactly that, adjacently bound. No format rule is violated.
  • The grader's own header is "tolerant of the separators a real answer uses", and its history records two prior fixes of exactly this class: single quotes accepted after the 2026-09-24 canary failed, FINDINGS_JSON accepted after the 2026-09-22 canary ("a correct scan failed a grader that understood only one answer syntax"). This is the third instance of that defect class, found by live runs.
  • The mutants corpus enforces content, not typography: every pj-s-question wrong variant targets a wrong line number, a paraphrase, a missed occurrence, an edit, or no citations at all. The batch transcripts are correct on every axis the corpus pins.
  • GRADER_SENSITIVITY_2026-10-03 (PR S02: grader sensitivity matrix; tighten eight graders it showed were too weak #34) tightened graders against escaping WRONG variants; its known-limit doctrine explicitly rejects strictness that "would reject the honest behaviour".

Change

  • graders/pj-s-question.mjs: new signpostCitations extraction — a file+line reference (`index.html` line 16, `index.html:16`) binds the exact string that follows on the same line after a dash/colon connector, or the single fenced/indented line directly under a colon-ended heading carrying the reference. Binding is tight on purpose: the text between number and string must contain a connector and no digits (range prose like "lines 18–20" never binds), blocks bind only directly under colon-ended headings. Verbatim containment, occurrence coverage and structural checks are unchanged — every bound triple goes through the same verification.
  • Outcome→summary normalization hoisted (withOutcomeSummary) so markdown and signpost extraction read the same rendered text.
  • New mutants: batch-form-reference (correct — the batch layout, must pass) and batch-form-paraphrased-tagline (same layout, paraphrased block string, must fail with "does not contain"). scripts/grader-sensitivity.test.mjs now allows more than one correct reference per scenario.
  • FINDINGS gains a ninth-defect note; GRADER_SENSITIVITY_2026-10-03.md regenerated via --write.
  • Supplementary regrade-batch-form.json beside each of the three failed records (graded with ef04ee2): all three PASS, no reasons. Originals untouched.

Tests

  • Full matrix regenerated: 75 variants — 61/61 wrong rejected for their targeted reason, 13/13 correct references accepted, 1 known limit intact, CLEAN.
  • pnpm run lint, pnpm run build, full pnpm test: 2386 tests, 0 failures (web deps installed locally for the deploy-package leg).
  • Note: one earlier local pnpm test run showed the known port-8901 deploy-recovery flake plus a missing-web/node_modules failure; both environmental, clean on rerun. CI runs the full matrix (grader-sensitivity.test.mjs) as usual.

The 2026-10-03/04 batch (eval-20261003-3/4, eval-20261004-1) failed
pj-s-question 0/3 with 'the tagline line (index.html:16) is not cited'
while citing it verbatim: file and line number separated by the word
'line', exact string after a dash on the same line or as a fenced or
indented block under a colon-ended heading. Ruled a grader extraction
defect — the task demands file, line number and exact string, not a
typography, and the grader's own history accepted two earlier answer
shapes after canary failures. New signpost extraction binds those
triples tightly (connector and no digits between number and string;
blocks only under colon-ended headings); verbatim and coverage checks
unchanged. Pinned by batch-form-reference (must pass) and
batch-form-paraphrased-tagline (must fail). Matrix re-run: 75 variants,
61/61 wrong rejected for their targeted reason, 13/13 references
accepted, CLEAN.
… runs

eval-20261003-3/4 and eval-20261004-1 re-run under ef04ee2: all three
pass, no reasons. Originals untouched.
@im-tyler
im-tyler merged commit b94ed75 into main Oct 4, 2026
4 checks passed
@im-tyler
im-tyler deleted the w5/grader-question-ruling branch October 4, 2026 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant