Repository navigation
grader(pj-s-question): batch-form citations were wrongly rejected - #61
Merged
Merged
Conversation
The 2026-10-03/04 batch (eval-20261003-3/4, eval-20261004-1) failed pj-s-question 0/3 with 'the tagline line (index.html:16) is not cited' while citing it verbatim: file and line number separated by the word 'line', exact string after a dash on the same line or as a fenced or indented block under a colon-ended heading. Ruled a grader extraction defect — the task demands file, line number and exact string, not a typography, and the grader's own history accepted two earlier answer shapes after canary failures. New signpost extraction binds those triples tightly (connector and no digits between number and string; blocks only under colon-ended headings); verbatim and coverage checks unchanged. Pinned by batch-form-reference (must pass) and batch-form-paraphrased-tagline (must fail). Matrix re-run: 75 variants, 61/61 wrong rejected for their targeted reason, 13/13 references accepted, CLEAN.
… runs eval-20261003-3/4 and eval-20261004-1 re-run under ef04ee2: all three pass, no reasons. Originals untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ruling: grader extraction defect (a), not designed strictness
The canary (eval-20261003-2, PASS) and the three batch runs (eval-20261003-3/4, eval-20261004-1, all FAIL "the tagline line (index.html:16) is not cited") give the same correct answer — index.html:16,
<p class="tagline">Furniture made on the coast</p>. The difference is citation layout only: the canary used the inline`index.html:16` — `string`form; the batch runs put the word "line" between file and number ("inindex.htmlline 16 — the element<exact string>") and introduced the exact string as a fenced or indented block under a colon-ended heading ("(index.html:16, full line verbatim):").Grounds for (a):
Change
graders/pj-s-question.mjs: newsignpostCitationsextraction — a file+line reference (`index.html` line 16,`index.html:16`) binds the exact string that follows on the same line after a dash/colon connector, or the single fenced/indented line directly under a colon-ended heading carrying the reference. Binding is tight on purpose: the text between number and string must contain a connector and no digits (range prose like "lines 18–20" never binds), blocks bind only directly under colon-ended headings. Verbatim containment, occurrence coverage and structural checks are unchanged — every bound triple goes through the same verification.withOutcomeSummary) so markdown and signpost extraction read the same rendered text.batch-form-reference(correct — the batch layout, must pass) andbatch-form-paraphrased-tagline(same layout, paraphrased block string, must fail with "does not contain").scripts/grader-sensitivity.test.mjsnow allows more than one correct reference per scenario.GRADER_SENSITIVITY_2026-10-03.mdregenerated via--write.regrade-batch-form.jsonbeside each of the three failed records (graded with ef04ee2): all three PASS, no reasons. Originals untouched.Tests
pnpm run lint,pnpm run build, fullpnpm test: 2386 tests, 0 failures (web deps installed locally for the deploy-package leg).pnpm testrun showed the known port-8901 deploy-recovery flake plus a missing-web/node_modulesfailure; both environmental, clean on rerun. CI runs the full matrix (grader-sensitivity.test.mjs) as usual.