From a4996098f2c43e34f376af4abe51bdc21bccb3e6 Mon Sep 17 00:00:00 2001 From: Omar Eid Date: Wed, 12 Aug 2026 13:19:30 -0500 Subject: [PATCH 01/95] The trial says which mark, softly enough not to say where MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A crop of print this size holds several marks and often two of the same name, so a trial that only said "put the box on the mark" was a trial about whichever mark the reader picked. That is not a small error: a placement on the neighbouring mark is a whole letter out, and the residual would have recorded a whole letter of registration error that had nothing to do with registration. So identification now travels with the trial rather than being recoverable from it. The corpus reader keeps the ligature a mark was drawn inside — the only join in that print between a mark and the letters it belongs to — and the rank is counted over the ligature rather than the word, because a word can run off the edge of a crop and "the third of three" over letters half of which are off-screen is worse than saying nothing. The first way of pointing at those letters was a mistake worth keeping in the record. Drawing them crisply in colour looks obviously right and would have quietly destroyed the measurement: the letters come from the other printing's drawing, carried onto our frame by exactly the fit that places the rectangle, so the visible gap between them and the ink underneath IS the correction, about a page unit, which at the size these panels are worked is a finger's width on screen. An hour of that and the landings would have been a tracing of our own answer. It is a wide blurred wash now, with an edge several times softer than the correction and no dependence on the fit being right — it says these letters and refuses to say anything finer. The mark itself is never washed. And one line of CSS, because giving .trial a display of its own beat the browser's rule for the hidden attribute and put all sixty cards on screen at once while the drag still moved the rectangle on whichever one the session thought was current. Restating the hidden case is the fix; the comment says why, since element screenshots cannot see this. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt --- packages/etl/scripts/build-mark-nudge.mjs | 153 +++++++++++++++++- packages/etl/scripts/lib/adjudication.mjs | 15 ++ .../etl/scripts/lib/adjudication.test.mjs | 36 +++++ packages/etl/scripts/lib/diacritics.mjs | 52 +++++- packages/etl/scripts/lib/marks.mjs | 46 +++++- 5 files changed, 292 insertions(+), 10 deletions(-) diff --git a/packages/etl/scripts/build-mark-nudge.mjs b/packages/etl/scripts/build-mark-nudge.mjs index c72fb61..baf3489 100644 --- a/packages/etl/scripts/build-mark-nudge.mjs +++ b/packages/etl/scripts/build-mark-nudge.mjs @@ -83,6 +83,105 @@ function fingerprint(s) { const n2 = (v) => Math.round(v * 100) / 100; const esc = (s) => String(s).replace(/[<>&"]/g, (c) => ({ "<": "<", ">": ">", "&": "&", '"': """ })[c]); +/** + * Which one, in words. + * + * Small numbers spelled out because they are read mid-sentence — "the 2nd of 3" + * is a table cell and this is a sentence somebody reads once, quickly, before + * looking at the ink. + */ +const NTH = ["", "first", "second", "third", "fourth", "fifth", "sixth", "seventh", "eighth"]; +const HOW_MANY = ["", "one", "two", "three", "four", "five", "six", "seven", "eight"]; +const nth = (n) => NTH[n] ?? `${n}th`; +const howMany = (n) => HOW_MANY[n] ?? String(n); + +/** + * How soft the wash over the host letters is, in page units: the blur, and how + * far it is grown past the letter outline before blurring. + * + * These two numbers are the whole safety argument, so they are named rather than + * buried. The correction being measured is about one unit; the wash's edge is a + * gradient a good deal wider than that, so a wash drawn a unit out of place looks + * exactly like one drawn correctly. Shrink them and this stops being true. + */ +const WASH_BLUR = 0.9; +const WASH_GROW = 1.6; + +/** + * A soft wash over the letters the mark was drawn on — which ones, and nothing + * about where. + * + * ## Why it is a wash and not a tint + * + * The first version of this filled the host letters in a second colour, and it + * was wrong in a way worth writing down, because it looked obviously safe. + * + * The ink in the panel is the **shipped** page — about fifteen paths, one per + * line, so there is no way to pick a word out of it. The letter outlines come + * from the **ligature corpus** instead, a different vendored drawing of the same + * print, carried onto our frame by the page's recorded fit. That fit is exactly + * what places the mark rectangle. So the gap between a crisply-drawn corpus + * letter and the shipped ink underneath it *is the correction*, drawn on the + * screen at about twenty-five pixels, in every panel, for an hour. A reader would + * not even have to notice they were copying it. + * + * A wash cannot say that. It is grown past the outline and then blurred, so its + * edge is a gradient several times wider than the correction; the same wash is + * produced whether the fit is right or a unit out. What survives is the only part + * we wanted — *these letters, not those* — and what is destroyed is the part that + * would have been an answer key. + * + * It is painted under the ink rather than over it, so the print the reader is + * judging against stays exactly as black and as crisp as it was. + * + * ## The mark itself + * + * Never washed, never traced, never drawn. Its position is the entire + * measurement. It stays as the print set it, indistinguishable from its + * neighbours, which is the condition under which a placement means anything. + */ +function hostWash(lig, fit, i) { + if (!lig || !lig.letters.length) return ""; + const ds = lig.letters.map((d) => ``).join(""); + // The filter sits on the outer group and the frame change on the inner one, so + // the blur and the growth are both in page units — the units the correction is + // measured in, and therefore the only units in which "wider than the thing it + // has to hide" means anything. + return ( + `` + + `` + + `` + + `${ds}` + ); +} + +/** + * The sentence that says which mark, above the panel. + * + * Three parts, and each is there because without it a real card in this session + * is ambiguous: the mark's name, the letters it sits on, and — only when those + * letters carry more than one mark of that name — which of them, counting from + * the right, because that is the direction the letters are read in. + * + * The rank is counted over the washed letters and says so, rather than over the + * whole word. A word can run off the edge of the crop, and "the third of three" + * over letters the reader cannot all see is worse than saying nothing at all. + * + * A mark the corpus drew under a word but inside no ligature gets no letters and + * says so. That is rare and it is better said than papered over: a reader told + * "the fatha on ذ" when nothing is washed would spend the trial hunting for the + * wash rather than placing the box. + */ +function whichOne(t) { + const on = t.lig && t.lig.text ? ` on ${esc(t.lig.text)}` : ""; + const scope = t.lig ? "on those letters" : "in this word"; + const rank = t.of > 1 ? ` — the ${nth(t.nth)} of ${howMany(t.of)} ${scope}, counting from the right` : ""; + const lost = t.lig ? "" : " — this one sits under the word rather than on a letter, so nothing is washed"; + return `

the ${esc(t.name)}${on}${rank}${lost}

`; +} + /** The page's own outlines, clipped to the window. Ink and nothing else. */ function inkPaths(shapes, vx, vy, side) { const parts = []; @@ -137,7 +236,8 @@ const cards = trials data-x="${n2(x)}" data-y="${n2(y)}" data-w="${n2(w)}" data-h="${n2(h)}" data-vx="${n2(vx)}" data-vy="${n2(vy)}" data-side="${n2(side)}" data-sx="${n2(sx)}" data-sy="${n2(sy)}" hidden> - +${whichOne(t)} +

page ${t.page} · ${t.surah}:${t.aya}

`; }) @@ -159,22 +259,49 @@ const head = { * feature. */ asks: ["place", "size"], + /** + * What a card put in front of the reader. It is part of the browser's resume + * key, below, and that is the point of writing it down: a placement made on a + * card that never said which mark it meant is not the same measurement as one + * made on a card that did, and a rebuild that changes the question has to + * start the session over rather than quietly stack the two on top of each + * other. The scorer reads it for the same reason the forced-choice page's + * `asks` is read — so a reading cannot assume a question the page had no way + * to put. + */ + shows: ["ink", "box", "which-mark"], }; const html = `Hifth — put the box where it goes + + +
+

Is this rectangle on the right mark?

+ +

The orange rectangle is where the app would draw this mark. Around it is the +ink the mus'haf actually prints there, at whatever size your screen gives it. The letters under +the heading are the ones the print drew this mark on — that is the mark the rectangle should be +sitting on, and when a crop holds several of the same name, the line beside them says which. +These marks were drawn at random, so most of them should look right — saying so is a real +answer, and it is the one you should expect to give most often. When something is wrong, more +than one thing can be.

+

+ +
+
+
+ +
+ + +
+ +

+
+ + + + + +
+ +
    +

    +
    + + +
    +
    +
    + +
    +
    + + +
    +
    +
    + + +
    + + +

    + +
    + + +
    +     + + + + +`; + +mkdirSync(dirname(out), { recursive: true }); +writeFileSync(out, html); +console.log( + `${HEAD.pool} of ${HEAD.of} marks in the ${set} set` + + (HEAD.alreadyAnswered ? ` · ${HEAD.alreadyAnswered} already answered, left out` : "") + + ` · ${cards.length} drawn · displacements ${rowsFp}`, +); +console.log(` drawn by ${Object.entries(HEAD.drawnBy).map(([k, v]) => `${k} ${v}`).join(", ")}`); + +/** + * What this many marks can and cannot find — printed before the sitting rather than + * discovered after it. With n answers and nothing wrong in any of them, the rule of + * three puts the upper bound on the true rate at about 3/n; a sitting of 60 that + * comes back clean has shown the failure rate is under roughly 5%, and has shown + * nothing whatever about whether it is 1% or 0.01%. + * + * This is here because a clean sitting is exactly the result somebody will want to + * quote as "we checked it", and the honest version of that sentence has a number in + * it. The instrument that will be over-read is the one that has to say so itself. + */ +console.log(` a clean sitting of ${cards.length} bounds the failure rate at about ${((3 / cards.length) * 100).toFixed(1)}% — not at zero`); +console.log(`wrote ${out.replace(`${ROOT}/`, "")}`); diff --git a/packages/etl/scripts/build-mark-report.test.mjs b/packages/etl/scripts/build-mark-report.test.mjs new file mode 100644 index 0000000..531f28b --- /dev/null +++ b/packages/etl/scripts/build-mark-report.test.mjs @@ -0,0 +1,648 @@ +/** + * The arithmetic behind the words "we looked at all of them". + * + * Somebody asked to validate the remaining marks until we are at 100%, and the only + * way a person gets to say that at the end is if the files they sat really were the + * population — every mark in exactly one of them. That is a property of how the + * pages are cut, it is invisible from inside any single page, and it fails silently: + * sixteen files that between them miss forty marks look exactly like sixteen files + * that do not, and the shortfall is discovered — if ever — by somebody re-deriving + * it months later, after the claim has been written into a register. + * + * So the partition is checked here rather than by eye, and it is checked by reading + * the ids back out of the built pages rather than by re-running the selection: the + * artifact a person sits is the thing the claim is about. Re-running the chooser and + * comparing it against itself would pass for a builder that writes empty files. + * + * The band tests are here for a different reason. A band is how the sitting plan + * proposes to spend forty hours on the marks the machine is least sure of, so a band + * that quietly included a mark from outside it, or that overlapped its neighbour, + * would corrupt the one comparison the whole plan turns on. + */ +import { execFileSync } from "node:child_process"; +import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { markPageFile, marksOf } from "./lib/marks.mjs"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const SCRIPT = join(HERE, "build-mark-report.mjs"); + +const RADIUS = 3; +const IOU = 0.55; + +/** + * Sixty marks on one line of page 1 — one line so the line-tilt correction clears + * its twenty-mark floor and the refused marks are drawn by a fit that actually ran, + * and page 1 because the builder reads that page's real ink and a made-up page + * number has none. The spread of `iouBest` is deliberate: it puts marks in every + * band the sitting plan cuts, so a band test has something to be wrong about. + * + * The index and the name are the print's own, not invented, because the builder now + * looks up which letters each mark was drawn on in order to caption it, and refuses + * a row whose name does not match the mark that index names. A fixture of sixty + * hand-written kasras would fail that check on the first card — correctly, which is + * the point of the check, and uselessly, which is why the fixture reads them. The + * geometry stays synthetic: the boxes are what puts the marks on one line and in + * every band, and nothing here is measuring where they really are. + */ +const REAL = existsSync(markPageFile(1)) ? marksOf(1).slice(0, 60) : []; +const ROWS = REAL.map((m, k) => ({ + page: 1, + k: m.k, + line: 1, + name: m.name, + box: [180 - k * 2.5, 40, 5.6, 3.6], + ink: 0.1, + dx: k % 17 === 0 ? RADIUS : 0.5, + dy: -0.25, + iou0: 0, + iouBest: 0.4 + (k % 12) * 0.05, + phi0: 0, + nullPhi: 0, +})); + +/** + * The ligature corpus is a 380 MB cache that is not in the repo, and the builder + * cannot caption a mark without it. Skipping says so out loud rather than failing + * on a machine that was never going to be able to run this. + */ +const haveCorpus = ROWS.length === 60; + +const atEdge = (r) => Math.abs(Math.abs(r.dx) - RADIUS) < 1e-6 || Math.abs(Math.abs(r.dy) - RADIUS) < 1e-6; +const isPlaced = (r) => r.iouBest >= IOU && !atEdge(r); +const id = (r) => `${r.page}:${r.k}`; + +let dir; +let rowsPath; +let nth = 0; + +beforeAll(() => { + dir = mkdtempSync(join(tmpdir(), "hifth-build-")); + rowsPath = join(dir, "rows.json"); + writeFileSync(rowsPath, JSON.stringify(ROWS)); +}); +afterAll(() => rmSync(dir, { recursive: true, force: true })); + +/** Build one page and hand back what a reader would be shown, read off the page. */ +function build(...flags) { + const out = join(dir, `page-${(nth += 1)}.html`); + execFileSync(process.execPath, [SCRIPT, "--rows", rowsPath, "--out", out, ...flags], { + encoding: "utf8", + stdio: "pipe", + }); + const html = readFileSync(out, "utf8"); + return { + html, + ids: [...html.matchAll(/"id":"(\d+:\d+)"/g)].map((m) => m[1]), + head: JSON.parse(/^const HEAD = (\{.*\});$/m.exec(html)[1]), + cards: JSON.parse(/^const CARDS = (\[.*\]);$/m.exec(html)[1]), + }; +} + +describe.skipIf(!haveCorpus)("cutting a population into parts somebody can finish", () => { + it("makes the parts add up to the population exactly", () => { + const want = ROWS.filter((r) => !isPlaced(r)).map(id); + const seen = new Map(); + let cards = 0; + for (let n = 1; n <= 5; n += 1) { + const got = build("--set", "fallback", "--part", `${n}/5`); + cards += got.ids.length; + for (const one of got.ids) seen.set(one, n); + } + // Three separate claims, because they fail differently: nothing shown twice, + // nothing left out, and no mark shown that was never in the population. + expect(cards).toBe(seen.size); + expect([...seen.keys()].sort()).toEqual([...want].sort()); + expect(seen.size).toBe(want.length); + }); + + it("gives each part its own place to keep answers", () => { + // Same fingerprint, same set, same seed — only the part differs. If that did not + // reach the storage key, part 2 would open showing part 1's answers already + // given, and the reader would page past marks nobody ever looked at. + const a = build("--set", "fallback", "--part", "1/5"); + const b = build("--set", "fallback", "--part", "2/5"); + expect(a.head.slice).not.toBe(b.head.slice); + }); + + it("leaves the key alone when no part and no band were asked for", () => { + // A sitting banked before these flags existed is keyed without them. Adding a + // flag must not orphan somebody's finished half-hour. + expect(build("--set", "fallback", "--count", "5").head.slice).toBe(""); + }); + + it("puts the slice into the storage key", () => { + // The key is assembled at runtime from the head, so the tests above compare + // slices rather than keys. This is the other half: that the slice reaches the + // key at all. Rebuilding the formula here instead would go on passing happily + // the day somebody drops it, which is the only failure worth catching. + expect(build("--set", "fallback", "--count", "5").html).toMatch(/const KEY = .*HEAD\.slice/); + }); + + it("refuses a part that is not one of m", () => { + expect(() => build("--set", "fallback", "--part", "6/5")).toThrow(); + expect(() => build("--set", "fallback", "--part", "0/5")).toThrow(); + }); +}); + +describe.skipIf(!haveCorpus)("drawing one band of confidence at a time", () => { + it("shows only marks whose match falls inside the band", () => { + const got = build("--set", "placed", "--band", "0.65,0.75", "--count", "100"); + const inside = new Set( + ROWS.filter((r) => isPlaced(r) && r.iouBest >= 0.65 && r.iouBest < 0.75).map(id), + ); + expect(got.ids.length).toBeGreaterThan(0); + for (const one of got.ids) expect(inside.has(one)).toBe(true); + }); + + it("does not let one band hand a mark to the next", () => { + const lower = build("--set", "placed", "--band", "0.65,0.75", "--count", "100"); + const upper = build("--set", "placed", "--band", "0.75,0.85", "--count", "100"); + const shared = lower.ids.filter((one) => upper.ids.includes(one)); + expect(shared).toEqual([]); + }); + + it("keeps two bands' answers apart", () => { + const lower = build("--set", "placed", "--band", "0.65,0.75", "--count", "5"); + const upper = build("--set", "placed", "--band", "0.75,0.85", "--count", "5"); + expect(lower.head.slice).not.toBe(upper.head.slice); + }); + + it("stops rather than build a sitting with nothing in it", () => { + // An empty band is a typo almost every time. Writing the page anyway produces a + // file that opens, says "that is all of them", and banks a transcript claiming a + // clean run over zero marks. + expect(() => build("--set", "placed", "--band", "0.01,0.02")).toThrow(); + }); + + it("refuses a band that runs backwards", () => { + expect(() => build("--set", "placed", "--band", "0.8,0.6")).toThrow(); + }); +}); + +/** + * The count only comes down if answering actually takes a mark out of the pool. + * + * Somebody asked how to submit answers one at a time and see the number left fall, + * and the two halves are separable: the submitting is a server, but the falling is + * this. Without it the sixteen parts are rebuilt from the whole population every + * time and a reader who has answered two hundred marks is handed back exactly what + * they started with — which is both dispiriting and, worse, indistinguishable from + * a rebuild that lost their work. + */ +describe.skipIf(!haveCorpus)("taking answered marks out of what is left", () => { + const answersAt = (name, evs) => { + const p = join(dir, name); + writeFileSync(p, JSON.stringify({ said: evs })); + return p; + }; + + it("does not show a mark somebody has already answered", () => { + const before = build("--set", "fallback", "--count", "100"); + const gone = before.ids.slice(0, 4); + const after = build( + "--set", "fallback", "--count", "100", + "--answered", answersAt("a1.json", gone.map((one) => ({ kind: "looks-right", id: one }))), + ); + for (const one of gone) expect(after.ids).not.toContain(one); + expect(after.ids.length).toBe(before.ids.length - gone.length); + }); + + it("counts a retracted answer as no answer at all", () => { + // Said and then unsaid is not an answer, and the marks a reader took back are + // the hard ones — precisely the ones it would be worst to quietly bury. The + // running log keeps both statements because it is appended to, never rewritten, + // so this arithmetic is the only thing standing between the two readings. + const before = build("--set", "fallback", "--count", "100"); + const one = before.ids[0]; + const p = join(dir, "a2.jsonl"); + writeFileSync( + p, + [ + JSON.stringify({ kind: "report", payload: { kind: "placement", id: one } }), + JSON.stringify({ kind: "report", payload: { kind: "retracted", id: one, was: "placement" } }), + ].join("\n") + "\n", + ); + expect(build("--set", "fallback", "--count", "100", "--answered", p).ids).toContain(one); + }); + + it("gives a shrunken sitting its own place to keep answers", () => { + // The kept position is an index into the cards. Drop marks and leave the key + // alone and that index points into a set it was never measured against — a + // reader who reached card ninety of a hundred and seventeen reopens a rebuilt + // sitting of eighty and is told it is finished, having never seen most of it. + const before = build("--set", "fallback", "--count", "100"); + const after = build( + "--set", "fallback", "--count", "100", + "--answered", answersAt("a3.json", [{ kind: "looks-right", id: before.ids[0] }]), + ); + expect(after.head.slice).not.toBe(before.head.slice); + expect(before.head.slice).toBe(""); + }); + + it("says how big the population was before the answered ones came out", () => { + // "80 of 117" is unreadable a month later without the number it came down from, + // and a transcript that cannot say which pass of the population it belongs to + // cannot be pooled with the one before it. + const before = build("--set", "fallback", "--count", "100"); + const after = build( + "--set", "fallback", "--count", "100", + "--answered", answersAt("a4.json", before.ids.slice(0, 3).map((one) => ({ kind: "looks-right", id: one }))), + ); + expect(after.head.alreadyAnswered).toBe(3); + expect(after.head.population).toBe(after.head.pool + 3); + }); + + it("stops rather than build a sitting of marks that were all answered", () => { + const all = build("--set", "fallback", "--count", "1000"); + const p = answersAt("a5.json", all.ids.map((one) => ({ kind: "looks-right", id: one }))); + expect(() => build("--set", "fallback", "--count", "1000", "--answered", p)).toThrow(); + }); +}); + +/** + * Answers survive a reload; for one sitting the drawn rectangle did not, and neither + * did the reader's place. Twenty-five carefully nudged marks opened again at the + * rectangle they shipped with, on card one, with nothing on the page saying the + * corrections were still held — so the only reasonable thing to do was the work + * again, and a second pass appends rather than replaces. + * + * The page's own load path cannot be exercised here: it wants a document and a store, + * and this file builds HTML with node. So these read the script the builder emits and + * hold it to the two things that were missing. It is a coarse test for a defect that + * cost an evening, and a coarse test that would have caught it is worth having. + */ +describe.skipIf(!haveCorpus)("giving a reader back the sitting they left", () => { + it("puts every standing answer back on its rectangle before the first draw", () => { + const { html } = build("--set", "fallback", "--count", "20"); + const replay = html.indexOf("for (const id of new Set(said.map("); + expect(replay).toBeGreaterThan(-1); + // Before the first draw, or the reader still sees one untouched page and has no + // way to know a second one is coming. + expect(replay).toBeLessThan(html.lastIndexOf("\nrender();")); + }); + + it("keeps where the reader was standing, not only how far they got", () => { + const { html } = build("--set", "fallback", "--count", "20"); + // Two different questions, and only the high-water mark used to survive. + expect(html).toContain("keepAt(at);"); + expect(html).toContain("Math.min(keptAt(), DECK.length - 1)"); + }); + + /** + * A reader banked an hour of answers, watched the count under the card stay where + * it was, banked again, and watched it stay again — and said so twice. Nothing was + * lost either time; the deal is fixed when the page is built and the count was of + * the deal. But an instrument that cannot show a reader their own work is one they + * stop trusting, and it takes forty hours of theirs on trust. + * + * So the two lists are separate now, and these are the three things that have to + * stay true about them together. + */ + it("takes the handed-over marks off the deck and leaves the transcript whole", () => { + const { html } = build("--set", "fallback", "--count", "20"); + // The deal is still the deal, and what is left is derived from it. + expect(html).toContain("let DECK = CARDS.filter(function (c) { return !GONE.has(c.id); });"); + // Retiring is what handing over does, on every path a file leaves the page by. + expect(html).toContain("const went = retire();"); + // It survives a reload, or the retired marks come back and the reader is told + // their work did not count after all — the same failure, one refresh later. + expect(html).toContain("keepGone([...GONE]);"); + }); + + it("never lets a shrinking deck shrink the file that gets written", () => { + const { html } = build("--set", "fallback", "--count", "20"); + // Every hand-over writes the whole transcript under one name, so a later write + // that had lost answers would silently destroy an earlier one that had them. + // That is why what comes back from the session is filtered against the deal. + expect(html).toContain("const mine = new Set(CARDS.map(function (c) { return c.id; }));"); + // And why retiring touches the deck and the place, and nothing else. + const retire = html.slice(html.indexOf("function retire()"), html.indexOf("function handOver(")); + expect(retire).not.toContain("said ="); + expect(retire).not.toContain("keep(said)"); + }); + + /** + * The rest of this file reads the emitted script as text. This one runs it, because + * the thing that can be wrong here is arithmetic on two indices that shift under + * each other — where the reader is standing, and how far they had got — and no + * amount of string-matching can tell you a subtraction is right. + * + * Five marks, and the reader is placed somewhere different in each: the point is + * that the position follows the card the reader was on rather than being reset, and + * that a mark passed in silence stays owed while an answered one goes. + */ + it("moves the reader with the deck instead of resetting them", () => { + const { html } = build("--set", "fallback", "--count", "20"); + const src = html.slice(html.indexOf("function retire()"), html.indexOf("function handOver(")); + const noop = () => {}; + const run = (deck, at, seen, answers) => + new Function( + "DECK", "GONE", "at", "seen", "said", "keepGone", "keepSeen", "keepAt", "render", + `${src}\nconst w = retire(); return { went: w, left: DECK.map((c) => c.id), at, seen };`, + )(deck.map((id) => ({ id })), new Set(), at, seen, answers.map((id) => ({ id })), noop, noop, noop, noop); + const all = ["a", "b", "c", "d", "e"]; + + // The reported case: one answered, standing on the next. It becomes card one of + // four, and none of the four has been looked at — which is the truth. + expect(run(all, 1, 1, ["a"])).toEqual({ went: 1, left: ["b", "c", "d", "e"], at: 0, seen: 0 }); + // Two go from either side of where they stand; they stay on the same card, and + // the one they passed in silence is still counted as looked at and still owed. + expect(run(all, 3, 3, ["a", "c"])).toEqual({ went: 2, left: ["b", "d", "e"], at: 1, seen: 1 }); + // Nothing survives at or after where they stood, so the sitting is over. Past the + // end rather than wrapped to the start: there is genuinely no next card. + expect(run(all, 3, 4, ["d", "e"])).toEqual({ went: 2, left: ["a", "b", "c"], at: 3, seen: 3 }); + // Handing over an empty sitting, and handing the same one over twice, both do + // nothing — which is what lets the button stay pressable without a reader having + // to remember whether they already pressed it. + expect(run(all, 2, 2, [])).toEqual({ went: 0, left: all, at: 2, seen: 2 }); + expect(run(["b", "c"], 0, 0, ["a"])).toEqual({ went: 0, left: ["b", "c"], at: 0, seen: 0 }); + }); +}); + +/** + * A crop of print holds several marks, often several of the same name, and a page + * that says only "fatha" has not asked a question anybody can answer — the reader + * is shown a rectangle and left to guess which of four fathas it was supposed to be + * on. The letters are the answer, and the rank settles it when the letters alone + * cannot. This is what the person sitting the first fallback page said was missing, + * so it is checked here rather than left to be noticed again. + */ +describe.skipIf(!haveCorpus)("saying which mark is being asked about", () => { + it("names the letters the print drew each mark on", () => { + const got = build("--set", "fallback", "--count", "20"); + expect(got.cards.length).toBeGreaterThan(0); + for (const c of got.cards) { + // One or the other, always: the letters when the print drew the mark inside a + // ligature, and the word when it drew it under a word but on no letter of its + // own. A card with neither names nothing and is the failure this test exists + // for; a card with both is normal. + expect(c.on || c.word).toBeTruthy(); + } + }); + + it("tells two marks of one name on one ligature apart", () => { + const got = build("--set", "fallback", "--count", "20"); + for (const c of got.cards) { + expect(c.of).toBeGreaterThanOrEqual(1); + expect(c.nth).toBeGreaterThanOrEqual(1); + // The rank has to be sayable: "the fourth of two" is worse than saying nothing, + // because a reader who counts will conclude the page is broken and stop. + expect(c.nth).toBeLessThanOrEqual(c.of); + } + }); + + it("puts the mark under discussion in the middle of both windows", () => { + // The row's rectangle and the drawn one are a correction apart, and framing on + // the first while drawing the second walks the mark off toward the edge — + // furthest exactly when the correction is largest, which is when a reader most + // needs to see it. Looking closer magnified that rather than fixing it. + const got = build("--set", "fallback", "--count", "20"); + for (const c of got.cards) { + const cx = c.at[0] + c.at[2] / 2; + const cy = c.at[1] + c.at[3] / 2; + for (const vb of [c.vb, c.near]) { + expect(Math.abs(vb[0] + vb[2] / 2 - cx)).toBeLessThan(0.02); + expect(Math.abs(vb[1] + vb[3] / 2 - cy)).toBeLessThan(0.02); + } + } + }); + + it("keeps the drawn rectangle a correction away from the row's own", () => { + // The guard on the test above: if `at` ever collapsed onto the row's rectangle, + // the centring assertion would pass trivially and stop meaning anything. These + // are the refused marks, so every one of them carries a real correction. + const got = build("--set", "fallback", "--count", "20"); + const moved = got.cards.filter((c) => Math.abs(c.at[0] - c.box[0]) + Math.abs(c.at[1] - c.box[1]) > 0.01); + expect(moved.length).toBe(got.cards.length); + }); + + it("refuses to caption a row it cannot match to the print", () => { + // The index walks the print in document order, so rows built from a different + // revision line up off by one and every card is captioned with its neighbour's + // letters — silently, and in the direction that makes a correct placement look + // wrong. Better to stop than to mislead a whole sitting. + const bad = join(dir, "rows-mismatched.json"); + writeFileSync(bad, JSON.stringify(ROWS.map((r) => ({ ...r, name: "no-such-mark" })))); + expect(() => + execFileSync(process.execPath, [SCRIPT, "--rows", bad, "--out", join(dir, "bad.html"), "--set", "fallback"], { + encoding: "utf8", + stdio: "pipe", + }), + ).toThrow(); + }); +}); + +/** + * The page is a JavaScript template literal, all of it — style, markup and script. + * + * A backtick anywhere inside it ends the string early, and the file then fails to + * parse somewhere else entirely, with a message about the wrong line. It has broken + * this file three times, twice inside a comment that was never meant to be code. + * Two assertions cost nothing and say the trap out loud to whoever trips it next. + */ +describe.skipIf(!haveCorpus)("the trap the page is built in", () => { + it("emits no backtick and no interpolation of its own", () => { + const { html } = build("--set", "fallback", "--count", "5"); + expect(html).not.toContain("`"); + expect(html).not.toContain("${"); + }); +}); + +/** + * Nothing drawn on the paper is themed. + * + * The crop is a photograph of print and stays white in both themes on purpose, but + * the two rectangle colours were in the re-themed set, so dark mode lightened them + * and then drew them on that unchanged white: 5.05:1 fell to 2.49:1 and 4.89:1 to + * 1.70:1. That is not invisible — it is worse than invisible, because a reader can + * still make out something rectangular, believe they have looked, and affirm. The + * finding runs in the direction that reads as success, which is why it gets a test + * rather than a comment. + */ +const NEVER_THEMED = ["paper", "ink", "ours-line", "ours-wash", "yours-line", "yours-wash"]; + +const channels = (hex) => + [1, 3, 5].map((i) => { + const c = parseInt(hex.slice(i, i + 2), 16) / 255; + return c <= 0.03928 ? c / 12.92 : ((c + 0.055) / 1.055) ** 2.4; + }); +const luminance = (hex) => { + const [r, g, b] = channels(hex); + return 0.2126 * r + 0.7152 * g + 0.0722 * b; +}; +const contrast = (a, b) => { + const [hi, lo] = [luminance(a), luminance(b)].sort((x, y) => y - x); + return (hi + 0.05) / (lo + 0.05); +}; + +describe.skipIf(!haveCorpus)("keeping the rectangles legible on paper that never changes", () => { + const tokensIn = (css) => { + const found = {}; + for (const m of css.matchAll(/--([a-z-]+):\s*(#[0-9a-f]{6,8})/gi)) found[m[1]] = m[2]; + return found; + }; + const sheets = () => { + const { html } = build("--set", "fallback", "--count", "5"); + return { + light: tokensIn(/:root \{([\s\S]*?)\n\}/.exec(html)[1]), + // Lazy to the first close at column zero — the selector inside it is indented, + // so its own brace cannot end the match early. + dark: tokensIn(/@media \(prefers-color-scheme: dark\) \{([\s\S]*?)\n\}/.exec(html)[1]), + }; + }; + + it("re-themes nothing that is drawn on the crop", () => { + const { light, dark } = sheets(); + // Every one of them defined once, in the theme-independent block... + for (const name of NEVER_THEMED) expect(light[name]).toMatch(/^#[0-9a-f]{6,8}$/i); + // ...and named nowhere in the dark one. This is the whole invariant. + for (const name of NEVER_THEMED) expect(dark[name]).toBeUndefined(); + }); + + it("still re-themes the chrome, which is the reason the two sets exist", () => { + // The guard on the test above. If the dark block ever emptied out, that test + // would pass forever while the page stopped having a dark theme at all. + const { dark } = sheets(); + expect(Object.keys(dark).length).toBeGreaterThan(0); + for (const name of ["field", "text", "ours", "yours"]) expect(dark[name]).toBeTruthy(); + }); + + it("draws both rectangles at 3:1 or better against the paper", () => { + const { light } = sheets(); + // 3:1 is the threshold for a graphical object rather than for text, and these + // are graphical objects: a hairline rectangle around a mark, not a label. + expect(contrast(light["ours-line"], light.paper)).toBeGreaterThanOrEqual(3); + expect(contrast(light["yours-line"], light.paper)).toBeGreaterThanOrEqual(3); + }); + + it("tells the two rectangles apart by something other than hue", () => { + // Ours and the reader's are orange and green, which is the one pair a red-green + // reader cannot separate. The dash carries the distinction on its own, and it + // survives anybody re-picking the palette later. + const { html } = build("--set", "fallback", "--count", "5"); + expect(html).toMatch(/rect\.mine \{ stroke-dasharray:/); + }); +}); + +/** + * The crop is parsed once per card, not once per pointer frame. + * + * The drag handler used to rebuild the stage from the card's path data on every + * `pointermove` — 2.0 KB on the smallest crop, 23.2 KB on the largest. A correction + * that stutters is one the reader gives up on and affirms instead, which turns a + * performance defect into a wrong answer in the transcript. + * + * This reads the emitted script rather than running it, in the same spirit as the + * replay tests above: coarse, and it catches the regression that matters. + */ +describe.skipIf(!haveCorpus)("parsing the crop once per card", () => { + it("never rebuilds the stage while drawing a frame", () => { + const { html } = build("--set", "fallback", "--count", "5"); + const paint = /\nfunction paint\(\) \{\n([\s\S]*?)\n\}\n/.exec(html); + expect(paint).toBeTruthy(); + expect(paint[1]).not.toContain("innerHTML"); + }); + + it("writes the crop into the page in exactly one place, and that place is mount", () => { + const { html } = build("--set", "fallback", "--count", "5"); + const script = html.slice(html.indexOf("function mount(")); + expect([...script.matchAll(/c\.svg/g)].length).toBe(1); + const mount = /\nfunction mount\(c\) \{\n([\s\S]*?)\n\}\n/.exec(html); + expect(mount[1]).toContain("c.svg"); + }); + + it("measures the rectangle's line in screen pixels, not in page units", () => { + // The old stroke width was the viewBox over 220, which is 1.64px only while the + // stage is exactly as wide as the column. Anything that sizes the stage + // differently — a dock, a landscape phone — makes line weight a function of each + // card's shape, and the thinnest cards are the ones hardest to judge. + const { html } = build("--set", "fallback", "--count", "5"); + expect(html).toContain('vector-effect="non-scaling-stroke"'); + expect(html).toMatch(/rect\.grab, svg\.stage rect\.mine \{ stroke-width:/); + }); +}); + +/** + * Saying which mark, in words a reader can check against the print. + * + * The person who sat the first fallback page said they could not tell which of four + * fathas the rectangle was supposed to be on. The card has always carried the word + * the mark sits in and thrown it away whenever it also had the letters, which is the + * common case; and the sentence under the letters named the mark in only one of its + * three branches, so a reader was told "the second of four on these letters" without + * ever being told the second *what*. + */ +describe.skipIf(!haveCorpus)("naming the mark and the word it sits in", () => { + it("carries a longer containing word on every card that names single letters", () => { + // This is a fact about the data, and it is what makes the fix possible: if the + // word were only ever the letters again, appending it would add nothing and this + // would be the test that noticed. + const got = build("--set", "fallback", "--count", "40"); + const single = got.cards.filter((c) => c.on && [...c.on].length === 1); + expect(single.length).toBeGreaterThan(0); + for (const c of single) { + expect(c.word).toBeTruthy(); + expect([...c.word].length).toBeGreaterThan([...c.on].length); + } + }); + + it("says the mark's name whichever of the three things it has to say", () => { + const { html } = build("--set", "fallback", "--count", "5"); + const identify = /\nfunction identify\(([\s\S]*?)\n\}\n/.exec(html); + expect(identify).toBeTruthy(); + // No letters at all · one of several · the only one. Three branches, three names. + expect([...identify[1].matchAll(/c\.name/g)].length).toBeGreaterThanOrEqual(3); + }); + + it("puts the word on the card in its own direction", () => { + const { html } = build("--set", "fallback", "--count", "5"); + expect(html).toMatch(/\.ident \.word \{ direction: rtl;/); + expect(html).toContain('w.setAttribute("lang", "ar")'); + }); +}); + +/** + * The instructions get out of the way without making any answer cost more. + * + * The explanation was about 275px tall and never went away, which put the affirm + * button below the fold on a 393px phone for the whole sitting. It cannot simply be + * deleted — the first card needs it — and the fault buttons must not go behind a + * disclosure either, because adding a tap to reporting a fault while affirming stays + * free biases the exact ratio the sitting exists to measure. So: two ledes and a + * quiet toggle, and the short one appears once the reader has demonstrably read the + * long one. + */ +describe.skipIf(!haveCorpus)("getting the explanation off the fold", () => { + it("carries both a short lede and the full one", () => { + const { html } = build("--set", "fallback", "--count", "5"); + expect(html).toMatch(/id="brief"/); + expect(html).toMatch(/id="full"/); + expect(html).toMatch(/id="ledeSwap"/); + }); + + it("remembers which one the reader asked for, and lets them ask for either", () => { + const { html } = build("--set", "fallback", "--count", "5"); + // Three states, not two. A boolean derived from progress alone would re-collapse + // the long version on the very next answer, one tap after the reader asked for it. + expect(html).toContain("KEY + \"-read\""); + expect(html).toMatch(/chose === null \? seen > 0 : chose === "1"/); + }); + + it("keeps the number of taps an answer costs where it was", () => { + // The whole design constraint in one assertion. Shortening the explanation must + // not be paid for by hiding the fault buttons: a fault that costs a tap more than + // an affirmation biases the one ratio these sittings exist to measure, and it + // biases it toward agreement. All five stay on the card, none behind a disclosure. + const { html } = build("--set", "fallback", "--count", "5"); + const acts = /
    ([\s\S]*?)<\/div>/.exec(html); + expect(acts).toBeTruthy(); + for (const id of ["off", "shape", "draw", "print", "skip"]) { + expect(acts[1]).toContain(`id="${id}"`); + } + expect(html).not.toMatch(/
    { + const s = [...xs].sort((a, b) => a - b); + return s.length ? s[Math.floor(s.length / 2)] : 0; +}; + +/** + * Where along its line a mark sits. Across only: justification stretches a line + * along its own direction, so that is the axis a within-line tilt is fitted in. + */ +const centreX = (r) => r.box[0] + r.box[2] / 2; + +export const pageKey = (r) => r.page; +export const lineKey = (r) => `${r.page}:${r.line}`; + +/** + * The straight line through a set of points, and how far the points sit from it. + * + * One copy, because several things here are the same regression asked at + * different grains — what transform the ink implies for a whole page, what + * stretch it implies along one printed line — and a second copy of a + * least-squares fit is a second place for them to quietly disagree. + */ +export function fitLine(obs, want) { + const n = obs.length; + const mx = obs.reduce((a, b) => a + b, 0) / n; + const my = want.reduce((a, b) => a + b, 0) / n; + let sxy = 0; + let sxx = 0; + for (let i = 0; i < n; i += 1) { + sxy += (obs[i] - mx) * (want[i] - my); + sxx += (obs[i] - mx) * (obs[i] - mx); + } + const a = sxx > 0 ? sxy / sxx : 1; + const b = my - a * mx; + let e = 0; + for (let i = 0; i < n; i += 1) { + const d = want[i] - (a * obs[i] + b); + e += d * d; + } + return { a, b, sd: Math.sqrt(e / Math.max(1, n - 2)) }; +} + +/** Every group a keying function names, with its rows, ink-empty rows dropped. */ +export function groupBy(rs, key) { + const g = new Map(); + for (const r of rs) { + if (r.ink < EMPTY) continue; + const k = key(r); + if (k === null || k === undefined || (typeof k === "number" && !Number.isFinite(k))) continue; + if (typeof k === "string" && k.endsWith(":NaN")) continue; + if (!g.has(k)) g.set(k, []); + g.get(k).push(r); + } + return g; +} + +/** + * The displacement of every group, as the median of the group's marks. + * + * The median rather than the mean, because the thing being estimated is what the + * group as a whole does, and a handful of marks that really are on the wrong + * letter would drag a mean. Groups too small to estimate anything are left out + * rather than corrected by a number built from four observations. + */ +export function shiftsBy(rs, key, floor = FLOOR) { + const out = new Map(); + for (const [k, xs] of groupBy(rs, key)) { + if (xs.length < floor) continue; + out.set(k, { dx: median(xs.map((r) => r.dx)), dy: median(xs.map((r) => r.dy)), n: xs.length }); + } + return out; +} + +/** + * Build the correction a grain would apply, fitted on `train`. + * + * Finer grains are layered on top of the page rather than replacing it. A line + * with too few marks to estimate anything then falls back to its page's + * displacement instead of to nothing — the difference between a sparse line + * being left alone and a sparse line being made worse. + * + * Returns a function from a row to `{dx, dy}`, plus the tables it was built + * from, so a caller can report how many groups actually earned a correction. + */ +export function correctionFor(grain, train, floor = FLOOR) { + if (!GRAINS.includes(grain)) throw new Error(`unknown grain: ${grain}`); + const page = shiftsBy(train, pageKey, floor); + const after = train.map((r) => { + const p = page.get(r.page) ?? { dx: 0, dy: 0 }; + return { ...r, dx: r.dx - p.dx, dy: r.dy - p.dy }; + }); + + let finer = () => ({ dx: 0, dy: 0 }); + let line = new Map(); + if (grain === "line") { + line = shiftsBy(after, lineKey, floor); + finer = (r) => line.get(lineKey(r)) ?? { dx: 0, dy: 0 }; + } else if (grain === "line-tilt") { + for (const [k, xs] of groupBy(after, lineKey)) { + if (xs.length < floor) continue; + const cx = xs.map(centreX); + line.set(k, { x: fitLine(cx, xs.map((r) => r.dx)), y: fitLine(cx, xs.map((r) => r.dy)) }); + } + finer = (r) => { + const l = line.get(lineKey(r)); + if (!l) return { dx: 0, dy: 0 }; + const cx = centreX(r); + return { dx: l.x.a * cx + l.x.b, dy: l.y.a * cx + l.y.b }; + }; + } + + const apply = (r) => { + const p = page.get(r.page); + if (!p) return { dx: 0, dy: 0 }; + const f = finer(r); + return { dx: p.dx + f.dx, dy: p.dy + f.dy }; + }; + return { apply, pages: page, lines: line }; +} + +/** + * The same grain, with every group wearing some *other* group's correction. + * + * This is the control that separates a grain from a shape of noise. If the + * per-line numbers are real facts about how the two prints set that line, then + * wearing the wrong line's is worse than wearing none at all. If they are noise + * the model absorbed, the shuffle makes no odds — and that is the answer, at the + * cost of one rotation. It rotates rather than randomises so the control is the + * same on every re-run. + */ +export function shuffledCorrectionFor(grain, train, floor = FLOOR) { + const built = correctionFor(grain, train, floor); + if (grain === "page") return built; + const keys = [...built.lines.keys()]; + if (keys.length < 2) return built; + const swap = new Map(keys.map((k, i) => [k, keys[(i + 7) % keys.length]])); + const apply = (r) => { + const p = built.pages.get(r.page); + if (!p) return { dx: 0, dy: 0 }; + const l = built.lines.get(swap.get(lineKey(r))); + if (!l) return { dx: p.dx, dy: p.dy }; + if (grain === "line") return { dx: p.dx + l.dx, dy: p.dy + l.dy }; + const cx = centreX(r); + return { dx: p.dx + l.x.a * cx + l.x.b, dy: p.dy + l.y.a * cx + l.y.b }; + }; + return { ...built, apply }; +} + +/** + * A reproducible coin, so a split half is the same split on every re-run. + * + * Keyed on the mark's own identity rather than on its position in a list: the + * sampler that produced the list is seeded, but a run over a different page set + * would otherwise put a different half of each page in training and nothing + * would say so. + */ +export function half(r) { + let h = 2166136261 ^ Math.imul(r.page, 1000003) ^ r.k; + h = Math.imul(h ^ (h >>> 15), 2246822507); + h = Math.imul(h ^ (h >>> 13), 3266489909); + return ((h ^ (h >>> 16)) >>> 0) % 2; +} + +/** + * What is left after a correction, in page units. + * + * `over` counts marks landing further than `off` from their own ink. That + * threshold is the one the rest of this work reports against; it is a + * "badly out" test, not a quality bar. + */ +export function residualsUnder(rs, apply, off = 0.75) { + const ex = []; + const ey = []; + const d = []; + for (const r of rs) { + if (r.ink < EMPTY) continue; + const c = apply(r); + const a = r.dx - c.dx; + const b = r.dy - c.dy; + ex.push(a); + ey.push(b); + d.push(Math.hypot(a, b)); + } + const sorted = [...d].sort((a, b) => a - b); + const at = (p) => (sorted.length ? sorted[Math.min(sorted.length - 1, Math.floor(p * sorted.length))] : 0); + const spread = (xs) => { + const m = xs.reduce((a, b) => a + b, 0) / Math.max(1, xs.length); + return Math.sqrt(xs.reduce((a, b) => a + (b - m) ** 2, 0) / Math.max(1, xs.length - 1)); + }; + return { + n: d.length, + med: at(0.5), + p95: at(0.95), + over: d.length ? (100 * d.filter((x) => x > off).length) / d.length : 0, + sdx: spread(ex), + sdy: spread(ey), + }; +} + +/** + * A grain, fitted on one half of the marks and scored on the other. + * + * This is the whole defence against the circularity the rest of this work is + * built to avoid: the correction is fitted to maximise agreement with the ink + * and then scored by agreement with the ink, so a richer model improves every + * time — including when what it is fitting is noise. It cannot improve on marks + * it was never shown. `trained` is returned beside `heldOut` on purpose: the gap + * between them is the overfit, and a rung that only wins on its own training + * marks has earned nothing. + */ +export function splitHalfLadder(rows, grain, { floor = SPLIT_FLOOR, shuffled = false } = {}) { + const a = rows.filter((r) => half(r) === 0); + const b = rows.filter((r) => half(r) === 1); + const build = shuffled ? shuffledCorrectionFor : correctionFor; + const { apply, pages, lines } = build(grain, a, floor); + return { + grain, + shuffled, + groups: { pages: pages.size, lines: lines.size }, + heldOut: residualsUnder(b, apply), + trained: residualsUnder(a, apply), + }; +} diff --git a/packages/etl/scripts/lib/registration-grain.test.mjs b/packages/etl/scripts/lib/registration-grain.test.mjs new file mode 100644 index 0000000..0e3689b --- /dev/null +++ b/packages/etl/scripts/lib/registration-grain.test.mjs @@ -0,0 +1,335 @@ +/** + * The grain search, on prints whose true difference we built ourselves. + * + * The failure this has to catch is the one the whole method is arranged around: + * a correction fitted to agree with the ink and then scored by agreement with + * the ink gets better every time it is given more parameters, *including* when + * the thing it is fitting is noise. Read on the marks it was fitted to, a + * per-line model always looks like progress. So the fixtures come in a pair — + * one page set where the lines really do differ, and one where they differ by + * nothing at all — and the second must **appear** to improve and then lose on + * the marks it never saw. A version of this file where both fixtures pass on + * trained-on numbers alone is a version that would have let us ship noise. + * + * Everything here is deterministic. A fixture that wobbles between runs cannot + * be argued from months later, which is the only reason these numbers exist. + */ +import { describe, expect, it } from "vitest"; +import { + FLOOR, + SPLIT_FLOOR, + correctionFor, + fitLine, + half, + lineKey, + residualsUnder, + shiftsBy, + shuffledCorrectionFor, + splitHalfLadder, +} from "./registration-grain.mjs"; + +/** A repeatable wobble in [-1, 1), so a fixture is the same fixture every run. */ +function wobble(seed) { + let s = (seed * 1103515245 + 12345) >>> 0; + return () => { + s = (Math.imul(s, 1103515245) + 12345) >>> 0; + return (s >>> 8) / 8388608 - 1; + }; +} + +/** + * A synthetic corpus: `pages` pages, 15 printed lines each, `per` marks a line. + * + * `truth(page, line, cx)` is the displacement between the two prints at that + * spot — the thing a correction is trying to recover. On top of it sits a + * quarter of a unit of wobble, which stands in for everything the model is not + * entitled to reach: where inside its letter a mark was set, and how well the + * ink was found under it. + */ +function corpus(truth, { pages = 8, per = 60, noise = 0.25, seed = 7 } = {}) { + const rnd = wobble(seed); + const rows = []; + let k = 0; + for (let p = 1; p <= pages; p += 1) { + for (let line = 1; line <= 15; line += 1) { + for (let i = 0; i < per; i += 1) { + const cx = 20 + (i / per) * 300; + const t = truth(p, line, cx); + rows.push({ + page: p, + k: k += 1, + line, + box: [cx - 2.8, 100 + line * 30, 5.6, 3.6], + ink: 0.4, + dx: t.dx + rnd() * noise, + dy: t.dy + rnd() * noise, + }); + } + } + } + return rows; +} + +/** The whole difference is the page's: what today's four-number fit can reach. */ +const PAGE_ONLY = (p) => ({ dx: 0.5 + p * 0.03, dy: -1.1 - p * 0.02 }); + +/** Each printed line slides on its own, as two prints justifying differently do. */ +const PER_LINE = (p, line) => ({ + dx: PAGE_ONLY(p).dx + (line % 5) * 0.18 - 0.36, + dy: PAGE_ONLY(p).dy + (line % 3) * 0.22 - 0.22, +}); + +/** The same, plus error accumulating along each line and resetting at the next. */ +const PER_LINE_TILT = (p, line, cx) => { + const b = PER_LINE(p, line); + return { dx: b.dx + (cx - 170) * 0.004, dy: b.dy + (cx - 170) * 0.0012 }; +}; + +describe("fitLine", () => { + it("recovers a slope and an intercept it was given", () => { + const xs = Array.from({ length: 30 }, (_, i) => i * 0.5); + const out = fitLine(xs, xs.map((x) => 0.4 + 0.03 * x)); + expect(out.a).toBeCloseTo(0.03, 10); + expect(out.b).toBeCloseTo(0.4, 10); + expect(out.sd).toBeCloseTo(0, 10); + }); + + it("reports how far the points sit from the line it drew", () => { + const xs = Array.from({ length: 40 }, (_, i) => i); + const out = fitLine(xs, xs.map((x, i) => x * 0.1 + (i % 2 ? 0.2 : -0.2))); + expect(out.a).toBeCloseTo(0.1, 2); + expect(out.sd).toBeCloseTo(0.2, 1); + }); +}); + +describe("shiftsBy", () => { + const rows = corpus(PAGE_ONLY, { pages: 3, per: 40 }); + + it("recovers each page's displacement", () => { + const s = shiftsBy(rows, (r) => r.page); + expect(s.size).toBe(3); + expect(s.get(2).dx).toBeCloseTo(PAGE_ONLY(2).dx, 1); + expect(s.get(2).dy).toBeCloseTo(PAGE_ONLY(2).dy, 1); + }); + + it("leaves out any group too small to estimate anything", () => { + const thin = rows.filter((r) => r.page === 1 || r.k % 97 === 0); + const s = shiftsBy(thin, (r) => r.page); + expect(s.has(1)).toBe(true); + expect(s.has(3)).toBe(false); + }); + + it("ignores marks with no ink under them, which measure nothing", () => { + const blanked = rows.map((r) => (r.page === 3 ? { ...r, ink: 0 } : r)); + expect(shiftsBy(blanked, (r) => r.page).has(3)).toBe(false); + }); + + it("is not dragged by a handful of marks that really are on the wrong letter", () => { + const withStrays = rows.map((r, i) => (i % 20 === 0 ? { ...r, dx: r.dx + 40 } : r)); + const s = shiftsBy(withStrays, (r) => r.page); + expect(s.get(1).dx).toBeCloseTo(PAGE_ONLY(1).dx, 1); + }); +}); + +describe("correctionFor", () => { + it("refuses a grain it does not know", () => { + expect(() => correctionFor("word", corpus(PAGE_ONLY))).toThrow(/unknown grain/); + }); + + it("recovers a per-line difference the page grain cannot reach", () => { + const rows = corpus(PER_LINE); + const { apply } = correctionFor("line", rows); + const r = rows.find((x) => x.page === 4 && x.line === 7); + expect(apply(r).dx).toBeCloseTo(PER_LINE(4, 7).dx, 1); + expect(apply(r).dy).toBeCloseTo(PER_LINE(4, 7).dy, 1); + }); + + it("recovers the tilt along a line, not just its offset", () => { + const rows = corpus(PER_LINE_TILT); + const { apply } = correctionFor("line-tilt", rows); + const one = rows.filter((r) => r.page === 3 && r.line === 9); + // Read at both ends of the line as well as the middle: a model that got the + // offset right and the slope wrong is correct in the middle and nowhere else. + // A tenth of a unit is the bar because that is what a line fitted from sixty + // marks, each carrying a quarter of a unit of wobble, is worth at its ends — + // and it is an order of magnitude below the error being corrected. + for (const r of [one[0], one[Math.floor(one.length / 2)], one[one.length - 1]]) { + expect(apply(r).dx - PER_LINE_TILT(3, 9, r.box[0] + 2.8).dx).toBeLessThan(0.1); + expect(apply(r).dx - PER_LINE_TILT(3, 9, r.box[0] + 2.8).dx).toBeGreaterThan(-0.1); + } + }); + + it("leaves a line with too few marks wearing its page's correction, not none", () => { + const rows = corpus(PER_LINE).filter((r) => !(r.page === 2 && r.line === 5 && r.k % 7 !== 0)); + const { apply, lines } = correctionFor("line", rows); + expect(lines.has(lineKey({ page: 2, line: 5 }))).toBe(false); + const sparse = rows.find((r) => r.page === 2 && r.line === 5); + expect(apply(sparse).dx).toBeCloseTo(PAGE_ONLY(2).dx, 1); + }); + + it("gives a page it never measured no correction at all, rather than a guess", () => { + const { apply } = correctionFor("page", corpus(PAGE_ONLY, { pages: 2 })); + expect(apply({ page: 99, line: 3, box: [0, 0, 5.6, 3.6], ink: 0.4, dx: 1, dy: 1 })).toEqual({ + dx: 0, + dy: 0, + }); + }); +}); + +describe("the split half", () => { + it("puts a mark on the same side on every run, and both sides get some", () => { + const rows = corpus(PAGE_ONLY, { pages: 2, per: 40 }); + const first = rows.map(half); + expect(rows.map(half)).toEqual(first); + const ones = first.filter((x) => x === 1).length; + expect(ones).toBeGreaterThan(rows.length * 0.35); + expect(ones).toBeLessThan(rows.length * 0.65); + }); +}); + +/** + * The pair the whole file exists for. + * + * Both fixtures are handed the same ladder. The one whose lines really differ + * must win on marks it never saw; the one whose lines differ by nothing must + * win only on the marks it was fitted to. Nothing else distinguishes them — + * same page displacements, same wobble, same counts. + */ +describe("a per-line correction, against a corpus that has per-line structure and one that does not", () => { + const real = corpus(PER_LINE); + const flat = corpus(PAGE_ONLY); + const ladder = (rows, grain, shuffled = false) => splitHalfLadder(rows, grain, { shuffled }); + + it("pays on held-out marks when the lines genuinely differ", () => { + const page = ladder(real, "page"); + const line = ladder(real, "line"); + expect(line.heldOut.sdx).toBeLessThan(page.heldOut.sdx * 0.75); + expect(line.heldOut.sdy).toBeLessThan(page.heldOut.sdy * 0.85); + expect(line.heldOut.over).toBeLessThan(page.heldOut.over); + }); + + it("appears to pay and then loses, when the lines differ by nothing", () => { + const page = ladder(flat, "page"); + const line = ladder(flat, "line"); + // On its own training marks it fits the wobble, so it looks like progress. + expect(line.trained.sdx).toBeLessThan(page.trained.sdx); + expect(line.trained.sdy).toBeLessThan(page.trained.sdy); + // On marks it never saw, that same fitted wobble is added noise. + expect(line.heldOut.sdx).toBeGreaterThan(page.heldOut.sdx); + expect(line.heldOut.sdy).toBeGreaterThan(page.heldOut.sdy); + }); + + it("fits the wobble harder still when given a slope, and still loses", () => { + const page = ladder(flat, "page"); + const line = ladder(flat, "line"); + const tilt = ladder(flat, "line-tilt"); + expect(tilt.trained.sdx).toBeLessThan(line.trained.sdx); + expect(tilt.heldOut.sdx).toBeGreaterThan(page.heldOut.sdx); + }); + + it("keeps improving on held-out marks when there is a real tilt to find", () => { + const rows = corpus(PER_LINE_TILT); + const line = ladder(rows, "line"); + const tilt = ladder(rows, "line-tilt"); + expect(tilt.heldOut.sdx).toBeLessThan(line.heldOut.sdx); + }); + + it("says how many groups earned a correction, so a win cannot be a win on four lines", () => { + const line = ladder(real, "line"); + expect(line.groups.pages).toBe(8); + expect(line.groups.lines).toBe(8 * 15); + }); +}); + +/** + * The other control, and the one that needs no held-out half at all. + * + * Split-half answers *did this model learn something*. The shuffle answers + * *was there anything at this grain to learn* — because if the per-line numbers + * are facts about how each line was set, wearing a different line's must be + * worse than wearing none. + */ +describe("wearing another line's correction", () => { + it("is worse than no line correction at all, when the lines really differ", () => { + const rows = corpus(PER_LINE); + const page = splitHalfLadder(rows, "page"); + const wrong = splitHalfLadder(rows, "line", { shuffled: true }); + expect(wrong.heldOut.sdx).toBeGreaterThan(page.heldOut.sdx); + expect(wrong.heldOut.over).toBeGreaterThan(page.heldOut.over); + }); + + it("makes almost no odds when the lines differ by nothing", () => { + const rows = corpus(PAGE_ONLY); + const right = splitHalfLadder(rows, "line"); + const wrong = splitHalfLadder(rows, "line", { shuffled: true }); + expect(wrong.heldOut.sdx).toBeCloseTo(right.heldOut.sdx, 1); + }); + + it("leaves the page grain alone, there being nothing to shuffle", () => { + const rows = corpus(PAGE_ONLY, { pages: 3 }); + const a = shuffledCorrectionFor("page", rows); + const b = correctionFor("page", rows); + const r = rows[0]; + expect(a.apply(r)).toEqual(b.apply(r)); + }); +}); + +/** + * The trap the halved floor exists to close, on a corpus the size of a real page. + * + * A printed line carries about thirty-six marks. Split in half for a held-out + * measurement it arrives at the fit with eighteen — under the production floor of + * twenty, so it is refused and falls back to its page. What then gets measured is + * not how well a per-line correction works but how many lines survived the + * halving, and the answer comes back several times weaker than the truth. This + * fixture has real per-line structure and thirty-six marks a line, so the only + * thing separating the two readings below is the floor. + */ +describe("the floor a split-half measurement is held to", () => { + const rows = corpus(PER_LINE, { per: 36 }); + const LINES = 8 * 15; + const at = (floor, grain) => splitHalfLadder(rows, grain, { floor }); + + // Not every line: the split is a coin, so a line of thirty-six lands a training + // half of eighteen on average and occasionally of nine, and one of the hundred + // and twenty here does. That is the floor doing its job rather than failing — + // what matters is that it is one line and not two in three. + // The fixture reproduces the real thing closely: it keeps a bit over a quarter + // of its lines at the production floor, where the corpus kept 635 of 1,723. + it("refuses most lines at the production floor and almost none at the halved one", () => { + expect(at(FLOOR, "line").groups.lines).toBeLessThan(LINES * 0.5); + expect(at(SPLIT_FLOOR, "line").groups.lines).toBeGreaterThan(LINES * 0.95); + }); + + it("reads the grain as weaker than it is when held to the production floor", () => { + const page = at(SPLIT_FLOOR, "page"); + const strict = at(FLOOR, "line"); + const fair = at(SPLIT_FLOOR, "line"); + // Both are honest numbers about *something*; only the second is a number + // about the grain. The refused lines drag the first back toward the page. + expect(fair.heldOut.sdx).toBeLessThan(strict.heldOut.sdx); + expect(strict.heldOut.sdx).toBeCloseTo(page.heldOut.sdx, 1); + }); + + it("is the default, so nobody has to remember it", () => { + expect(splitHalfLadder(rows, "line").groups.lines).toBe(at(SPLIT_FLOOR, "line").groups.lines); + }); +}); + +describe("residualsUnder", () => { + it("measures against the correction applied, and counts what is still badly out", () => { + const rows = corpus(PAGE_ONLY, { pages: 2, per: 40, noise: 0 }); + const none = residualsUnder(rows, () => ({ dx: 0, dy: 0 })); + const { apply } = correctionFor("page", rows); + const fixed = residualsUnder(rows, apply); + expect(none.over).toBe(100); + expect(fixed.over).toBe(0); + expect(fixed.med).toBeLessThan(0.01); + }); + + it("leaves out marks with no ink, which have no displacement to report", () => { + const rows = corpus(PAGE_ONLY, { pages: 1, per: 40 }); + const blanked = rows.map((r, i) => (i % 2 ? { ...r, ink: 0 } : r)); + expect(residualsUnder(blanked, () => ({ dx: 0, dy: 0 })).n).toBe(rows.length / 2); + }); +}); diff --git a/packages/etl/scripts/probe-mark-ink.mjs b/packages/etl/scripts/probe-mark-ink.mjs index d07989a..10857e8 100644 --- a/packages/etl/scripts/probe-mark-ink.mjs +++ b/packages/etl/scripts/probe-mark-ink.mjs @@ -84,6 +84,16 @@ import { readPageInk, integral, rasterise } from "./lib/ink.mjs"; import { cachedMarkPages, markPageFile as pageFile, marksOf } from "./lib/marks.mjs"; import { bestPlacement, outlineRings, rng, sampleSizeFor, scoreAt, stamp, wilson } from "./lib/mark-ink.mjs"; import { classifier, loadExemplars, shapeGroups, similarityMatrix } from "./lib/mark-shape.mjs"; +import { + FLOOR, + GRAINS, + SPLIT_FLOOR, + correctionFor, + fitLine, + half, + pageKey, + shiftsBy, +} from "./lib/registration-grain.mjs"; const HERE = dirname(fileURLToPath(import.meta.url)); const ROOT = join(HERE, "..", "..", ".."); @@ -275,6 +285,7 @@ function scoreMark(mark, ink, lib, other, corr) { surah: mark.surah, aya: mark.aya, idx: mark.idx, + line: mark.line, box: [box[0] + cx0, box[1] + cy0, box[2], box[3]], best, ink: inkFrac, @@ -428,8 +439,15 @@ const skipped = []; * would claim if each page's whole-page displacement were taken out. Running it * twice rather than reasoning about the first pass's numbers is the difference * between predicting the correction would work and showing that it does. + * + * `onlyCorrected` drops any mark the correction has nothing to say about, rather + * than scoring it uncorrected alongside the ones that moved. That is wrong for + * the ordinary run — a handful of pages carry too few marks to be corrected at + * all, and they are part of what ships — and it is the only honest thing to do + * when the correction was deliberately kept away from half the marks, because + * there the uncorrected ones are precisely the training set. */ -function scoreAll(corrections) { +function scoreAll(corrections, onlyCorrected = false) { const out = []; let lastPage = null; let ink = null; @@ -447,6 +465,7 @@ function scoreAll(corrections) { const pool = pageMarks.filter((o) => o.k !== m.k && o.name !== m.name); const other = pool.length ? pool[Math.floor(pick() * pool.length)] : null; const c = corrections?.get(`${m.page}:${m.k}`) ?? null; + if (onlyCorrected && !c) continue; const r = scoreMark(m, ink, lib, other, c); if (r) out.push({ ...r, hafs: m.hafs, d: m.d, fit: m.fit }); else if (!corrections) skipped.push(m); @@ -457,23 +476,60 @@ function scoreAll(corrections) { const raw = scoreAll(null); /** - * Each page's own displacement, as the median of its marks'. + * Which grain the corrected pass is scored at, and the correction it applies. + * + * The page is the default because it is what ships, and because leaving it as + * the default means the corrected numbers in every earlier run still mean the + * same thing. * - * The median rather than the mean, because the thing being estimated is what the - * page as a whole does, and a handful of marks that really are on the wrong - * letter would drag a mean. Pages with too few marks to estimate anything are - * left alone rather than corrected by a number built from four observations. + * The models themselves live in `lib/registration-grain.mjs` rather than here. + * They are arithmetic on displacements the rasteriser has already measured, so a + * search over grains can run in seconds outside this probe — and they are the + * one part of this file whose answer can be checked against a corpus we built + * ourselves, which is what `registration-grain.test.mjs` does. What cannot be + * checked that way, and is why the winning grain still comes back through here, + * is how much ink a rectangle covers once it has moved: overlap is not linear in + * the displacement and cannot be predicted from it. */ -const perPageShift = new Map(); -for (const p of new Set(raw.map((r) => r.page))) { - const rs = raw.filter((r) => r.page === p && r.ink >= 0.02); - if (rs.length < 20) continue; - perPageShift.set(p, { - dx: q(rs.map((r) => r.dx), 0.5), - dy: q(rs.map((r) => r.dy), 0.5), - n: rs.length, - }); +const GRAIN = arg("--grain", "page"); +if (!GRAINS.includes(GRAIN)) { + console.error(`--grain must be ${GRAINS.join(", ")}; got ${GRAIN}`); + process.exit(2); } + +/** + * Whether the correction is allowed to see the marks it is then graded on. + * + * A correction is fitted to make the rectangles agree with the ink and is then + * scored by how well the rectangles agree with the ink, so a model with more + * parameters wins every time — including when what it is fitting is the noise in + * this particular sample. There is exactly one way to tell those apart, and it + * is to grade on marks the model never saw. + * + * Off by default, because the corrected block in every earlier run means "what + * would these same marks look like if we corrected them", and quietly changing + * what a printed number means is worse than making somebody ask for the other + * one. On, it is the number to quote when comparing grains: a finer grain that + * wins here has found something about the print, and a finer grain that wins + * only with this off has found something about these marks. + */ +const SPLIT_HALF = process.argv.includes("--split-half"); + +const perPageShift = shiftsBy(raw, pageKey); +const corrections = new Map(); +/** The fitted correction itself, kept so `--shift-out` can write it down. */ +let fitted; +{ + const train = SPLIT_HALF ? raw.filter((r) => half(r) === 0) : raw; + const graded = SPLIT_HALF ? raw.filter((r) => half(r) === 1) : raw; + fitted = correctionFor(GRAIN, train, SPLIT_HALF ? SPLIT_FLOOR : FLOOR); + for (const r of graded) { + if (!perPageShift.has(r.page)) continue; + const c = fitted.apply(r); + corrections.set(`${r.page}:${r.k}`, { dx: c.dx, dy: c.dy, best: r.best }); + } +} + /** * What the fit between the two prints would have been if it had been fitted to * the ink instead of to the ornaments. @@ -486,38 +542,21 @@ for (const p of new Set(raw.map((r) => r.page))) { * transform from a few hundred observations spread over the whole page rather * than from a handful of ornaments clustered on it. * - * Reported, not applied. It is evidence about the fit, and what to do about the - * fit is a decision — this only has to make the decision possible by saying how - * far off it is and in which direction. + * Reported, not applied. A whole-corpus read of it said the scale term is real + * and worth about two per cent of the scatter it would have to remove, which is + * how the search moved off the four-number family and onto the grain question + * above — so this stays as evidence, and the correction that ships comes from + * `correctionFor`. */ function refit(rs, fit) { - const ls = (obs, want) => { - const n = obs.length; - const mx = obs.reduce((a, b) => a + b, 0) / n; - const my = want.reduce((a, b) => a + b, 0) / n; - let sxy = 0; - let sxx = 0; - for (let i = 0; i < n; i += 1) { - sxy += (obs[i] - mx) * (want[i] - my); - sxx += (obs[i] - mx) * (obs[i] - mx); - } - const a = sxx > 0 ? sxy / sxx : 1; - const b = my - a * mx; - let e = 0; - for (let i = 0; i < n; i += 1) { - const d = want[i] - (a * obs[i] + b); - e += d * d; - } - return { a, b, sd: Math.sqrt(e / Math.max(1, n - 2)) }; - }; // Back out the corpus-frame centre the pinned fit was applied to, then ask // what transform would have carried it to where the ink actually is. const theirsX = rs.map((r) => (r.box[0] + r.box[2] / 2 - fit.tx) / fit.sx); const theirsY = rs.map((r) => (r.box[1] + r.box[3] / 2 - fit.ty) / fit.sy); const oursX = rs.map((r, i) => fit.sx * theirsX[i] + fit.tx + r.dx); const oursY = rs.map((r, i) => fit.sy * theirsY[i] + fit.ty + r.dy); - const fx = ls(theirsX, oursX); - const fy = ls(theirsY, oursY); + const fx = fitLine(theirsX, oursX); + const fy = fitLine(theirsY, oursY); return { n: rs.length, sx: fx.a, @@ -533,18 +572,50 @@ function refit(rs, fit) { }; } -const corrections = new Map(); -for (const r of raw) { - const s = perPageShift.get(r.page); - if (s) corrections.set(`${r.page}:${r.k}`, { dx: s.dx, dy: s.dy, best: r.best }); -} -const fixed = corrections.size ? scoreAll(corrections) : raw; +const fixed = corrections.size ? scoreAll(corrections, SPLIT_HALF) : raw; // Everything below reports the boxes as they are. The corrected pass is // reported alongside it, never instead of it: what ships today is the first // number, and the second is what a change would buy. const rows = raw; +/** + * Every scored mark, one row, for a grain search to be run outside this script. + * + * Searching for the right grain does not need the rasteriser. A correction moves + * the rectangle, and `dx, dy` already says where the ink is relative to where the + * rectangle claims to be — so the residual under any candidate correction is + * arithmetic on these rows, and a search over grains costs seconds instead of a + * pass a rung. The winning grain still comes back through `scoreAll`, which is + * the only thing that reports overlap, because overlap is not linear in the + * displacement and cannot be predicted from it. + * + * Off by default and written only where asked. It carries no text: page, mark, + * printed line, the mark's own name, the rectangle, and numbers. + */ +const rowsOut = arg("--rows-out", null); +if (rowsOut) { + writeFileSync( + rowsOut, + JSON.stringify( + rows.map((r) => ({ + page: r.page, + k: r.k, + line: r.line, + name: r.name, + box: r.box, + ink: r.ink, + dx: r.dx, + dy: r.dy, + iou0: r.iou0, + iouBest: r.iouBest, + phi0: r.phi0, + nullPhi: r.nullPhi, + })), + ), + ); +} + // ------------------------------------------------------------ aggregation -- /** @@ -658,6 +729,10 @@ const report = { ran: new Date().toISOString().slice(0, 10), res: RES, radius: RADIUS, + // Which grain the `corrected` block below was scored at. Without it, two runs + // of this script produce the same field names for two different models. + grain: GRAIN, + splitHalf: SPLIT_HALF, seed: pageArg ? null : seed, pages: [...new Set(rows.map((r) => r.page))].length, pagesDrawn, @@ -1139,8 +1214,73 @@ writeFileSync(outPath, surface()); * sessions asked their questions exclusively about the pages the correction was * already fitted to. A reader of this file should be able to see that without * being told. + * + * `shifts` is always the per-page median and always means the same thing, so a + * run at any grain still writes the file the two by-eye scorers already read. + * A run at a FINER grain adds `perLine` beside it, and that block is the whole + * point of measuring a grain at all: the per-page family was measured to be + * exhausted, so whatever ships is a table the four pinned numbers cannot hold, + * and a table nobody wrote down is a finding rather than a correction. It is + * additive on purpose — a page-grain run writes byte-identical output to the + * one before this field existed, which is what keeps the committed rulings and + * their fingerprints valid. + * + * The two grains store different things because they ARE different things. A + * per-line grain stores one displacement a line, on top of its page's. A tilted + * one stores a line through the marks of that line, so it needs two numbers an + * axis — read them as `dx = ax·cx + bx`, where `cx` is the centre of the mark + * across the page, which is the whole reason the model exists: a difference in + * glyph advances accumulates ALONG a line rather than sitting constant on it. */ const MUSHAF_PAGES = 604; + +/** + * The finer-than-a-page half of the correction, or nothing at all. + * + * Spread into the object below, so a page-grain run adds no key and the file + * stays what it was. `fitted.lines` is keyed `page:line` by the model, and it + * is unpacked back into two fields here rather than shipped as a joined string, + * because whatever reads this has a page and a line and should not have to + * know how this file happened to spell them together. + */ +function finerGrain() { + if (GRAIN === "page" || !fitted?.lines?.size) return {}; + const rows = [...fitted.lines.entries()] + .map(([k, v]) => { + const [page, line] = k.split(":").map(Number); + return { page, line, v }; + }) + .sort((a, b) => a.page - b.page || a.line - b.line); + return { + grain: GRAIN, + perLine: { + how: + GRAIN === "line" + ? "one further displacement for every printed line, applied ON TOP of that page's row above" + : "a straight line through the marks of one printed line, applied ON TOP of that page's row above: dx = ax·cx + bx, where cx is the centre of the mark across the page", + floor: SPLIT_HALF ? SPLIT_FLOOR : FLOOR, + note: "a line with no row here was not measured; leave it on its page's displacement rather than guessing one", + lines: + GRAIN === "line" + ? rows.map(({ page, line, v }) => ({ + page, + line, + dx: Number(v.dx.toFixed(4)), + dy: Number(v.dy.toFixed(4)), + n: v.n, + })) + : rows.map(({ page, line, v }) => ({ + page, + line, + ax: Number(v.x.a.toFixed(6)), + bx: Number(v.x.b.toFixed(4)), + ay: Number(v.y.a.toFixed(6)), + by: Number(v.y.b.toFixed(4)), + })), + }, + }; +} + const SHIFT_OUT = arg("--shift-out", join(dirname(outPath), "mark-shift.json")); writeFileSync( SHIFT_OUT, @@ -1164,6 +1304,7 @@ writeFileSync( shifts: [...perPageShift.entries()] .sort((a, b) => a[0] - b[0]) .map(([page, s]) => ({ page, dx: Number(s.dx.toFixed(4)), dy: Number(s.dy.toFixed(4)), n: s.n })), + ...finerGrain(), }, null, 2, diff --git a/packages/etl/scripts/probe-ornament-witness.mjs b/packages/etl/scripts/probe-ornament-witness.mjs new file mode 100644 index 0000000..ff4551d --- /dev/null +++ b/packages/etl/scripts/probe-ornament-witness.mjs @@ -0,0 +1,170 @@ +#!/usr/bin/env node +/** + * The ayah-end ornaments, turned from the thing we fit on into the thing we check. + * + * Every other measurement in this line of work has the same shape and the same + * weakness: a correction is fitted to make our rectangles agree with the print's + * ink, and then it is graded by how well those rectangles agree with the ink. + * That number goes up every time the model is given more parameters, *including* + * when what it is fitting is noise. Split halves and shuffled controls + * (`lib/registration-grain.mjs`) answer that from inside — this answers it from + * outside, with an object the ink fit has never seen and could not have been + * fitted to. + * + * The ornaments are the only objects both corpora label. Ours are + * ``, theirs ``, and the four numbers + * every page's geometry rides — `word-boxes.pin.json`, via `applierFromPin` — + * were fitted on exactly these points and nothing else. So the question this + * asks is the inverse of the one that fit answered: + * + * **if we correct the text to sit on its ink, where do the ornaments end up?** + * + * Two answers, and they mean opposite things: + * + * - **About where they are now.** Then the two prints differ by a single + * displacement, the ornaments were merely an unlucky place to have measured + * it, and the correction is a correction of the whole page. + * - **Worse, by about the size of the correction.** Then the two prints really do + * agree about their ornaments and disagree about their text — which is the + * diagnosis this work started from, and it carries a consequence for what + * ships: the correction belongs to the text, and moving the ayah-end markers + * with it would break the one thing that is currently right. + * + * That second reading is what the corpus says, and it says it on every page it + * was asked. The numbers are in `docs/design/mark-registration.md`. + * + * Nothing here is fitted. The correction comes from the rows a scoring run + * dumped (`probe-mark-ink.mjs --rows-out`), the pairing is the one `fitFrames` + * uses, and this only reads. It is a probe and not a gate for the usual reason: + * a red line here is a finding somebody reads, not a build failure. + * + * Usage: + * node packages/etl/scripts/probe-mark-ink.mjs --pages-n 120 --sample 200000 \ + * --rows-out rows.json --out /tmp/x.html --shift-out /tmp/x.json + * node packages/etl/scripts/probe-ornament-witness.mjs --rows rows.json + * node packages/etl/scripts/probe-ornament-witness.mjs --rows rows.json --grain page + */ +import { readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { readOurs, readTheirs, readingOrder } from "./lib/mushaf-frame.mjs"; +import { GRAINS, SPLIT_FLOOR, correctionFor } from "./lib/registration-grain.mjs"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const ETL = join(HERE, ".."); +const REPO = join(ETL, "..", ".."); +const CACHE = join(ETL, "data", "pages", ".cache", "words"); +const OURS = join(REPO, "apps", "web", "public", "assets", "pages", "hafs-kfqc"); +const PIN = JSON.parse(readFileSync(join(ETL, "data", "pages", "word-boxes.pin.json"), "utf8")); + +const arg = (name, fallback) => { + const i = process.argv.indexOf(name); + return i >= 0 && process.argv[i + 1] ? process.argv[i + 1] : fallback; +}; + +const rowsPath = arg("--rows", null); +const GRAIN = arg("--grain", "line"); +if (!rowsPath) { + console.error("--rows is required; produce it with probe-mark-ink.mjs --rows-out"); + process.exit(2); +} +if (!GRAINS.includes(GRAIN)) { + console.error(`--grain must be ${GRAINS.join(", ")}; got ${GRAIN}`); + process.exit(2); +} + +const pad3 = (n) => String(n).padStart(3, "0"); +const med = (xs) => { + const s = [...xs].sort((a, b) => a - b); + return s.length ? s[Math.floor(s.length / 2)] : NaN; +}; + +const rows = JSON.parse(readFileSync(rowsPath, "utf8")).filter((r) => r.ink >= 0.02); +const { apply: correct } = correctionFor(GRAIN, rows, SPLIT_FLOOR); +const pages = [...new Set(rows.map((r) => r.page))].sort((a, b) => a - b); + +const shipped = []; +const corrected = []; +const perPage = []; +const skipped = []; + +for (const page of pages) { + const pin = PIN.pages.find((r) => r.page === page); + if (!pin || pin.sx === undefined) { + skipped.push(page); + continue; + } + let theirs; + let ours; + try { + theirs = readTheirs(readFileSync(join(CACHE, `${pad3(page)}.svg`), "utf8")); + ours = readOurs(readFileSync(join(OURS, `${page}.svg`), "utf8")); + } catch { + skipped.push(page); + continue; + } + + // The pairing `fitFrames` uses, and for the same reason: emitted order is not + // reading order, and pairing on it fits a mirror. Using the same one here means + // these are the very points the shipped four numbers were fitted on, rather + // than a second opinion about which ornament is which. + const T = readingOrder( + theirs.marks.map((t) => [(t.box[0] + t.box[2]) / 2, (t.box[1] + t.box[3]) / 2]), + 8, + ); + const O = readingOrder(ours.marks, 11); + if (T.length !== O.length || T.length < 3) { + skipped.push(page); + continue; + } + + // Which printed line an ornament sits on. Their ornaments carry no line number + // — only their words do — so the line is the word band whose centre is nearest. + // A per-line correction has to be asked about the line the ornament is on, and + // asking about the wrong one would make this test easier than it should be. + const bands = new Map(); + for (const w of theirs.words) { + if (!Number.isFinite(w.line)) continue; + if (!bands.has(w.line)) bands.set(w.line, []); + bands.get(w.line).push((w.box[1] + w.box[3]) / 2); + } + const centres = [...bands.entries()].map(([line, ys]) => [line, med(ys)]); + const lineOf = (y) => + centres.reduce((best, c) => (Math.abs(c[1] - y) < Math.abs(best[1] - y) ? c : best), centres[0])?.[0]; + + const before = []; + const after = []; + for (let i = 0; i < T.length; i += 1) { + const x = pin.sx * T[i][0] + pin.tx; + const y = pin.sy * T[i][1] + pin.ty; + const c = correct({ page, line: lineOf(T[i][1]), box: [x - 2.8, y - 1.8, 5.6, 3.6], ink: 1 }); + before.push(Math.hypot(x - O[i][0], y - O[i][1])); + after.push(Math.hypot(x + c.dx - O[i][0], y + c.dy - O[i][1])); + } + shipped.push(...before); + corrected.push(...after); + perPage.push({ page, n: T.length, before: med(before), after: med(after) }); +} + +const worse = perPage.filter((r) => r.after > r.before).length; +const out = { + grain: GRAIN, + pairs: shipped.length, + pages: perPage.length, + skipped, + underShippedFit: med(shipped), + underInkCorrection: med(corrected), + pagesMovedFurtherOut: worse, +}; + +if (process.argv.includes("--json")) { + console.log(JSON.stringify(out, null, 2)); +} else { + console.log(`${out.pairs} ornament pairs over ${out.pages} pages, corrected at the "${GRAIN}" grain\n`); + console.log(` where the ornaments sit under the shipped fit ${out.underShippedFit.toFixed(3)} units`); + console.log(` where they would sit under the ink correction ${out.underInkCorrection.toFixed(3)} units`); + console.log(` pages the correction moves them further out ${worse} of ${out.pages}`); + if (skipped.length) console.log(` pages with nothing to pair ${skipped.length}`); + console.log("\n The ink fit never saw an ornament, so none of this was available to it."); + console.log(" A difference the whole page shared would read the same on both lines."); +} diff --git a/packages/etl/scripts/score-mark-report.mjs b/packages/etl/scripts/score-mark-report.mjs new file mode 100644 index 0000000..5fe6a9b --- /dev/null +++ b/packages/etl/scripts/score-mark-report.mjs @@ -0,0 +1,501 @@ +/** + * Reads a mark-report sitting and says what it did and did not establish. + * + * node packages/etl/scripts/score-mark-report.mjs [--rows ] + * + * ## What this is for, and why it is a separate instrument + * + * The other two scorers in this directory read sittings that asked a *closed* + * question — is this rectangle better than that one, and where exactly does this + * one belong. Both produce a number in page units, and both are about a correction + * that has already been proposed. + * + * This one reads a sitting that asked an open one: look at this rectangle on this + * printed page and tell me, in five words, what is wrong with it. That produces a + * *rate* rather than a distance — how often the thing we are about to ship is + * visibly wrong to somebody who knows the print — and rates are the easier of the + * two to quote past what they can carry. So most of what follows is about the ways + * this particular number can be over-read, each of them printed whether or not it + * is convenient. + * + * ## The four things it refuses to do + * + * **It will not pool the two populations.** The marks placed from their own ink and + * the marks that fell back to the printed line are placed by *different rules*, and + * a rate over both is a fact about neither — it is an average weighted by whatever + * mix the sampler happened to draw, which is a property of the sampler. Every number + * below is per population, and the two are never added. This is the same rule the + * corpus figures follow, one level up. + * + * **It will not trust the transcript about which population a mark is in.** Each + * answer carries the rule that drew it, but that field is a claim, and a transcript + * can be old, hand-edited, or built before the field existed. The rule is a pure + * function of the displacements — convincing match, and the search had room — so it + * is re-derived here from the rows and the transcript's own thresholds, and the + * transcript's claim is checked against it rather than believed. A disagreement + * stops the run. + * + * **It will not read a sitting against different displacements.** Same guard as the + * other two instruments: the rectangles are a function of the measurements, so a + * report read against a different measurement is a set of statements about marks + * that were never on the screen. The fingerprint has to match. + * + * **It will not add a print defect to the error count.** "Something is odd in the + * print" is not this correction getting a mark wrong — it is a mark that cannot be + * graded, because the thing it would be graded against is itself in question. Adding + * those to the numerator inflates our error; dropping them silently deflates it. They + * come out of the *denominator*, the rate is printed both ways so the size of that + * choice is visible, and they are routed to docs/issues.json where a defect in + * vendored data belongs. + * + * ## Silence is the verdict, which is why `seen` matters + * + * The sitting has no "this one looks right" button, deliberately — the common case + * must not cost a click or nobody finishes. So passing a mark is the verdict that + * nothing is wrong with it, and the rate is answers over marks-actually-looked-at. + * That denominator lives in the transcript as `seen`. A transcript without it (one + * saved before the field existed) cannot distinguish a reader who cleared sixty + * marks and found four bad ones from one who looked at five and shut the tab, so + * both readings are printed as bounds and no single rate is claimed. + * + * ## And what a sitting this size can find + * + * Sixty is the size that fits in one sitting, not the size that resolves a rate. The + * report ends with what the sample can support: an interval on every share, and — + * when nothing at all was found — the bound that a clean result actually carries, + * which is about 3/n and is nowhere near zero. A clean sitting is exactly the result + * somebody will want to quote as "we checked it", and the honest version of that + * sentence has a number in it. + * + * Exits 2 when it refuses to read the file at all. It does not exit non-zero for bad + * news: a sitting that finds a high rate has done its job, and a scorer that fails + * the build for it teaches everyone to stop sitting them. + */ +import { readFileSync } from "node:fs"; +import { dirname, isAbsolute, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { correctionFor } from "./lib/registration-grain.mjs"; +import { wilson } from "./lib/mark-ink.mjs"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const ETL = join(HERE, ".."); +const ROOT = join(ETL, "..", ".."); + +const argv = process.argv.slice(2); +const arg = (k, d) => { + const i = argv.indexOf(k); + return i >= 0 && argv[i + 1] ? argv[i + 1] : d; +}; +let path = null; +for (let i = 0; i < argv.length; i += 1) { + if (argv[i].startsWith("--")) i += 1; + else if (!path) path = argv[i]; +} +if (!path) { + process.stderr.write("usage: score-mark-report.mjs [--rows ]\n"); + process.exit(2); +} + +const die = (msg) => { + process.stderr.write(`refused: ${msg}\n`); + process.exit(2); +}; + +const doc = JSON.parse(readFileSync(path, "utf8")); +if (doc.built !== "mark-report") die(`${path} is not a mark-report transcript (built: ${doc.built ?? "absent"}).`); + +/** The same hash the builder stamps in. One copy of it per instrument, on purpose: */ +/* a shared helper would let a change to it silently re-bless every old ruling. */ +function fingerprint(s) { + let h = 0x811c9dc5; + for (let i = 0; i < s.length; i += 1) { + h ^= s.charCodeAt(i); + h = Math.imul(h, 0x01000193); + } + return (h >>> 0).toString(16).padStart(8, "0"); +} + +// The builder records the path repo-relative when the rows live inside the checkout +// and absolute when they do not — a working file in a scratch directory is the +// normal case, and joining that onto the root produces a path that exists nowhere. +const rowsPath = arg("--rows", doc.rows ? (isAbsolute(doc.rows) ? doc.rows : join(ROOT, doc.rows)) : null); +if (!rowsPath) die("the transcript names no displacements file and --rows was not given."); +let rowsText; +try { + rowsText = readFileSync(rowsPath, "utf8"); +} catch { + die(`cannot read the displacements at ${rowsPath}. Pass --rows if they have moved.`); +} +const rowsFp = fingerprint(rowsText); +if (rowsFp !== doc.rowsFingerprint) { + die( + `this sitting was made against displacements ${doc.rowsFingerprint}, and ${rowsPath} is ${rowsFp}.\n` + + " Every rectangle in it would be a statement about a mark that was never on the screen.", + ); +} +const parsedRows = JSON.parse(rowsText); +const allRows = Array.isArray(parsedRows) ? parsedRows : parsedRows.rows; + +/* ── the split, re-derived rather than believed ───────────────────────────── */ + +const radius = Number(doc.radius ?? 3); +const iouFloor = Number(doc.iouFloor ?? 0.55); +const EPS = 1e-6; +// Per axis, because the search window is a square — the same reasoning as the +// builder's, and the same error this repo has already paid for once. +const atEdge = (r) => Math.abs(Math.abs(r.dx) - radius) < EPS || Math.abs(Math.abs(r.dy) - radius) < EPS; +const placed = (r) => r.iouBest >= iouFloor && !atEdge(r); +const ruleOf = (r) => (placed(r) ? "ink" : "line-tilt"); + +const byId = new Map(); +for (const r of allRows) byId.set(`${r.page}:${r.k}`, r); + +const { apply } = correctionFor("line-tilt", allRows); +/** Where the sitting drew the rectangle — the builder's rule, re-run here. */ +const drawnAt = (r) => { + const c = placed(r) ? { dx: r.dx, dy: r.dy } : apply(r); + return [r.box[0] + c.dx, r.box[1] + c.dy, r.box[2], r.box[3]]; +}; + +const said = Array.isArray(doc.said) ? doc.said : []; +const unknown = []; +const mismatched = []; +let repaired = 0; +for (const e of said) { + const r = byId.get(e.id); + if (!r) { + unknown.push(e.id); + continue; + } + const truth = ruleOf(r); + if (e.rule == null) repaired += 1; + else if (e.rule !== truth) mismatched.push(`${e.id}: says ${e.rule}, is ${truth}`); + e.rule = truth; +} +if (unknown.length) { + die( + `${unknown.length} answers name marks that are not in the displacements (first: ${unknown[0]}).\n` + + " The fingerprint matched, so this is a hand-edited transcript rather than a stale one.", + ); +} +if (mismatched.length) { + die( + `${mismatched.length} answers disagree with the displacements about which rule drew them.\n` + + mismatched.slice(0, 5).map((m) => ` ${m}`).join("\n") + + "\n Reading these would file answers about one option against the other.", + ); +} + +/* ── the denominator ──────────────────────────────────────────────────────── */ + +const shown = Number(doc.shown ?? 0); +/** + * Which cards this sitting actually looked at — recovered from the rows in the same + * order the builder drew them, because the transcript carries answers rather than + * cards. `seen` is a count of cards passed, so the population mix of the denominator + * is exactly the mix of the first `seen` of them. + */ +const seen = doc.seen == null ? null : Number(doc.seen); + +/** + * The builder draws in page-then-index order, so the answers can be put back in the + * order they were given even though the transcript carries no cards. Only the marks + * that were answered are known here, which is enough to locate the last answer + * within the sitting and no more. + */ +const cardOrder = [...new Set(said.map((e) => e.id))].sort((a, b) => { + const [pa, ka] = a.split(":").map(Number); + const [pb, kb] = b.split(":").map(Number); + return pa - pb || ka - kb; +}); +/** + * The bounds a transcript with no `seen` leaves us with. At the low end the reader + * looked at exactly the marks they said something about and no others, which would + * make the rate 100%; at the high end they cleared the whole sitting. Both are + * consistent with the file, which is the point — and the gap between them is usually + * the difference between a finding and a non-finding. + */ +const lowBound = cardOrder.length; + +/* ── the report ───────────────────────────────────────────────────────────── */ + +const WORDS = { + placement: "moved it", + "wrong-shape": "wrong shape", + "intended-ink": "boxed the right ink", + "print-defect": "odd in the print", + exception: "banked, could not say", +}; +/** The two that say our placement is wrong. Everything else is not that. */ +const FAULTS = ["placement", "wrong-shape", "intended-ink"]; +/** + * The affirmation, and it is deliberately not in WORDS. That table answers "what + * was wrong, and how often", and a row reading "nothing wrong 54 — 90.0%" in it + * would be read by somebody skimming as a 90% failure of something. It gets its + * own line below, where it says the thing it actually says: how much of the + * denominator a human explicitly vouched for, as against passed in silence. + */ +const AFFIRMED = "looks-right"; + +const pct = (v) => `${(v * 100).toFixed(1)}%`; +const n3 = (v) => (Math.round(v * 1000) / 1000).toFixed(3); +const median = (xs) => { + if (!xs.length) return NaN; + const s = [...xs].sort((a, b) => a - b); + const m = s.length >> 1; + return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2; +}; +const quantile = (xs, q) => { + if (!xs.length) return NaN; + const s = [...xs].sort((a, b) => a - b); + return s[Math.min(s.length - 1, Math.floor(q * s.length))]; +}; + +const out = []; +const say = (s = "") => out.push(s); + +say(`${path.replace(`${ROOT}/`, "")}`); +say(` ${doc.set} set · seed ${doc.seed} · displacements ${doc.rowsFingerprint} · ${shown} drawn`); +say(` drawn by ${Object.entries(doc.drawnBy ?? {}).map(([k, v]) => `${k} ${v}`).join(", ") || "unrecorded"}`); +if (repaired) { + say(` ${repaired} answers carried no rule and were re-derived from the displacements.`); + say(" That is safe here only because the fingerprint matched: the rule is a function of the rows."); +} +say(); + +if (seen == null) { + say("This transcript does not record how far the reader got, so it has no denominator."); + say(` Between ${lowBound} and ${shown} marks were looked at — ${lowBound} if they said something`); + say(` about every mark they saw, ${shown} if they cleared the sitting. The rates below use the`); + say(" upper figure, which is the charitable one, and are quotable only with that stated."); + say(" Sittings built after this field was added carry it; this one predates it."); + say(); +} + +/** + * Everything from here is per population. The header says so each time rather than + * once at the top, because a reader who skims to the number they wanted is exactly + * the reader who will otherwise quote it as the number for the whole book. + */ +const groups = new Map(); +for (const e of said) { + if (!groups.has(e.rule)) groups.set(e.rule, []); + groups.get(e.rule).push(e); +} +// The denominator's own split — how many of the marks looked at were drawn by each +// rule. The transcript carries answers rather than cards, so this comes from the +// builder's recorded tally rather than from counting them here. +const drawn = doc.drawnBy ?? {}; +const drawnTotal = Object.values(drawn).reduce((a, b) => a + b, 0); +const denomOf = (rule) => { + // A sitting drawn entirely by one rule — which is what --set placed and --set + // fallback both produce — needs no apportioning at all. A mixed one gets the + // cards split in the proportion the builder recorded, which assumes the reader + // worked front to back; they can only go forward and back one card at a time, so + // that assumption is the shape of the instrument rather than a guess about them. + const only = Object.keys(drawn).length === 1 && drawnTotal === shown; + const share = only ? 1 : (drawn[rule] ?? 0) / (drawnTotal || 1); + if (seen == null) return [Math.round(lowBound * share), Math.round(shown * share)]; + return [Math.round(seen * share), Math.round(seen * share)]; +}; + +for (const rule of [...groups.keys()].sort()) { + const evs = groups.get(rule); + const [dLo, dHi] = denomOf(rule); + const label = rule === "ink" ? "placed from their own ink" : "that inherited the printed line"; + say(`── ${rule} — the marks ${label} ──`); + if (dLo === dHi) say(` ${dLo} marks looked at, ${evs.length} answers given about ${new Set(evs.map((e) => e.id)).size} of them`); + else say(` between ${dLo} and ${dHi} marks looked at, ${evs.length} answers about ${new Set(evs.map((e) => e.id)).size} of them`); + + const defects = new Set(evs.filter((e) => e.kind === "print-defect").map((e) => e.id)); + const banked = new Set(evs.filter((e) => e.kind === "exception").map((e) => e.id)); + const faulted = new Set(evs.filter((e) => FAULTS.includes(e.kind)).map((e) => e.id)); + const affirmed = new Set(evs.filter((e) => e.kind === AFFIRMED).map((e) => e.id)); + + say(); + say(" What was said, by word — a mark can carry more than one, so these do not sum:"); + for (const kind of Object.keys(WORDS)) { + const ids = new Set(evs.filter((e) => e.kind === kind).map((e) => e.id)); + if (!ids.size && kind !== "placement" && kind !== "wrong-shape") continue; + const [lo, hi] = wilson(ids.size, dHi || 1); + say(` ${WORDS[kind].padEnd(24)} ${String(ids.size).padStart(3)} ${pct(ids.size / (dHi || 1)).padStart(6)} 95% ${pct(lo)}–${pct(hi)}`); + } + + say(); + say(" Our placement is wrong on this mark — moved, wrong shape, or boxed elsewhere:"); + const gradable = (dHi || 0) - defects.size; + const [fLo, fHi] = wilson(faulted.size, Math.max(1, gradable)); + say(` ${faulted.size} of ${gradable} gradable marks — ${pct(faulted.size / Math.max(1, gradable))}, 95% ${pct(fLo)}–${pct(fHi)}`); + if (defects.size) { + say(` ${defects.size} ${defects.size === 1 ? "mark is" : "marks are"} out of the denominator because the print itself was`); + say(" called into question there — not our error, and not evidence that we are right either."); + say(` Counted in instead, the rate would read ${pct(faulted.size / (dHi || 1))}; the gap is the size of that choice.`); + } + if (banked.size) say(` ${banked.size} more ${banked.size === 1 ? "was" : "were"} banked as could-not-say, and ${banked.size === 1 ? "is" : "are"} in neither column.`); + + /** + * How much of that denominator a person actually vouched for. + * + * The rate above divides by every mark the reader passed, which quietly assumes + * that passing one meant judging it. That assumption is only as good as the + * reader's attention, and it is invisible in the arithmetic — a sitting clicked + * through in ninety seconds and one worked through carefully produce the same + * denominator and the same clean 0%. So the affirmations are reported against it: + * a large silent share does not invalidate the rate, it says how much of the rate + * rests on somebody having actually looked. + */ + say(); + const spoken = new Set([...affirmed, ...faulted, ...defects, ...banked]); + const silent = Math.max(0, (dHi || 0) - spoken.size); + say(" How much of that denominator somebody actually vouched for:"); + say(` ${affirmed.size} said outright that nothing was wrong · ${silent} passed in silence`); + if (!affirmed.size && silent) { + say(" Nothing here was affirmed out loud. Every clean mark in the rate above is a mark the"); + say(" reader moved past, which is consistent with a careful sitting and with a skimmed one."); + say(" Sittings built before the affirmation button existed all read this way."); + } else if (silent > affirmed.size) { + say(" More were passed in silence than were affirmed, so the rate leans on the passing."); + } + + /** + * A mark carrying both an affirmation and a fault. The transcript is appended to + * rather than rewritten, so this is a reader who changed their mind and did not + * take the first answer back — which is a real event, not a corruption. It is + * counted as a fault, because the conservative reading of "I said it was fine and + * then moved it" is that it needed moving, and it is printed because a scorer that + * silently picks one of two contradictory statements is a scorer nobody can check. + */ + const bothWays = [...affirmed].filter((id) => faulted.has(id)); + if (bothWays.length) { + say(); + say(` ${bothWays.length} ${bothWays.length === 1 ? "mark was" : "marks were"} called fine and also faulted: ${bothWays.join(", ")}`); + say(" Counted as faults above. A later answer was not taken back, so both statements stand in"); + say(" the transcript; this line is here so the choice between them is visible rather than made"); + say(" quietly. If the affirmation was the mistake, the fix is to retract it and score again."); + } + + /** + * One row per mark, and the last standing placement is the row. + * + * This used to median the increments across every event, and an increment is the + * wrong thing to median twice over. A reader settles a rectangle by pushing it one + * way and then a little back, so opposite-signed nudges cancel; and one mark nudged + * forty-four times outvotes twenty-five marks moved once each. On the banked sitting + * that printed 0.000 across and 0.000 down for twenty-six marks that had every one + * of them been dragged the better part of two units. The rectangle's final resting + * place is the reader's statement; the route it took there is not. + * + * The transcript is walked in order and the last placement carrying a total wins. + * A hand-over has its retractions already applied, so what survives here is only + * what the reader still stood behind when they handed it over. + */ + const finals = new Map(); + let goes = 0; + for (const e of evs) { + if (e.kind !== "placement") continue; + goes += 1; + if (Array.isArray(e.to)) finals.set(e.id, e); + } + if (finals.size) { + /** + * Two different distances, and printing either one alone has already misled + * somebody in this repo. + * + * `to` is measured from the mark's raw box, so it carries the correction this + * pipeline had already applied *plus* whatever the reader added on top. Subtract + * the applied part — `drawnAt(r) - r.box`, which is exactly what the sitting drew + * with — and what is left is the reader's own hand: how wrong our rectangle looked + * to a person sitting in front of it. Leave it in and you have the whole correction + * their answer implies, which is the figure that can be set beside a measurement + * of the ink, because that one is measured from the raw box too. + */ + const hand = []; + const whole = []; + for (const [id, e] of finals) { + const r = byId.get(id); + const at = drawnAt(r); + whole.push(e.to); + hand.push([e.to[0] - (at[0] - r.box[0]), e.to[1] - (at[1] - r.box[1])]); + } + const mags = hand.map((v) => Math.hypot(v[0], v[1])); + + say(); + say(" How far the reader's own hand moved the rectangle we drew, in page units:"); + say(` median ${n3(median(mags))} · p90 ${n3(quantile(mags, 0.9))} · worst ${n3(Math.max(...mags))}`); + say(` across ${n3(median(hand.map((v) => v[0])))} · down ${n3(median(hand.map((v) => v[1])))} — medians, signed`); + say(` ${finals.size} ${finals.size === 1 ? "mark" : "marks"}, one row each, settled over ${goes} separate nudges and drags.`); + if (goes > finals.size * 2) { + say(" That ratio is itself a finding about the pad rather than about the print: a rectangle"); + say(" that takes several goes to settle is one the controls are not letting anybody place."); + } + + say(); + say(" Where those rectangles ended up, measured from the box before any correction:"); + say(` across ${n3(median(whole.map((v) => v[0])))} · down ${n3(median(whole.map((v) => v[1])))} — medians, signed`); + say(" This is the whole correction their answers imply — ours already in it, theirs on top —"); + say(" so it is the one to set beside a measurement taken off the ink, which is measured from"); + say(" that same uncorrected box. It is not a second opinion about the line above and the two"); + say(" are never differenced: the gap between them is only the correction we already applied."); + + say(); + say(" A distance and the count above it are different questions and are never combined: a"); + say(" mark called the wrong shape has no distance, and a mark moved a hundredth of a unit is"); + say(" not the same finding as one moved a whole mark-height."); + } + + const drawnBoxes = evs.filter((e) => e.kind === "intended-ink" && Array.isArray(e.box)); + if (drawnBoxes.length) { + const cds = []; + const ars = []; + for (const e of drawnBoxes) { + const r = byId.get(e.id); + const at = drawnAt(r); + cds.push(Math.hypot(e.box[0] + e.box[2] / 2 - (at[0] + at[2] / 2), e.box[1] + e.box[3] / 2 - (at[1] + at[3] / 2))); + ars.push((e.box[2] * e.box[3]) / Math.max(1e-9, at[2] * at[3])); + } + say(); + say(" Where the reader drew the ink instead, against where we drew the rectangle:"); + say(` centres apart — median ${n3(median(cds))} units · worst ${n3(Math.max(...cds))} (n=${cds.length})`); + say(` their area over ours — median ${median(ars).toFixed(2)}×`); + say(" An area far from 1 is the wrong-size complaint with a number on it; a large centre gap"); + say(" with an area near 1 is the same rectangle in the wrong place, which is displacement."); + } + + if (defects.size) { + say(); + say(" Called odd in the print — these belong in docs/issues.json as a defect in vendored data,"); + say(" not in this correction's error figures:"); + for (const e of evs.filter((x) => x.kind === "print-defect")) { + say(` page ${e.page}, line ${e.line}, ${e.name} — ${e.note || "(no note)"}`); + } + } + if (banked.size) { + say(); + say(" Banked, could not say:"); + for (const e of evs.filter((x) => x.kind === "exception")) { + say(` page ${e.page}, line ${e.line}, ${e.name} — ${e.note || "(no note)"}`); + } + } + + say(); + say(" What this many marks can support:"); + if (!faulted.size) { + say(` Nothing was found, and that bounds the rate at about ${pct(3 / Math.max(1, gradable))} — not at zero.`); + say(" It is consistent with one mark in twenty being wrong. It is not a clean bill of health."); + } else { + say(` The interval above is the finding, not the point estimate: ${gradable} marks cannot tell`); + say(` ${pct(fLo)} from ${pct(fHi)}, and a decision that needs them apart needs a larger sitting.`); + } + say(` It says nothing whatever about the other population — see the ${rule === "ink" ? "fallback" : "placed"} sitting for that.`); + say(); +} + +if (groups.size > 1) { + say("Two populations appeared in one transcript and are reported apart above. They are not"); + say("summed: the marks placed from their own ink and the marks that fell back to the printed"); + say("line are placed by different rules, so a rate over both is weighted by whatever mix the"); + say("sampler drew, which is a fact about the sampler rather than about the print."); + say(); +} + +process.stdout.write(`${out.join("\n")}\n`); diff --git a/packages/etl/scripts/score-mark-report.test.mjs b/packages/etl/scripts/score-mark-report.test.mjs new file mode 100644 index 0000000..8412bef --- /dev/null +++ b/packages/etl/scripts/score-mark-report.test.mjs @@ -0,0 +1,372 @@ +/** + * The refusals, on transcripts built to be wrong in one way each. + * + * Every other test in this package imports a module and checks a number. This one + * runs the script, because what is worth holding here is not an estimator — it is + * the four things the scorer will not do, and all four live in the top-level flow + * where a `die()` ends the process. Extracting them into a lib to make them + * importable would leave the tested copy and the running copy as two different + * things, which is the failure the test exists to prevent. + * + * The failure they guard against is quiet and expensive. A sitting is half an hour + * of the scarcest thing in this project, and every one of these mistakes produces a + * *plausible* number rather than an error: answers about marks that were never on + * the screen, answers about one placement rule filed against the other, a rate that + * counts a defect in somebody else's print as our error. A wrong percentage looks + * exactly like a right one, gets quoted into a register, and is argued from months + * later by somebody who was not here. + * + * So the fixtures are built so the truth is a matter of construction: a row whose + * match is convincing and whose search had room is in the ink population *by + * definition*, and a transcript that claims otherwise must stop the run rather + * than be read. + */ +import { execFileSync } from "node:child_process"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const SCRIPT = join(HERE, "score-mark-report.mjs"); + +/** + * A fourth copy of the hash, written from the algorithm rather than copied from + * the script. The instruments each carry their own on purpose — a shared helper + * would let a change to it silently re-bless every ruling ever made — and this + * copy is what makes that arrangement checkable instead of merely stated. + */ +function fingerprint(s) { + let h = 0x811c9dc5; + for (let i = 0; i < s.length; i += 1) { + h ^= s.charCodeAt(i); + h = Math.imul(h, 0x01000193); + } + return (h >>> 0).toString(16).padStart(8, "0"); +} + +const RADIUS = 3; +const IOU = 0.55; + +/** + * Forty marks on one line of one page — one line rather than several so the line + * grain clears its twenty-mark floor and the fallback rows are corrected by a fit + * that actually ran, and one page so the page median exists. + * + * Twenty-four are in the ink population and sixteen are not, and the sixteen are + * split between the two ways out of it: fifteen whose best match is unconvincing, + * and one — `1:24` — whose match is excellent but which reached the edge of the + * search window, so the true best may lie outside it and was never seen. That one + * is the whole reason the population test is two clauses rather than one. + */ +const ROWS = Array.from({ length: 40 }, (_, k) => ({ + page: 1, + k, + line: 1, + name: "kasra", + box: [180 - k * 4, 40, 5.6, 3.6], + ink: 0.1, + dx: k === 24 ? RADIUS : 0.5, + dy: -0.25, + iou0: 0, + iouBest: k <= 24 ? 0.8 : 0.2, + phi0: 0, + nullPhi: 0, +})); +const INK_IDS = ROWS.filter((r) => r.iouBest >= IOU && Math.abs(r.dx) !== RADIUS).map((r) => `${r.page}:${r.k}`); +const FALLBACK_IDS = ROWS.map((r) => `${r.page}:${r.k}`).filter((id) => !INK_IDS.includes(id)); + +let dir; +let rowsPath; +let fp; +let nth = 0; + +beforeAll(() => { + dir = mkdtempSync(join(tmpdir(), "hifth-score-")); + rowsPath = join(dir, "rows.json"); + const text = JSON.stringify(ROWS); + writeFileSync(rowsPath, text); + fp = fingerprint(text); +}); +afterAll(() => rmSync(dir, { recursive: true, force: true })); + +/** A transcript on disk, with the head the builder would have stamped. */ +function sit(over = {}) { + const doc = { + built: "mark-report", + rows: rowsPath, + rowsFingerprint: fp, + set: "placed", + seed: 23, + radius: RADIUS, + iouFloor: IOU, + pool: ROWS.length, + of: ROWS.length, + shown: 10, + drawnBy: { ink: 10 }, + finished: true, + seen: 10, + said: [], + ...over, + }; + const path = join(dir, `sitting-${(nth += 1)}.json`); + writeFileSync(path, JSON.stringify(doc)); + return path; +} + +function run(...args) { + try { + const out = execFileSync(process.execPath, [SCRIPT, ...args], { encoding: "utf8" }); + return { code: 0, out, err: "" }; + } catch (e) { + return { code: e.status, out: e.stdout ?? "", err: e.stderr ?? "" }; + } +} + +/** An answer, with the population it really came from filled in by construction. */ +const say = (id, kind, extra = {}) => ({ + id, + kind, + rule: INK_IDS.includes(id) ? "ink" : "line-tilt", + page: 1, + line: 1, + name: "kasra", + ...extra, +}); + +describe("what it refuses to read", () => { + it("stops on a sitting made against different displacements", () => { + const { code, err } = run(sit({ rowsFingerprint: "deadbeef" })); + expect(code).toBe(2); + expect(err).toMatch(/deadbeef/); + expect(err).toMatch(/never on the screen/); + }); + + it("stops when an answer disagrees with the displacements about which rule drew it", () => { + const id = INK_IDS[0]; + const { code, err } = run(sit({ said: [{ ...say(id, "placement", { by: [0.4, 0.1] }), rule: "line-tilt" }] })); + expect(code).toBe(2); + expect(err).toMatch(new RegExp(`${id}: says line-tilt, is ink`)); + expect(err).toMatch(/one option against the other/); + }); + + it("stops on an answer naming a mark the displacements do not have", () => { + const { code, err } = run(sit({ said: [{ ...say(INK_IDS[0], "wrong-shape"), id: "1:999" }] })); + expect(code).toBe(2); + expect(err).toMatch(/1:999/); + expect(err).toMatch(/hand-edited/); + }); + + it("stops on a file that is not one of these sittings at all", () => { + const { code, err } = run(sit({ built: "mark-nudge" })); + expect(code).toBe(2); + expect(err).toMatch(/not a mark-report transcript/); + }); + + it("stops when the displacements it names are gone, and says the flag that fixes it", () => { + const { code, err } = run(sit({ rows: join(dir, "moved-away.json") })); + expect(code).toBe(2); + expect(err).toMatch(/--rows/); + }); + + it("re-derives a rule the transcript never carried, and says it did", () => { + const said = [{ ...say(INK_IDS[0], "wrong-shape"), rule: undefined }]; + const { code, out } = run(sit({ said })); + expect(code).toBe(0); + expect(out).toMatch(/1 answers carried no rule and were re-derived/); + expect(out).toMatch(/── ink —/); + }); +}); + +describe("the two populations", () => { + it("are reported apart and never summed", () => { + const said = [say(INK_IDS[0], "wrong-shape"), say(FALLBACK_IDS[0], "wrong-shape")]; + const { code, out } = run(sit({ set: "mixed", shown: 20, seen: 20, drawnBy: { ink: 12, "line-tilt": 8 }, said })); + expect(code).toBe(0); + expect(out).toMatch(/── ink — the marks placed from their own ink ──/); + expect(out).toMatch(/── line-tilt — the marks that inherited the printed line ──/); + // Each population's denominator is its own share of the cards, not the sitting. + expect(out).toMatch(/12 marks looked at/); + expect(out).toMatch(/8 marks looked at/); + expect(out).toMatch(/are reported apart above\. They are not\nsummed/); + // No line anywhere adds the two together into one figure for the sitting. + expect(out).not.toMatch(/of 20 gradable marks/); + }); + + it("puts a mark whose search reached the edge in the fallback set, however good its match", () => { + // 1:24 matches at 0.8, well over the floor, and is still not placed from ink. + expect(FALLBACK_IDS).toContain("1:24"); + const { code, out } = run(sit({ set: "fallback", drawnBy: { "line-tilt": 10 }, said: [say("1:24", "wrong-shape")] })); + expect(code).toBe(0); + expect(out).toMatch(/── line-tilt —/); + expect(out).not.toMatch(/── ink —/); + }); +}); + +describe("what counts as our error", () => { + it("takes a defect in the print out of the denominator, and prints the rate both ways", () => { + const said = [ + say(INK_IDS[0], "placement", { by: [0.4, 0.1] }), + say(INK_IDS[1], "print-defect", { note: "the alif is broken in the print" }), + ]; + const { code, out } = run(sit({ said })); + expect(code).toBe(0); + // One fault, ten cards, one of them ungradable: 1/9, and 1/10 counted in. + expect(out).toMatch(/1 of 9 gradable marks — 11\.1%/); + expect(out).toMatch(/the rate would read 10\.0%/); + expect(out).toMatch(/the alif is broken in the print/); + }); + + it("leaves a banked answer in neither column", () => { + const said = [say(INK_IDS[0], "exception", { note: "could not tell" })]; + const { code, out } = run(sit({ said })); + expect(code).toBe(0); + expect(out).toMatch(/0 of 10 gradable marks/); + expect(out).toMatch(/1 more was banked as could-not-say/); + }); + + it("does not fail the run for bad news", () => { + const said = INK_IDS.slice(0, 10).map((id) => say(id, "wrong-shape")); + const { code, out } = run(sit({ said })); + expect(code).toBe(0); + expect(out).toMatch(/10 of 10 gradable marks — 100\.0%/); + }); +}); + +describe("saying nothing is wrong", () => { + it("does not count an affirmation as a fault", () => { + const said = INK_IDS.slice(0, 10).map((id) => say(id, "looks-right")); + const { code, out } = run(sit({ said })); + expect(code).toBe(0); + expect(out).toMatch(/0 of 10 gradable marks/); + expect(out).toMatch(/10 said outright that nothing was wrong · 0 passed in silence/); + }); + + it("keeps it out of the table of what was wrong", () => { + const { code, out } = run(sit({ said: [say(INK_IDS[0], "looks-right")] })); + expect(code).toBe(0); + // A row reading "nothing wrong 1 — 10.0%" in that table is the misread this avoids. + expect(out).not.toMatch(/nothing wrong with it/); + }); + + it("says how much of the denominator was passed in silence rather than affirmed", () => { + const said = [say(INK_IDS[0], "looks-right"), say(INK_IDS[1], "wrong-shape")]; + const { code, out } = run(sit({ said })); + expect(code).toBe(0); + // Ten looked at, two spoken about, so eight went by on Next alone. + expect(out).toMatch(/1 said outright that nothing was wrong · 8 passed in silence/); + expect(out).toMatch(/the rate leans on the passing/); + }); + + it("tells a sitting that predates the button from one that was skimmed", () => { + const { code, out } = run(sit({ said: [say(INK_IDS[0], "wrong-shape")] })); + expect(code).toBe(0); + expect(out).toMatch(/Nothing here was affirmed out loud/); + expect(out).toMatch(/before the affirmation button existed/); + }); + + it("prints a mark called fine and then faulted, and counts the fault", () => { + const id = INK_IDS[0]; + const said = [say(id, "looks-right"), say(id, "placement", { by: [0.4, 0.1] })]; + const { code, out } = run(sit({ said })); + expect(code).toBe(0); + expect(out).toMatch(/1 of 10 gradable marks/); + expect(out).toMatch(new RegExp(`1 mark was called fine and also faulted: ${id}`)); + expect(out).toMatch(/Counted as faults above/); + }); +}); + +describe("what the sitting can support", () => { + it("bounds a clean result rather than zeroing it", () => { + const { code, out } = run(sit({ shown: 60, seen: 60, drawnBy: { ink: 60 }, said: [say(INK_IDS[0], "print-defect", { note: "smudge" })] })); + expect(code).toBe(0); + // 3/59 — the bound a clean sixty actually carries, which is not zero. + expect(out).toMatch(/bounds the rate at about 5\.1% — not at zero/); + expect(out).toMatch(/not a clean bill of health/); + }); + + it("refuses a denominator it does not have, and prints both readings", () => { + const said = [say(INK_IDS[0], "wrong-shape"), say(INK_IDS[1], "wrong-shape")]; + const { code, out } = run(sit({ seen: null, said })); + expect(code).toBe(0); + expect(out).toMatch(/does not record how far the reader got/); + expect(out).toMatch(/Between 2 and 10 marks were looked at/); + }); + + it("says nothing about the population it did not draw from", () => { + const { code, out } = run(sit({ said: [say(INK_IDS[0], "wrong-shape")] })); + expect(code).toBe(0); + expect(out).toMatch(/nothing whatever about the other population — see the fallback sitting/); + }); +}); + +describe("the distances, which are not the rate", () => { + /** + * The fixture that would have caught the scorer printing 0.000 on a sitting where + * every mark had been dragged. + * + * One mark is settled in three goes — half a unit across, four tenths back, four + * tenths out again — and a second is moved half a unit and left alone. Both end up + * exactly half a unit from where we drew them, so the answer is half a unit, said + * by two marks. Reading the goes instead gives four rows, two of which are the + * reader changing their mind, and a median that is neither mark's answer. + * + * `to` is a running total measured from the uncorrected box, so each one here is + * the row's own displacement plus the hand so far — which is what the sitting + * writes, and what the subtraction in the scorer has to undo. + */ + const NUDGED = INK_IDS[0]; + const ONCE = INK_IDS[1]; + const D = [0.5, -0.25]; // the displacement every ink row in the fixture carries + const at = (hx, hy) => [D[0] + hx, D[1] + hy]; + + it("gives every mark one row, however many goes it took to settle", () => { + const said = [ + say(NUDGED, "placement", { by: [0.5, 0], to: at(0.5, 0) }), + say(NUDGED, "placement", { by: [-0.4, 0], to: at(0.1, 0) }), + say(NUDGED, "placement", { by: [0.4, 0], to: at(0.5, 0) }), + say(ONCE, "placement", { by: [0.5, 0], to: at(0.5, 0) }), + ]; + const { code, out } = run(sit({ said })); + expect(code).toBe(0); + // Half a unit, not the 0.45 that medianing the four goes would have printed. + expect(out).toMatch(/median 0\.500 · p90 0\.500 · worst 0\.500/); + expect(out).toMatch(/across 0\.500 · down 0\.000 — medians, signed/); + expect(out).toMatch(/2 marks, one row each, settled over 4 separate nudges and drags/); + expect(out).toMatch(/never combined/); + }); + + it("prints the hand and the whole correction as two numbers under two sentences", () => { + const said = [say(NUDGED, "placement", { by: [0.5, 0], to: at(0.5, 0) })]; + const { code, out } = run(sit({ said })); + expect(code).toBe(0); + // The reader's own hand: half a unit across, nothing down. + expect(out).toMatch(/hand moved the rectangle we drew[\s\S]*?across 0\.500 · down 0\.000/); + // And the whole correction their answer implies, our own displacement included. + expect(out).toMatch(/before any correction:\n {4}across 1\.000 · down -0\.250/); + expect(out).toMatch(/the two\n {4}are never differenced/); + }); + + it("says so when a mark took several goes to settle", () => { + const said = [ + say(NUDGED, "placement", { by: [0.1, 0], to: at(0.1, 0) }), + say(NUDGED, "placement", { by: [0.1, 0], to: at(0.2, 0) }), + say(NUDGED, "placement", { by: [0.1, 0], to: at(0.3, 0) }), + ]; + const { code, out } = run(sit({ said })); + expect(code).toBe(0); + expect(out).toMatch(/a finding about the pad rather than about the print/); + }); + + it("measures the reader's own rectangle against where we drew ours", () => { + // An ink-placed mark is drawn at its box plus its own displacement, so a reader + // box built there is the case where we were right — zero apart, same size. + const r = ROWS.find((x) => `${x.page}:${x.k}` === INK_IDS[0]); + const at = [r.box[0] + r.dx, r.box[1] + r.dy, r.box[2], r.box[3]]; + const { code, out } = run(sit({ said: [say(INK_IDS[0], "intended-ink", { box: at })] })); + expect(code).toBe(0); + expect(out).toMatch(/centres apart — median 0\.000 units · worst 0\.000/); + expect(out).toMatch(/their area over ours — median 1\.00×/); + }); +}); diff --git a/packages/etl/scripts/serve-sittings.mjs b/packages/etl/scripts/serve-sittings.mjs new file mode 100644 index 0000000..5cb4c79 --- /dev/null +++ b/packages/etl/scripts/serve-sittings.mjs @@ -0,0 +1,247 @@ +/** + * The sittings, served with somewhere for the answers to go. + * + * ── Why this exists ────────────────────────────────────────────────────── + * + * The sittings were being served by `python3 -m http.server`, which can hand out + * a page and can do nothing else. So every answer a reader gave lived in that one + * browser's own store and nowhere else: durable enough — it survives a reload, a + * screen lock, a rebuild — but invisible from this machine, unbanked until the + * reader reached the end of a hundred-and-seventeen-card sitting and pressed save, + * and gone for good if they ever cleared the browser's data. + * + * Somebody nine answers into a sitting asked how to submit them one at a time and + * watch the remaining count come down. Both halves of that need a server that can + * be posted to: the page already posts every answer the moment it is given, but + * only if it finds a sink on the window, and a static file server is not one. + * + * ── Why it replaces the static server rather than joining it ───────────── + * + * Same host, same port, same path — deliberately, and it is the whole reason this + * is not the session runner. Browser storage is per-origin, so a sitting begun at + * one address cannot be finished at another: move the reader to a different port + * and every answer they have already given becomes unreachable, silently, with the + * page cheerfully starting again from card one. This serves the same origin the + * answers were banked under, so a reader mid-sitting reloads and picks up exactly + * where they were, now with the answers also landing here as they go. + * + * A second server was ruled out earlier for the same reason it is being ruled out + * again. This is a replacement, not an addition. + * + * ── What it writes ─────────────────────────────────────────────────────── + * + * One line per answer, appended, flushed per write, to `out/mark-answers.jsonl`. + * Append-only because the thing being defended against is losing a scarce person's + * attention: a closed laptop or a killed terminal costs one line in the worst case, + * where a rewritten document costs the file. It is the same shape and the same + * reasoning as the co-working transcript, and `score-mark-report.mjs` reads it the + * same way. + * + * That file is also what makes the remaining count fall. `build-mark-report.mjs + * --answered` drops every mark carrying a standing answer, so the next round is + * built from what is genuinely left rather than from the whole pool again. + */ +import { appendFileSync, createReadStream, existsSync, readFileSync, statSync, writeFileSync } from "node:fs"; +import { createServer } from "node:http"; +import { randomBytes } from "node:crypto"; +import { extname, join, normalize, resolve } from "node:path"; + +const arg = (name, fallback) => { + const i = process.argv.indexOf(name); + return i >= 0 && process.argv[i + 1] ? process.argv[i + 1] : fallback; +}; + +const DIR = resolve(arg("--dir", new URL("../out", import.meta.url).pathname)); +const HOST = arg("--host", "127.0.0.1"); +const PORT = Number(arg("--port", 4180)); +const LOG = join(DIR, arg("--log", "mark-answers.jsonl")); + +// Not a secret and not trying to be. It is bound to one interface on a private +// tailnet; the token only means that a page which was not served from here cannot +// post into the log by guessing the path. +const TOKEN = randomBytes(9).toString("hex"); + +const TYPES = { + ".html": "text/html; charset=utf-8", + ".js": "text/javascript; charset=utf-8", + ".json": "application/json; charset=utf-8", + ".svg": "image/svg+xml", + ".css": "text/css; charset=utf-8", +}; + +/** + * The sink the sitting looks for, in the shape it already knows. + * + * `build-mark-report.mjs` reads `window.HIFTH_SESSION` and falls back to a + * download when there is none — so this is an opt-in the page takes, not an + * interception, and a sitting opened straight off the filesystem keeps working + * exactly as it did. + */ +const SHIM = ``; + +/** + * In front of the page's own scripts, or not at all. + * + * Three anchors tried in order, because a generated single-file page very often + * opens with a doctype and a meta and never writes a head tag — the parser + * supplies one. Prepending ahead of the doctype instead would put the page in + * quirks mode, breaking its layout to fix its reporting. A page none of these + * match is served untouched and says so on the terminal, which is better than + * serving something that will quietly bank nothing. + */ +const ANCHORS = [/]*)?>/i, /]*>/i, /]*)?>/i]; + +function inject(html) { + for (const re of ANCHORS) { + const m = html.match(re); + if (m) return html.replace(re, (hit) => hit + SHIM); + } + return null; +} + +const body = (req) => + new Promise((ok, no) => { + let s = ""; + req.on("data", (c) => { + s += c; + if (s.length > 8e6) no(new Error("too big")); + }); + req.on("end", () => ok(s)); + req.on("error", no); + }); + +const json = (res, code, obj) => { + res.writeHead(code, { "content-type": "application/json; charset=utf-8" }); + res.end(JSON.stringify(obj)); +}; + +let banked = 0; + +/** + * Everything this machine holds for one sitting, handed back to it. + * + * The browser's own store was the only place a reader's *place* and their drawn + * corrections lived, and it turns out to be the least dependable thing in the chain: + * it is per-origin, so the tailnet name and the tailnet address are two different + * memories of the same sitting, and a browser is free to throw it away. This machine + * has had every answer since the first one. So it hands them back, and the browser + * store stops being the single copy of anything. + * + * Two halves, because they are different kinds of record and the page merges them: + * `banked` is the reduced snapshot a hand-over wrote — retractions already applied — + * and `log` is the raw append-only stream, retractions included as their own lines. + * Filtering is left to the page: one log covers every sitting, and only the page + * knows which marks are its own. + */ +function answersFor(name) { + const banked = []; + if (name) { + const path = join(DIR, name.replace(/[^a-zA-Z0-9._-]/g, "")); + if (path.startsWith(DIR) && existsSync(path)) { + try { + const doc = JSON.parse(readFileSync(path, "utf8")); + for (const ev of doc.said || []) banked.push(ev); + } catch { + // A half-written hand-over is not a reason to refuse the log as well. + } + } + } + + const log = []; + if (existsSync(LOG)) { + for (const line of readFileSync(LOG, "utf8").split("\n")) { + if (!line.trim()) continue; + try { + const rec = JSON.parse(line); + if (rec.payload) log.push(rec.payload); + } catch { + // One torn line at the tail is what append-only costs; the rest still reads. + } + } + } + return { banked, log }; +} + +createServer(async (req, res) => { + const url = new URL(req.url, `http://${req.headers.host || "x"}`); + + if (req.method === "GET" && url.pathname === "/api/answers") { + if (url.searchParams.get("t") !== TOKEN) return json(res, 403, { ok: false, error: "not this server's page" }); + const got = answersFor(url.searchParams.get("name") || ""); + return json(res, 200, { ok: true, ...got }); + } + + if (req.method === "POST" && url.pathname.startsWith("/api/")) { + if (url.searchParams.get("t") !== TOKEN) return json(res, 403, { ok: false, error: "not this server's page" }); + let sent; + try { + sent = JSON.parse(await body(req)); + } catch { + return json(res, 400, { ok: false, error: "not json" }); + } + + if (url.pathname === "/api/event") { + // Stamped here rather than trusted from the browser: the device has its own + // clock and its own timezone and no particular reason to be right, and a log + // whose times came from two sources cannot be read as a sequence. + appendFileSync(LOG, `${JSON.stringify({ t: new Date().toISOString(), ...sent })}\n`, "utf8"); + banked += 1; + process.stdout.write(`\r${banked} answers banked to ${LOG.split("/").pop()} `); + return json(res, 200, { ok: true }); + } + + if (url.pathname === "/api/artifact") { + const name = String(sent.name || "").replace(/[^a-zA-Z0-9._-]/g, ""); + if (!name) return json(res, 400, { ok: false, error: "no name" }); + const path = join(DIR, name); + writeFileSync(path, JSON.stringify(sent.json, null, 1), "utf8"); + console.log(`\nbanked a whole sitting to ${path}`); + return json(res, 200, { ok: true, path }); + } + return json(res, 404, { ok: false, error: "no such endpoint" }); + } + + // Everything else is a file. `normalize` before joining, so a path with .. in it + // cannot walk out of the directory being served. + const rel = normalize(decodeURIComponent(url.pathname)).replace(/^(\.\.[/\\])+/, ""); + let path = join(DIR, rel); + if (existsSync(path) && statSync(path).isDirectory()) path = join(path, "index.html"); + if (!path.startsWith(DIR) || !existsSync(path)) { + res.writeHead(404, { "content-type": "text/plain" }); + return res.end("not here"); + } + + const type = TYPES[extname(path)] || "application/octet-stream"; + if (extname(path) !== ".html") { + res.writeHead(200, { "content-type": type }); + return createReadStream(path).pipe(res); + } + + const html = readFileSync(path, "utf8"); + const withSink = inject(html); + if (!withSink) console.log(`\n${rel}: nowhere to put the sink — served as-is, it will download instead`); + res.writeHead(200, { "content-type": type, "cache-control": "no-store" }); + res.end(withSink || html); +}).listen(PORT, HOST, () => { + console.log(`serving ${DIR}`); + console.log(` http://${HOST}:${PORT}/index.html`); + console.log(` answers append to ${LOG}`); +}); diff --git a/packages/etl/scripts/serve-sittings.test.mjs b/packages/etl/scripts/serve-sittings.test.mjs new file mode 100644 index 0000000..f44b158 --- /dev/null +++ b/packages/etl/scripts/serve-sittings.test.mjs @@ -0,0 +1,118 @@ +/** + * The server that gives a sitting back to the person who was in the middle of it. + * + * The reason this file exists is a bug that cost a reader most of an evening twice + * over. The browser's own store was the only place a sitting's *place* and its drawn + * corrections lived, and a browser store is the least dependable link in the chain: + * it is per-origin, so opening this machine by its tailnet name and by its tailnet + * address are two different memories of the same sitting, and neither can see the + * other. A reader who arrived by the other address was handed a sitting that had + * forgotten them — card one, every box back where it shipped — while every answer + * they had given sat safely on the machine that served them the page. + * + * So the machine hands them back, and these hold it to that. + */ +import { execFileSync } from "node:child_process"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { setTimeout as wait } from "node:timers/promises"; +import { spawn } from "node:child_process"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; + +const SCRIPT = new URL("./serve-sittings.mjs", import.meta.url).pathname; + +const AN = { kind: "placement", id: "13:35", page: 13, line: 2, name: "fatha", rule: "line-tilt", by: [0.5, 0], to: [1, 2] }; +const OTHER = { kind: "wrong-shape", id: "99:1", page: 99, line: 4, name: "sukun", rule: "line-tilt", size: [6, 4] }; + +let dir; +let proc; +let base; +let token; + +beforeAll(async () => { + dir = mkdtempSync(join(tmpdir(), "hifth-serve-")); + // A hand-over that has already been banked, and a live log holding one answer that + // came after it plus one from a different sitting entirely — because one log covers + // every sitting and the endpoint is deliberately not the thing that separates them. + writeFileSync(join(dir, "mark-report-fallback-p1of16.23.json"), JSON.stringify({ said: [AN] })); + writeFileSync( + join(dir, "answers.jsonl"), + [AN, OTHER].map((p) => JSON.stringify({ t: "2026-08-14T04:05:24.478Z", kind: "report", payload: p })).join("\n") + "\n", + ); + + // Port 0 is not offered: the script takes a number and prints where it landed, so + // a fixed high port is simpler than parsing. Collisions fail loudly rather than + // silently talking to somebody else's server, which is the failure worth avoiding. + base = "http://127.0.0.1:4187"; + proc = spawn(process.execPath, [SCRIPT, "--dir", dir, "--host", "127.0.0.1", "--port", "4187", "--log", "answers.jsonl"], { + stdio: ["ignore", "pipe", "pipe"], + }); + writeFileSync(join(dir, "probe.html"), "x"); + for (let i = 0; i < 100 && !token; i += 1) { + await wait(50); + try { + // The token is only readable from a page this server served, which is the whole + // of what it is for. + const page = await (await fetch(`${base}/probe.html`)).text(); + token = /t=([a-f0-9]+)/.exec(page)?.[1] || null; + } catch { + // not up yet + } + } +}); + +afterAll(() => { + proc?.kill(); + rmSync(dir, { recursive: true, force: true }); +}); + +describe("handing a sitting back its own answers", () => { + it("serves a page with somewhere for its answers to go", async () => { + const html = await (await fetch(`${base}/probe.html`)).text(); + expect(html).toContain("HIFTH_SESSION"); + expect(html).toContain("/api/answers"); + // Ahead of the page's own scripts, and never ahead of the doctype — prepending + // there puts the page in quirks mode, breaking its layout to fix its reporting. + expect(html.indexOf("HIFTH_SESSION")).toBeGreaterThan(html.indexOf("")); + }); + + it("gives back the hand-over and the live log as two separate records", async () => { + const t = token; + const r = await (await fetch(`${base}/api/answers?t=${t}&name=mark-report-fallback-p1of16.23.json`)).json(); + expect(r.ok).toBe(true); + // Separate because they are different kinds of record: a hand-over is a snapshot + // with its retractions already applied, and the log is the raw stream with the + // retractions still in it as their own lines. Only the page can merge them. + expect(r.banked).toEqual([AN]); + expect(r.log).toEqual([AN, OTHER]); + }); + + it("still answers when no hand-over was ever banked", async () => { + const t = token; + const r = await (await fetch(`${base}/api/answers?t=${t}&name=nothing-was-ever-written.json`)).json(); + // A reader who has never pressed hand-over has still given answers, and the log + // is the whole of what they are owed. Refusing here would hand them nothing. + expect(r.ok).toBe(true); + expect(r.banked).toEqual([]); + expect(r.log.length).toBe(2); + }); + + it("will not read a sitting out to a page it did not serve", async () => { + const r = await (await fetch(`${base}/api/answers?t=deadbeef&name=mark-report-fallback-p1of16.23.json`)).json(); + expect(r.ok).toBe(false); + }); + + it("cannot be walked out of the directory it is serving", async () => { + const t = token; + const r = await (await fetch(`${base}/api/answers?t=${t}&name=${encodeURIComponent("../../etc/hosts")}`)).json(); + expect(r.ok).toBe(true); + expect(r.banked).toEqual([]); + }); +}); + +describe("the script itself", () => { + it("parses", () => { + expect(() => execFileSync(process.execPath, ["--check", SCRIPT])).not.toThrow(); + }); +}); diff --git a/scripts/build-placement-options.mjs b/scripts/build-placement-options.mjs new file mode 100644 index 0000000..96381b3 --- /dev/null +++ b/scripts/build-placement-options.mjs @@ -0,0 +1,1558 @@ +#!/usr/bin/env node +/** + * Render docs/design/mark-placement.html — where the app should put the + * rectangles it draws over the print, with each answer drawn on a real page of + * the mus'haf at the size it would actually be used. + * + * ── Why this is a page and not a section of mark-registration.md ──────────── + * The write-up already carries the measurement, and it is not in dispute: the + * frame every rectangle rides on is fitted on the verse-end ornaments, the only + * objects the two prints both label, and it is excellent there and about a + * mark-height wrong at the text. What the write-up cannot do is answer the + * question that is actually open, which is how much correction is worth its + * cost — and that is a question about what a person sees on a screen, at the + * size they see it. A table of overlap fractions cannot be looked at. + * + * So the options are drawn. Every rectangle on this page is the rectangle the + * app would ship under that option, computed by the same code the shards would + * be built with, over the same print the app serves. + * + * ── Two modes, because the inputs live in two different places ────────────── + * node scripts/build-placement-options.mjs # render (repo bytes) + * node scripts/build-placement-options.mjs --extract # refresh the data file + * + * The render reads only committed bytes — the page SVG, the word shard, the + * manifest, the stylesheet — plus `mark-placement.data.json`, so anyone with a + * checkout rebuilds the identical file. + * + * `--extract` is maintainer-only twice over. It needs the 348 MB gitignored + * fetch cache under `packages/etl/data/pages/.cache/` for the ligature corpus, + * and it needs a per-mark displacement file — the one `probe-mark-ink.mjs` + * writes with `--rows-out`, which takes about an hour over all 604 pages and is + * far too large to commit. Both are named on the command line; without either + * it refuses rather than emitting a thinner data file, because a data file that + * is sometimes complete is worse than one that is absent. + * + * node scripts/build-placement-options.mjs --extract --rows + * + * That one file is the whole input. The held-out grades used to be read out of + * four separate scorer runs named with a `--held-out` flag, and that was worse + * in both directions: the four could silently be over different page sets, and + * omitting the flag emitted a data file with no held-out block at all — which + * every consumer then fell back out of, into figures graded on their own + * training marks, under captions still saying they were held out. They are now + * computed here, from these rows, on the same split for every rung. + * + * ── There is no Quran text in this file, or in the one it writes ──────────── + * Same property as `build-mark-options.mjs`, and for the same reason: the data + * file is rectangles and Latin mark names, the print is referenced rather than + * inlined, and the print itself is outlined `` with zero Arabic + * codepoints. A reader sees the word because the mus'haf draws it. + */ +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import { basename, join } from "node:path"; + +const ROOT = new URL("..", import.meta.url).pathname; +const DATA = join(ROOT, "docs/design/mark-placement.data.json"); +const OUT = join(ROOT, "docs/design/mark-placement.html"); +const OUT_ARTIFACT = join(ROOT, "docs/design/mark-placement.artifact.html"); + +/** + * Two copies, and one run writes both. They differ in exactly one thing: where + * the picture of the mus'haf comes from. + * + * The checked-in copy points at the print with a relative URL, which is right + * for somebody who opens the file inside a checkout and keeps it small. The + * published copy cannot do that — it is served from claude.ai under a CSP that + * blocks every external host, so the relative URL resolves to nothing and every + * specimen is an empty overlay over blank paper. That copy inlines the print + * ONCE as a `` and ``s it per specimen. + * + * This used to be a `--artifact` flag, and the flag is why the published page + * lost its subject: the two copies were produced by two hand-run commands, so + * the checked-in one could be rebuilt and look perfect while the published one + * stayed twenty minutes behind, missing an option. A reader of the repo has no + * way to notice — the copy they open is the one that is right. Writing both + * from one pass is not a convenience; it removes the only state in which the + * repo lies about what was published. + */ +const COPIES = [ + { artifact: false, out: OUT }, + { artifact: true, out: OUT_ARTIFACT }, +]; + +/** How far a rectangle may sit from its own ink before it is "badly out". */ +const FAR = 0.75; + +const GRAINS = ["shipped", "page", "line", "tilt"]; + +/** + * When is the mark's own ink search allowed to place it? — option H's guard. + * + * Two refusals, and both were found by measuring rather than by worrying. The + * fear that a per-mark search would snap onto the *neighbouring* mark turns out + * to be very nearly groundless on this print: the nearest mark of any name is a + * median 8.32 units away and the nearest one that looks identical is 24.80, + * against a search that reaches 3 and an error near 1. Across all 326,515 marks + * the search lands nearer some other mark 1.09% of the time and nearer an + * identical-looking one 0.01% — 48 marks in the whole mus'haf. That objection is + * retired; it was the main argument against this option and it did not survive + * contact with the numbers. + * + * What the measurement does refuse is different and sharper. Where the match is + * weak the search is chasing noise: below 0.55 it departs from the printed + * line's own answer by a median 1.7 to 2.8 units and throws 44–72% of marks more + * than 2 units, against a median 0.21 and 0.2% above the threshold. And a search + * that stops at its own boundary has not found anything — it has run out of room, + * so its answer is a fact about the window and not about the ink. Those two sets + * do not get placed from ink; they inherit the printed line, which is the whole + * reason this is an option and not a hazard. + * + * The boundary test is per axis, and getting that wrong cost this page a set of + * numbers. The search slides the mark's outline over a *square* window, ±radius + * in each direction independently, so it ran out of room exactly when one of the + * two components came back sitting on ±radius. An earlier version of this line + * asked whether the straight-line distance exceeded the radius, which draws a + * circle inside that square and throws away every corner of it — 8,358 marks + * whose match was a median 0.905 against an achievable 0.909, discarded under a + * caption saying the search had run out of room when it had not been near its + * edge. Distance travelled is not evidence of a bad placement; a weak match and + * a pinned search are. + */ +const TRUST = { iou: 0.55, radius: 3 }; +const atWindowEdge = (r) => + Math.abs(Math.abs(r.dx) - TRUST.radius) < 1e-6 || Math.abs(Math.abs(r.dy) - TRUST.radius) < 1e-6; +const trusted = (r) => r.iouBest >= TRUST.iou && !atWindowEdge(r); + +/** The grain names the correction builder knows, keyed by ours. */ +const GRAIN_ARG = { page: "page", line: "line", tilt: "line-tilt" }; + +const argOf = (name, fallback = null) => { + const i = process.argv.indexOf(name); + return i === -1 ? fallback : (process.argv[i + 1] ?? fallback); +}; + +const r3 = (n) => Math.round(n * 1000) / 1000; +const r2 = (n) => Math.round(n * 100) / 100; + +// ───────────────────────────────────────────────────────────── extract ────── + +async function extract() { + const ETL = join(ROOT, "packages/etl/scripts"); + const cache = join(ROOT, "packages/etl/data/pages/.cache"); + const rowsPath = argOf("--rows"); + if (!existsSync(cache)) { + console.error("--extract needs the quran-svg fetch cache at packages/etl/data/pages/.cache/"); + console.error("It is gitignored and 348 MB. Run the vendor step first, or drop --extract."); + process.exit(1); + } + if (!rowsPath || !existsSync(rowsPath)) { + console.error("--extract needs --rows , the per-mark displacements."); + console.error(" node packages/etl/scripts/probe-mark-ink.mjs --pages-n 604 --sample 400000 \\"); + console.error(" --grain line-tilt --out --shift-out --rows-out "); + process.exit(1); + } + + const { candidatePage } = await import(join(ETL, "lib/candidate-pages.mjs")); + const { applierFromPin, readDiacritics } = await import(join(ETL, "lib/diacritics.mjs")); + const { readOurs, readTheirs, readingOrder } = await import(join(ETL, "lib/mushaf-frame.mjs")); + const { correctionFor, shuffledCorrectionFor, splitHalfLadder, residualsUnder, half } = await import( + join(ETL, "lib/registration-grain.mjs") + ); + + const rows = JSON.parse(readFileSync(rowsPath, "utf8")); + const byPage = new Map(); + for (const r of rows) { + if (!byPage.has(r.page)) byPage.set(r.page, []); + byPage.get(r.page).push(r); + } + + /** How far each mark still is from its own ink, once a correction is applied. */ + const residuals = (rs, corr) => + rs.map((r) => { + const a = corr.apply(r); + return Math.hypot(r.dx - a.dx, r.dy - a.dy); + }); + const share = (xs) => xs.filter((v) => v > FAR).length / xs.length; + const median = (xs) => { + const s = [...xs].sort((a, b) => a - b); + return s.length ? s[Math.floor(s.length / 2)] : 0; + }; + + const NONE = { apply: () => ({ dx: 0, dy: 0 }) }; + const corrections = (rs) => ({ + shipped: NONE, + page: correctionFor("page", rs), + line: correctionFor("line", rs), + tilt: correctionFor("line-tilt", rs), + }); + + // ── the corpus, one row per page, so the page we draw can be placed in it ── + const perPage = []; + for (const [page, rs] of byPage) { + const c = corrections(rs); + const row = { page, n: rs.length, lines: new Set(rs.map((r) => r.line)).size }; + for (const g of GRAINS) row[g] = share(residuals(rs, c[g])); + perPage.push(row); + } + perPage.sort((a, b) => a.page - b.page); + + const corpusMedian = {}; + for (const g of GRAINS) corpusMedian[g] = r3(median(perPage.map((p) => p[g]))); + const corpusAll = {}; + { + // Pooled over every mark rather than averaged over pages: a page with more + // marks should weigh more, and the held-out figures are pooled too. + const all = []; + for (const g of GRAINS) all.push([g, []]); + const acc = new Map(all); + for (const [, rs] of byPage) { + const c = corrections(rs); + for (const g of GRAINS) acc.get(g).push(...residuals(rs, c[g])); + } + for (const g of GRAINS) { + corpusAll[g] = { far: r3(share(acc.get(g))), p50: r3(median(acc.get(g))) }; + } + } + + // The control that says a per-line correction is not a richer model flattering + // itself: give every line another line's correction and it gets worse than + // applying no per-line correction at all. + const shuffled = []; + for (const [, rs] of byPage) shuffled.push(...residuals(rs, shuffledCorrectionFor("line-tilt", rs))); + const shuffle = r3(share(shuffled)); + + // ── the page we draw ─────────────────────────────────────────────────────── + const PAGE = Number(argOf("--page", "179")); + const drawn = byPage.get(PAGE); + if (!drawn) { + console.error(`page ${PAGE} is not in ${basename(rowsPath)}`); + process.exit(1); + } + const corr = corrections(drawn); + const at = (r) => { + const o = {}; + for (const g of GRAINS) { + if (g === "shipped") continue; + const a = corr[g].apply(r); + o[g] = [r3(a.dx), r3(a.dy)]; + } + // Option H, which is not a grain and cannot be built by the grain builder: + // where the mark's own ink match is convincing, the mark goes exactly where + // the search found it; where it is not, it inherits the printed line. The + // fallback is what makes this shippable rather than reckless — see TRUST. + const a = corr.tilt.apply(r); + o.mark = trusted(r) ? [r3(r.dx), r3(r.dy)] : [r3(a.dx), r3(a.dy)]; + return o; + }; + + const pin = new Map( + JSON.parse(readFileSync(join(ROOT, "packages/etl/data/pages/word-boxes.pin.json"), "utf8")).pages.map( + (p) => [p.page, p], + ), + ); + const pinRow = pin.get(PAGE); + const apply = applierFromPin(pinRow); + const theirSvg = (await candidatePage(PAGE, { offline: true })).body.toString("utf8"); + const ourSvg = readFileSync(join(ROOT, `apps/web/public/assets/pages/hafs-kfqc/${PAGE}.svg`), "utf8"); + + // Word boxes, and which printed line each was set on. The shipped shard is + // the geometry the app actually draws; the corpus is the only thing that + // knows the line, so the two are joined on the print's own word index. + const lineOf = new Map(); + for (const w of readDiacritics(theirSvg, apply)) lineOf.set(`${w.surah}:${w.aya}:${w.idx}`, w.line); + const shard = JSON.parse( + readFileSync(join(ROOT, `apps/web/public/assets/words/hafs-kfqc/${PAGE}.json`), "utf8"), + ); + const words = []; + for (const [key, entry] of Object.entries(shard.words ?? {})) { + entry.boxes.forEach((b, i) => { + const line = lineOf.get(`${key}:${entry.from + i}`); + if (!line) return; + const box = b.map(r2); + words.push({ l: line, b: box, o: at({ page: PAGE, line, box }) }); + }); + } + + // The verse-end circles: the objects the whole frame was fitted on, and + // therefore the one held-out witness on this page that is not made of ink. + const theirs = readingOrder( + readTheirs(theirSvg).marks.map((t) => { + const [x0, y0, x1, y1] = apply(t.box); + return [(x0 + x1) / 2, (y0 + y1) / 2]; + }), + 11, + ); + const ours = readingOrder(readOurs(ourSvg).marks, 11); + const markLineY = new Map(); + for (const r of drawn) { + const y = r.box[1] + r.box[3] / 2; + const list = markLineY.get(r.line) ?? []; + list.push(y); + markLineY.set(r.line, list); + } + const lineMidY = [...markLineY].map(([l, ys]) => [l, median(ys)]); + const ornaments = []; + if (theirs.length === ours.length) { + theirs.forEach((t, i) => { + // An ornament carries no line number of its own, so it is given the + // printed line whose marks sit nearest it down the page — which is the + // line whose correction would move it. + const line = lineMidY.reduce((best, [l, y]) => + Math.abs(y - t[1]) < Math.abs(best[1] - t[1]) ? [l, y] : best, + )[0]; + const box = [t[0] - 2, t[1] - 2, 4, 4]; + ornaments.push({ + l: line, + fit: [r2(t[0]), r2(t[1])], + ours: [r2(ours[i][0]), r2(ours[i][1])], + o: at({ page: PAGE, line, box }), + }); + }); + } + + // ── the held-out ladder ─────────────────────────────────────────────────── + // Every figure the page quotes as evidence comes from here, and it is computed + // in this process from the same rows the rest of the file reads. It used to be + // read out of four separate scorer runs named on the command line, which was + // wrong twice: the four could be over different page sets and nothing said so, + // and forgetting the flag left the block empty, at which point the page quietly + // fell back to figures graded on their own training marks while still captioned + // as held out. Computing it here means it cannot go missing and it cannot + // disagree with the corpus figures beside it. + // + // The split is `half()`, which keys on the mark's own identity rather than its + // position, so the same mark is on the same side of the split in every run and + // in every rung. `shipped` is scored on the same held-out half as the rest — + // it has no correction to fit, but it must be graded on the same marks or the + // first row of the ladder is not comparable with the ones under it. + const wilson = (k, n) => { + if (!n) return []; + const z = 1.96; + const p = k / n; + const d = 1 + (z * z) / n; + const c = p + (z * z) / (2 * n); + const s = z * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n)); + return [r2(100 * ((c - s) / d)), r2(100 * ((c + s) / d))]; + }; + + const heldOut = {}; + for (const g of GRAINS) { + const rung = + g === "shipped" + ? { groups: { pages: byPage.size }, heldOut: residualsUnder(rows.filter((r) => half(r) === 1), NONE.apply), trained: residualsUnder(rows.filter((r) => half(r) === 0), NONE.apply) } + : splitHalfLadder(rows, GRAIN_ARG[g]); + const h = rung.heldOut; + heldOut[g] = { + pages: rung.groups.pages, + scored: h.n, + far: r2(h.over), + ci: wilson(Math.round((h.over / 100) * h.n), h.n), + p50: r3(h.med), + p95: r3(h.p95), + // The gap to the same correction graded on the marks it was fitted from. + // A rung that only wins on its own training marks has earned nothing, so + // the overfit is carried beside the figure rather than left to be trusted. + trained: r2(rung.trained.over), + }; + } + // The shuffle control, graded on the same held-out half as the rungs it is + // compared against. It was previously only available fitted-and-scored on + // every mark, which meant the page put a trained figure next to a held-out one + // and invited a reader to divide them. + heldOut.shuffle = r2(splitHalfLadder(rows, "line-tilt", { shuffled: true }).heldOut.over); + + // ── option H, which this instrument cannot grade and must still describe ─── + // + // Every rung above is a model fitted on half the marks and graded on the other + // half. H is not a model: on the marks it accepts it ships the measurement + // itself, so its residual there is zero by construction and that zero carries + // no information whatever — a fact stated on the page rather than dressed up + // as a win. What CAN be said honestly is two things, and both are here: how + // much of the book it places from direct evidence, and how the part it refuses + // to place that way scores under the model it falls back to. The second is the + // only error H can be blamed for by this measurement, and it is not small on + // its own terms, which is exactly why the fallback set is named and counted + // rather than averaged away. + { + const acc = rows.filter(trusted); + const fall = rows.filter((r) => !trusted(r)); + const { apply } = correctionFor("line-tilt", rows); + const resid = (rs) => { + const d = rs.map((r) => { + const a = apply(r); + return Math.hypot(r.dx - a.dx, r.dy - a.dy); + }).sort((x, y) => x - y); + const at2 = (p) => d[Math.min(d.length - 1, Math.floor(p * d.length))] ?? 0; + return { far: r2((100 * d.filter((x) => x > FAR).length) / d.length), p50: r3(at2(0.5)), p95: r3(at2(0.95)) }; + }; + const f = resid(fall); + heldOut.mark = { + placed: acc.length, + fellBack: fall.length, + share: r2((100 * acc.length) / rows.length), + weak: rows.filter((r) => r.iouBest < TRUST.iou).length, + clamped: rows.filter(atWindowEdge).length, + fallback: f, + // What H leaves badly out across the whole book, counting only the marks + // it did not place from ink — the accepted ones contribute a zero this + // instrument is not entitled to claim. + far: r2((f.far * fall.length) / rows.length), + }; + } + + // ── how close together the marks actually are ────────────────────────────── + // + // This answers the objection that killed the per-mark idea the first time it + // was written up: that a search for the nearest ink would find the *neighbour's* + // and centre a rectangle neatly on the wrong mark. It was asserted from a guess + // that adjacent marks sit about a unit apart. They do not, and the page now + // prints the real distances instead of the guess. Measured per page against + // every other mark on that page, from box centre to box centre — the same + // frame everything else here is in. + const neighbour = (() => { + const cen = (r) => [r.box[0] + r.box[2] / 2, r.box[1] + r.box[3] / 2]; + const any = []; + const same = []; + let jumpAny = 0; + let jumpSame = 0; + for (const [, rs] of byPage) { + const pts = rs.map(cen); + for (let i = 0; i < rs.length; i++) { + let dAny = Infinity; + let dSame = Infinity; + let nAny = null; + let nSame = null; + for (let j = 0; j < rs.length; j++) { + if (i === j) continue; + const g = Math.hypot(pts[i][0] - pts[j][0], pts[i][1] - pts[j][1]); + if (g < dAny) { dAny = g; nAny = j; } + if (rs[j].name === rs[i].name && g < dSame) { dSame = g; nSame = j; } + } + if (nAny !== null) any.push(dAny); + if (nSame !== null) same.push(dSame); + // Where the search actually ended up, against where the neighbours are. + // A "jump" is a landing that is closer to another mark's centre than to + // its own — the failure the objection predicted, counted rather than feared. + const lx = pts[i][0] + rs[i].dx; + const ly = pts[i][1] + rs[i].dy; + const own = Math.hypot(rs[i].dx, rs[i].dy); + if (nAny !== null && Math.hypot(lx - pts[nAny][0], ly - pts[nAny][1]) < own) jumpAny++; + if (nSame !== null && Math.hypot(lx - pts[nSame][0], ly - pts[nSame][1]) < own) jumpSame++; + } + } + const med = (a) => { a.sort((x, y) => x - y); return r2(a[Math.floor(a.length / 2)] ?? 0); }; + return { + anyP50: med(any), + sameP50: med(same), + radius: TRUST.radius, + jumpAny: r2((100 * jumpAny) / rows.length), + jumpSame: r3((100 * jumpSame) / rows.length), + jumpSameN: jumpSame, + }; + })(); + + const data = { + $comment: [ + "Generated by scripts/build-placement-options.mjs --extract. Geometry only — no Quran text.", + "Coordinates are the page frame the word shards use. `b` is the rectangle the app ships", + "today; `o` is how far each option would move it, and `ink` how far it would have to move", + "to sit on the printer's own ink. Mark names are Latin; words are referred to by the", + "print's own word index and never by their text.", + ], + ran: new Date().toISOString().slice(0, 10), + page: PAGE, + viewBox: (ourSvg.match(/viewBox="([^"]*)"/)?.[1] ?? "0 0 345 550").trim(), + far: FAR, + pin: { markers: pinRow.markers, residual: pinRow.residual, words: pinRow.words }, + marks: drawn.map((r) => ({ + n: r.name, + l: r.line, + b: r.box.map(r2), + ink: [r3(r.dx), r3(r.dy)], + // Whether option H is willing to place this mark from its own ink. Kept + // per mark rather than recomputed in the renderer so the picture and the + // count in §7 cannot drift apart. + t: trusted(r) ? 1 : 0, + o: at(r), + })), + words, + ornaments, + drawnPage: { + ...Object.fromEntries(GRAINS.map((g) => [g, r3(share(residuals(drawn, corr[g])))])), + // H on the drawn page. A mark it accepts contributes a zero, because it is + // placed at the very thing this column measures distance to — which is the + // honest arithmetic and also the reason the caption beside it says so. + mark: r3( + share( + drawn.map((r) => { + if (trusted(r)) return 0; + const a = corr.tilt.apply(r); + return Math.hypot(r.dx - a.dx, r.dy - a.dy); + }), + ), + ), + }, + corpus: { + pages: perPage.length, + marks: rows.length, + pooled: corpusAll, + medianPage: corpusMedian, + shuffle, + lines: perPage.reduce((t, p) => t + p.lines, 0), + }, + heldOut, + neighbour, + unmeasured: [1, 2, 603, 604], + }; + writeFileSync(DATA, `${JSON.stringify(data, null, 1)}\n`); + console.log( + `--extract — page ${PAGE}: ${data.marks.length} marks, ${words.length} words, ` + + `${ornaments.length} verse-end circles → ${basename(DATA)}`, + ); + console.log( + ` corpus ${perPage.length} pages, ${rows.length.toLocaleString("en")} marks: ` + + GRAINS.map((g) => `${g} ${(corpusAll[g].far * 100).toFixed(1)}%`).join(" · ") + + ` · shuffled lines ${(shuffle * 100).toFixed(1)}%`, + ); +} + +// ────────────────────────────────────────────────────────────── render ────── + +const esc = (s) => + String(s).replace(/&/g, "&").replace(//g, ">").replace(/"/g, """); +const pctS = (n) => `${(n * 100).toFixed(1)}%`; + +/** Colours read out of the app's own stylesheet, so the picture cannot drift. */ +function palette() { + const css = readFileSync(join(ROOT, "apps/web/src/styles/tokens.css"), "utf8"); + const one = (name) => css.match(new RegExp(`--${name}:\\s*([^;]+);`))?.[1]?.trim() ?? null; + return { + mark: one("tj-ghunnah") ?? "#7a4fb5", + word: one("tj-madd") ?? "#1f6f66", + ink: one("tj-qalqalah") ?? "#b5522f", + }; +} + +// The letters are the record's, not this page's. The record enumerated six +// options in the order they were written and this page draws four of them, so +// numbering them A–D here would have given the same option two names depending +// on which document a reader happened to open. C and D and E stay in the record +// and are named in §⑩ below; the two that are only in the record are the ones +// nothing would be gained by drawing. +const OPTIONS = [ + { + id: "A", + grain: "shipped", + title: "Leave the rectangles where they are", + tag: "what the app does today", + what: "Nothing changes. The rectangle for every mark and every word stays where the fit on the verse-end circles put it.", + }, + { + id: "B", + grain: "page", + title: "Line each page up as a whole", + tag: "two more numbers a page", + what: "Every rectangle on a page moves together, by one amount worked out from that page's own marks.", + }, + { + id: "F", + grain: "tilt", + title: "Line each printed line up on its own, and let it tilt", + tag: "the recommendation — four more numbers a line", + what: "Each printed line gets its own amount, and that amount is allowed to grow along the line rather than being the same at both ends. Marks and words move together.", + }, + { + id: "G", + grain: "tilt", + title: "Line the marks up, and leave the words where they are", + tag: "the same, for half the page's geometry", + what: "The same per-line correction, applied only to the marks. Word rectangles stay on the fit they ship on today.", + wordsStay: true, + }, + { + id: "H", + grain: "mark", + title: "Put each mark where its own ink is, and line up the rest", + tag: "the most accurate — and the one this measurement cannot grade", + what: "Every mark whose ink was found convincingly goes exactly where it was found. The rest — and every word rectangle — get the per-line correction instead.", + }, +]; + +// The tradeoffs, as a board rather than a table. Section 7 already grades every +// option on accuracy, and accuracy is the one axis on which F and G are the same +// number — so a page that stopped there would show a reader two identical rows +// and no reason to prefer either. What separates them is entirely cost: what has +// to be rebuilt, and what has to be kept in step forever afterwards. Those are +// prose facts, so they are stored as prose here and drawn beside the measured +// bar, which is the only figure on the board that comes out of the data. +// `tone` is semantic and deliberately not the page's accent: three muted hues +// that survive both themes, so severity reads before any of the words do. +const TRADEOFFS = { + A: { + verdict: "what the app does today", + build: { + tone: "good", + head: "Nothing", + body: "No new numbers to work out, nothing to write down, nothing to rebuild.", + }, + disturbs: { + tone: "good", + head: "Nothing", + body: "Every rectangle in the app stays exactly where it is.", + }, + risk: { + tone: "bad", + head: "The complaint stands", + body: "Almost every rectangle sits about a mark's height from the mark it is meant to name. Colouring one exact letter cannot be built on rectangles this far out — it would put the colour on the letter next door.", + }, + undo: { tone: "good", head: "Nothing to undo", body: "" }, + }, + B: { + verdict: "better, and measurably not enough", + build: { + tone: "good", + head: "Two numbers a page", + body: "About PAGES2 numbers for the whole book, in a table the app already has a place for.", + }, + disturbs: { + tone: "warn", + head: "Every word rectangle", + body: "Moving a page moves the words on it too, so every word rectangle the app ships is rebuilt and re-checked, and the pictures the tests compare against may move.", + }, + risk: { + tone: "warn", + head: "About one rectangle in six is still badly out", + body: "And that is the ceiling, not a first attempt: the best correction of this shape that could be fitted to a page from its own ink does not do better. Adding a stretch to it buys under two percent and makes a third of pages worse.", + }, + undo: { tone: "warn", head: "Reversible, not free", body: "Dropping the table means rebuilding everything it moved." }, + }, + F: { + verdict: "the recommendation", + build: { + tone: "warn", + head: "Four numbers a printed line", + body: "About LINES4 numbers for the whole book, in a table that does not exist yet — both the part that builds the rectangles and the part that reads them have to learn it.", + }, + disturbs: { + tone: "warn", + head: "Every word rectangle", + body: "Same as the option above: the words move with the marks, so all of them are rebuilt and re-checked and the comparison pictures may move.", + }, + risk: { + tone: "warn", + head: "What is left over has changed character", + body: "Not just shrunk. After this correction a small minority of rectangles are badly out for some reason of their own, rather than every rectangle being slightly out — and nobody has yet looked for what that reason is.", + }, + undo: { tone: "warn", head: "Reversible, not free", body: "Dropping the table means rebuilding everything it moved." }, + }, + G: { + verdict: "the fork worth noticing", + build: { + tone: "warn", + head: "The same table", + body: "Exactly the same numbers as the option above. The only difference is what they are applied to.", + }, + disturbs: { + tone: "good", + head: "Only the marks", + body: "No word rectangle moves, so nothing the app already ships for words is rebuilt, re-checked, or re-photographed.", + }, + risk: { + tone: "bad", + head: "Two geometries from one measurement, forever", + body: "The marks would sit on a lining-up we measured, and the words on one we have measured as wrong — and something has to hold the two to each other from then on. The saving is real and it is one-off; the obligation is permanent.", + }, + undo: { tone: "good", head: "Easiest to undo", body: "Nothing shipped has moved, so dropping it rebuilds nothing." }, + }, + H: { + verdict: "the most accurate, and the least provable", + build: { + tone: "bad", + head: "Two numbers for every mark", + body: "MARKS2 numbers for the whole book — not a rule that explains anything, but one measurement of one printing, written down in full. It cannot live in the small table the app reads today and needs a file of its own beside it.", + }, + disturbs: { + tone: "warn", + head: "Every word rectangle, still", + body: "The marks go where their own ink is, but words and verse-end circles have no such measurement, so they take the per-line correction — which means they move, are rebuilt, and are re-checked exactly as in the two options above.", + }, + risk: { + tone: "bad", + head: "It cannot be graded, and it says nothing about an unmeasured page", + body: "Every other option can be marked on marks it never saw. This one has nothing held back, so the only honest check is a person looking at rectangles and saying whether they sit right. And because it records rather than explains, it predicts nothing at all about a printing nobody has measured.", + }, + undo: { tone: "warn", head: "Reversible, not free", body: "Dropping the table means rebuilding everything it moved — the same cost as the two options above." }, + }, +}; + +function render({ artifact: ARTIFACT, out }) { + if (!existsSync(DATA)) { + console.error(`missing ${basename(DATA)} — run with --extract first (needs the maintainer cache)`); + process.exit(1); + } + const d = JSON.parse(readFileSync(DATA, "utf8")); + const pal = palette(); + const print = `../../apps/web/public/assets/pages/hafs-kfqc/${d.page}.svg`; + const [, , pw, ph] = d.viewBox.split(/\s+/).map(Number); + + const printDefs = ARTIFACT + ? `` + : ""; + + /** A rectangle, moved by whatever the option would move it by. */ + const moved = (o, grain) => { + const [x, y, w, h] = o.b; + if (grain === "shipped") return [x, y, w, h]; + const [dx, dy] = o.o[grain] ?? [0, 0]; + return [x + dx, y + dy, w, h]; + }; + const rect = ([x, y, w, h], attrs) => + ``; + + const markAttrs = `fill="${pal.mark}" fill-opacity="0.22" stroke="${pal.mark}" stroke-width="0.22" rx="0.3"`; + const wordAttrs = `fill="none" stroke="${pal.word}" stroke-width="0.22"`; + + /** + * A rectangle nowhere near the crop is a rectangle nobody will see, and every + * specimen would otherwise carry all 692 of them. Clipping to the crop is what + * keeps this page a hundred kilobytes rather than a megabyte. + */ + const inCrop = (crop) => { + if (!crop) return () => true; + const [vx, vy, vw, vh] = crop.split(/\s+/).map(Number); + return ([x, y, w, h]) => x + w > vx - 2 && x < vx + vw + 2 && y + h > vy - 2 && y < vy + vh + 2; + }; + + /** The layer one option would draw. */ + const layer = (opt, crop = null) => { + const keep = inCrop(crop); + const out = []; + for (const w of d.words) { + const b = moved(w, opt.wordsStay ? "shipped" : opt.grain); + if (keep(b)) out.push(rect(b, wordAttrs)); + } + for (const m of d.marks) { + const b = moved(m, opt.grain); + if (keep(b)) out.push(rect(b, markAttrs)); + } + return out.join(""); + }; + + /** Where the ink actually is: the rectangle moved onto its own ink. */ + const layerInk = (crop = null) => { + const keep = inCrop(crop); + return d.marks + .map((m) => [m.b[0] + m.ink[0], m.b[1] + m.ink[1], m.b[2], m.b[3]]) + .filter(keep) + .map((b) => + rect(b, `fill="none" stroke="${pal.ink}" stroke-width="0.22" stroke-dasharray="0.8 0.6" rx="0.3"`), + ) + .join(""); + }; + + /** The verse-end circles, and where an option would move them to. */ + const layerOrnaments = (grain) => + d.ornaments + .map((o) => { + const [dx, dy] = grain === "shipped" ? [0, 0] : (o.o[grain] ?? [0, 0]); + const a = ``; + const b = ``; + return a + b; + }) + .join(""); + + const specimen = (inner, { crop = null, label = "", cls = "" } = {}) => { + const [vx, vy, vw, vh] = (crop ?? d.viewBox).split(/\s+/).map(Number); + const pc = (n) => `${(n * 100).toFixed(4)}%`; + const print2 = ARTIFACT ? `` : ""; + return `
    +
    + ${ARTIFACT ? "" : ``} + +
    + ${label ? `
    ${label}
    ` : ""} +
    `; + }; + + // ── the crops, chosen from the drawn page's own geometry ────────────────── + // + // A whole page shows what the reader is being asked about; a band shows three + // printed lines at about the size a phone draws them; a close crop is the + // only size at which the difference between the last two options is visible + // at all, and saying that plainly is more honest than picking a crop that + // flatters the recommendation. + const lines = [...new Set(d.marks.map((m) => m.l))].sort((a, b) => a - b); + const midLine = lines[Math.floor(lines.length / 2)]; + const onMid = d.marks.filter((m) => m.l === midLine); + // The band is the true extent of three printed lines — measured off their own + // boxes rather than guessed as a multiple of the line spacing, which left a + // sliver of a fourth line clipped along the top edge. + const band = [midLine - 1, midLine, midLine + 1]; + const all = [...d.marks, ...d.words]; + const inBand = all.filter((m) => band.includes(m.l)); + const onLine = (l) => all.filter((m) => m.l === l); + // The window is cut through the white space between printed lines, not tight + // to the band's own ink: a word box is as tall as its line, so a crop that + // stops at the band's edge lets the next line's boxes lean into the picture. + const gapAbove = onLine(midLine - 2), gapBelow = onLine(midLine + 2); + const bandTop = Math.min(...inBand.map((m) => m.b[1])); + const bandBot = Math.max(...inBand.map((m) => m.b[1] + m.b[3])); + const top = gapAbove.length + ? (bandTop + Math.max(...gapAbove.map((m) => m.b[1] + m.b[3]))) / 2 + : bandTop - 2; + const bot = gapBelow.length + ? (bandBot + Math.min(...gapBelow.map((m) => m.b[1]))) / 2 + : bandBot + 2; + const BAND = `8 ${top.toFixed(1)} ${(pw - 16).toFixed(1)} ${(bot - top).toFixed(1)}`; + // The close crop sits at the mark on the middle line the tilt has moved + // furthest, so it shows the biggest honest difference between the last two + // options rather than an average one — and the caption says which. It has to + // be a mark with a page around it: the furthest-moved mark on this page sits + // hard against the outer margin, and a window half of which is off the paper + // shows a fragment nobody can read. + const CW = 34, CH = 17; + const roomy = onMid.filter( + (m) => m.b[0] > CW / 2 + 4 && m.b[0] < pw - CW / 2 - 4, + ); + const far = [...(roomy.length ? roomy : onMid)].sort( + (a, b) => Math.hypot(...b.o.tilt) - Math.hypot(...a.o.tilt), + )[0] ?? onMid[0]; + const CLOSE = `${(far.b[0] - CW / 2).toFixed(1)} ${(far.b[1] - CH / 2).toFixed(1)} ${CW} ${CH}`; + // How much bigger the close crop is than the band beside it, stated rather + // than asserted — the two are drawn to the same width on the page. + const closeMag = Math.round((pw - 16) / CW); + + const share = (g) => d.drawnPage[g]; + const ho = (g) => d.heldOut[g] ?? null; + const c = d.corpus; + // How big a mark actually is, so the distances in section 7 are in something a + // reader can picture. Taken from the page being drawn rather than written down, + // because the page being drawn is the one the reader is looking at. + const medOf = (xs) => [...xs].sort((a, b) => a - b)[Math.floor(xs.length / 2)] ?? 0; + const MARK_W = medOf(d.marks.map((m) => m.b[2])).toFixed(1); + const MARK_H = medOf(d.marks.map((m) => m.b[3])).toFixed(1); + + const optionCard = (opt) => { + const h = ho(opt.grain); + const perMark = opt.grain === "mark"; + const num = h + ? `${h.far.toFixed(2)}%` + : `${(c.pooled[opt.grain].far * 100).toFixed(1)}%`; + const ci = !perMark && h?.ci?.length === 2 ? ` [${h.ci[0].toFixed(2)}, ${h.ci[1].toFixed(2)}]` : ""; + // H's two numbers do not mean what the other four options' numbers mean, and + // the caption is where that is said rather than in a footnote nobody reaches. + // Its "badly out" counts only the marks it declined to place from ink; the + // ones it did place sit on the measurement itself and cannot be scored by it. + const note = perMark + ? `counting only the ${(100 - h.share).toFixed(1)}% it would not place from ink — the rest sit on the measurement and cannot be graded by it` + : "measured on marks the correction was not worked out from"; + const near = perMark + ? `

    Placed from the mark's own ink

    ${h.share.toFixed(1)}%
    ${h.fellBack.toLocaleString("en")} marks fall back to the printed line: ${h.clamped.toLocaleString("en")} where the search ran out of room and ${h.weak.toLocaleString("en")} where it found nothing convincing

    ` + : `

    On this page

    ${pctS(share(opt.grain))}
    of ${d.marks.length} marks further than ${d.far} of a unit from their own ink

    `; + return `
    +
    +

    ${opt.id} — ${esc(opt.title)}

    +

    ${esc(opt.tag)}

    +
    + ${specimen(layer(opt, BAND), { crop: BAND })} + ${specimen(layer(opt, CLOSE) + layerInk(CLOSE), { crop: CLOSE, label: `The same option, close up — about ${closeMag} times the size above, on the mark this correction moves furthest. The dashed outlines are where the marks really are, so the gap you can see is the error this option leaves behind.` })} +
    +

    What it does

    ${esc(opt.what)}

    +

    Rectangles badly out

    ${num}${ci}
    ${note}

    + ${near} +
    +
    `; + }; + + // The board. One column an option, one row a question a decider actually has. + // The only measured thing on it is the bar, and it is drawn to the same scale + // in every column so the collapse from A to F is a length rather than a digit. + // Laid out row-major in one grid rather than as four independent columns: the + // whole value of a board is that "what it disturbs" sits on one line across all + // four, and a per-column flex stack only lines up by luck. The label column is + // sticky because the board is wider than a phone and scrolls; a row whose label + // has scrolled off is a row of adjectives attached to nothing. + const fill = (s) => + s + .replace("PAGES2", (c.pages * 2).toLocaleString("en")) + .replace("LINES4", (c.lines * 4).toLocaleString("en")) + .replace("MARKS2", (c.marks * 2).toLocaleString("en")); + + const boardCell = (t) => + `
    ${esc(t.head)}${ + t.body ? `${esc(fill(t.body))}` : "" + }
    `; + + const barCell = (opt) => { + const h = ho(opt.grain); + const pct = h ? h.far : c.pooled[opt.grain].far * 100; + // H's bar is not the same measurement as the other four and the caption has + // to say so where the bar is, not in a note underneath it. Its figure counts + // only the marks it declines to place from ink; the ones it does place are + // sitting on the very thing that would grade them. + const cap = + opt.grain === "mark" + ? "of rectangles land badly out — counting only the ones it will not place from ink" + : "of rectangles land badly out"; + return `
    + ${pct.toFixed(1)}% +
    + ${cap} +
    `; + }; + + const boardRow = (label, pick) => + `
    ${esc(label)}
    ` + + OPTIONS.map((o) => boardCell(TRADEOFFS[o.id][pick])).join(""); + + const board = `
    +
    + ${OPTIONS.map( + (o) => `
    +

    ${o.id}

    +

    ${esc(o.title)}

    +

    ${esc(TRADEOFFS[o.id].verdict)}

    +
    `, + ).join("")} +
    How wrong it leaves things
    + ${OPTIONS.map(barCell).join("")} + ${boardRow("What it costs to work out", "build")} + ${boardRow("What it disturbs", "disturbs")} + ${boardRow("What it risks", "risk")} + ${boardRow("Getting back", "undo")} +
    `; + + const ladderRow = (g, label) => { + const h = ho(g); + return ` + ${esc(label)} + ${h ? `${h.far.toFixed(2)}%` : "—"} + ${h?.ci?.length === 2 ? `${h.ci[0].toFixed(2)} – ${h.ci[1].toFixed(2)}` : "—"} + ${h ? h.p50.toFixed(2) : "—"} + ${h ? h.p95.toFixed(2) : "—"} + ${h ? `${h.trained.toFixed(2)}%` : "—"} +`; + }; + + // H's row in the same table, which is a different row and has to look like one. + // Four of its six columns are honestly empty: splitting a page's marks in half + // grades a correction the other half never saw, and H has no other half — each + // mark carries its own two numbers, so there is nothing held back to grade with. + // What can be stated is the part that *is* a model: the marks it hands to the + // printed line, and how those fare. + const markRow = () => { + const h = ho("mark"); + if (!h) return ""; + return ` + H — each mark, where its own ink is
    on the ${(100 - h.share).toFixed(1)}% it hands back to the printed line     + ${h.far.toFixed(2)}% + — + ${h.fallback.p50.toFixed(2)} + ${h.fallback.p95.toFixed(2)} + — +`; + }; + + const html = `Where the rectangles go + + +${printDefs} +
    +

    A decision, drawn — Hifth

    +

    Should the app line up what it draws one page at a time, or one printed line at a time?

    +

    + Hifth draws a rectangle over every mark and every word in the mus'haf, so it knows what your + finger is touching. Today those rectangles sit about a mark's height away from the marks they + are meant to be on. There are five things we could do about it, and this page draws all five on + the same real page of the mus'haf, at the size you would actually see them. +

    +
    +
    ${(c.pooled.shipped.far * 100).toFixed(0)}%of rectangles are badly out today
    +
    ${ho("tilt") ? `${ho("tilt").far.toFixed(1)}%` : pctS(c.pooled.tilt.far)}after the recommended fix
    +
    ${c.marks.toLocaleString("en")}marks measured, on ${c.pages} pages
    +
    ${c.unmeasuredCount ?? d.unmeasured.length}pages nothing could measure
    +
    +
    + +
    + +
    +

    1A few words, before anything else

    +
    +
    Mark
    +
    One of the small signs printed above or below a letter — the vowel signs, the doubling sign, + the little circle over a silent letter. There are about ${Math.round(c.marks / c.pages / 10) * 10} + of them on a page.
    +
    Printed line
    +
    One line of text as the printer set it. A page of this mus'haf holds fifteen, and the + printer stretches each one on its own so both ends reach the margins.
    +
    Verse-end circle
    +
    The small ornament that closes each verse. There are ${d.pin.markers} of them on the page + drawn here, and they matter more than they look — see section 3.
    +
    Badly out
    +
    A rectangle sitting further than ${d.far} of a unit from the ink it is meant to be on, on a + page ${pw} units across, where a mark is about five units wide. Roughly: far enough that you + would point at it and say it is on the wrong thing.
    +
    +
    + +
    +

    2What does it look like today?

    +

    + This is a real page of the mus'haf with the app's own rectangles drawn on it. The filled boxes + are marks; the outlines are words. +

    + ${specimen(layer(OPTIONS[0], BAND), { crop: BAND, label: `Three printed lines of page ${d.page}, at about the size a phone draws them. Every filled box is where the app currently believes a mark is.` })} + ${specimen(layer(OPTIONS[0], CLOSE) + layerInk(CLOSE), { crop: CLOSE, label: `Close up. The dashed outlines are where each mark actually is; the filled boxes are where the app puts them. That gap is the whole subject of this page.` })} +

    + So why not simply draw the dashed outlines? It is the first thing anybody asks + here, and it is a good enough question that it became option H below. The honest answer has two + halves. The outlines are not a placement, they are the measurement: each one is where a search + around that one mark found the most ink, which makes them the answer sheet every other option is + marked against — ship them and the score is perfect by construction, and a perfect score that + could not have come out any other way tells you nothing. And they are a record rather than a + rule: two numbers written down for each of ${c.marks.toLocaleString("en")} marks, which say what + this printing did and predict nothing whatever about a page nobody has measured. +

    +

    + Neither of those makes it wrong to ship — they make it something a different kind of check has + to vouch for. Option H does ship them, with a guard, and section 12 says what would have to be + true before anybody could trust it. +

    +

    + Across the whole mus'haf, ${(c.pooled.shipped.far * 100).toFixed(0)} rectangles in every hundred + are badly out by the definition above. It is not a handful of bad pages: it is nearly every + rectangle on nearly every page, all sliding the same way. +

    +
    + +
    +

    3Why are they out at all?

    +

    + Because of how the app learned where things are on the page — and the answer is a little + embarrassing, which is exactly why it is worth reading. +

    +

    + Hifth ships pictures of the mus'haf from one source and the positions of the marks from another. + The two draw the same print at different sizes, so something has to line them up. The only + objects both sources name are the verse-end circles, so those are what the lining-up was + measured on — and it works beautifully. On the page drawn here the circles land within + ${d.pin.residual} of a unit of each other, which is a fraction of the width of one of them. +

    + ${specimen(layerOrnaments("shipped"), { crop: BAND, label: `The verse-end circles. The large outlines are where our picture of the page has them; the small dots are where the other source says they are, after the lining-up. They agree.` })} +

    + And then the text is out by a mark's height anyway. The two prints agree about + where their ornaments sit and disagree about where they set their words. So a lining-up measured + on the ornaments grades itself as excellent and is wrong where it is used. +

    +
    + +
    +

    4Why is this being asked now, and what if nobody answers?

    +

    + A by-eye session was opened to judge whether a proposed correction was better than what ships, + and it was stopped after a handful of trials because the person sitting it reported the + rectangles were wrong in several ways at once. That is not a verdict about the correction; it is + a report that both things being compared were frequently wrong. The measurement then confirmed + it. So the question has to be settled before any more of anyone's time is spent looking at + rectangles. +

    +

    + If nobody decides, option A is what happens, and it is not neutral: colouring a mark, tapping a + mark, or telling a reader which mark a rule lands on all rest on the rectangle being on the + right thing. Those are the next things this app was going to build, and they are all waiting on + this. +

    +
    + +
    +

    5Does anyone outside this project have this problem?

    +

    + Yes, and one half of it is a published result with a name. The other half we looked for and did + not find. +

    +
    +
    Lining up on one thing and using it for another is a known trap
    +
    + In medical imaging the error where you measured is called fiducial registration error and the + error where you actually care is target registration error, and + Fitzpatrick showed + the two are uncorrelated — how well the lining-up closed tells you nothing about how wrong + it is at the target. Our own numbers reproduce it: the closeness at the circles predicts + almost nothing about the error at the text, and on the component that matters most it is + slightly anti-correlated, which + theory also + predicts. There is even + a teaching tool built to break the + intuition, because practitioners keep falling for it. So did we, for three sittings. +
    +
    The remedy is the one below: hold out something the lining-up never saw
    +
    + SimpleITK's + worked notebook is the standard version. Section 8 is ours, inverted: the circles stop + being what we measure on and become what we check against. +
    +
    Why the printed line, and not something else
    +
    + Two settings of the same text drift apart because their letter widths and line spacing differ, + so the error builds up along a line and resets at the next one. That is documented in + typesetting — + inconsistent + glyph widths, + substituted + metrics moving line breaks, + line metrics, and most + sharply in + work showing letter positions leak redacted words. + This was written down before our own numbers were looked at, and then it held. +
    +
    Global first, then local, is the conventional shape
    +
    + Thin-plate splines + pair a global part with a local warp; + a 2024 document-registration + paper learns a coarse transform then a fine one; and there is + a patent on + line-based registration for moving annotations between images, which is almost exactly + what we are doing. +
    +
    What we looked for and did not find — said plainly
    +
    + Nobody appears to have published on our actual case: lining up two independently typeset + editions of the same text at the level of individual letters, where both sides are + drawn shapes with no photographs and no text recognition involved. The document work we found + is all about photographs of paper, where the distortion is physical. Nor did we find anyone + writing up the inversion in section 9 as a recipe. Two areas were not opened at all: + Arabic-typesetting corpora, and the Qur'anic-computing literature. Either could hold + something. +
    +
    +
    + +
    +

    6The options, drawn

    +

    + All five on the same page, the same crops, the same size. For the first four, the percentages + under each are measured on marks that option's correction was not worked out from, which is the + only way a finer correction cannot flatter itself. The last one, H, cannot be measured that way + at all, and its card says what it reports instead. +

    + ${OPTIONS.map(optionCard).join("\n")} +
    + +
    +

    7The same options, as numbers

    + + + + + + + + ${ladderRow("shipped", "A — as it ships")} + ${ladderRow("page", "B — each page")} + ${ladderRow("line", "E — each printed line, no tilt")} + ${ladderRow("tilt", "F and G — each line, tilted")} + ${markRow()} + +
    + Every mark on all ${c.pages} pages, ${c.marks.toLocaleString("en")} of them, split down the + middle: half of each page's marks used to work the correction out, the other half — which the + correction never saw — used to grade it. The first column is that grade. The last column is the + same correction marked on its own homework, and the gap between the two is what a correction + fitting noise rather than a real difference between the two prints would show up as. The two + middle distances are in the same units as the pictures above, where the typical mark on this + page is ${MARK_W} units across and ${MARK_H} tall — so a miss past ${d.far} of a unit, + which is what “badly out” counts, is a rectangle sitting off its own mark. +
    OptionBadly out95% rangeTypical missWorst 5%Marked on its own homework
    +
    +

    + Why H's row is mostly empty, and why that is the honest way to print it. + Every other row is graded by holding half of each page's marks back and marking the + correction on marks it never saw. H has nothing to hold back: each mark carries its own two + numbers, so there is no second half of anything. What the row does state is the part of H + that is a model — the ${d.heldOut.mark.fellBack.toLocaleString("en")} marks it refuses to + place from ink and hands to the printed line instead, which is + ${(100 - d.heldOut.mark.share).toFixed(1)}% of the mus'haf and where all of its remaining + error lives. The other ${d.heldOut.mark.share.toFixed(1)}% sit exactly on the ink they were + measured against, so their score is zero by construction and means nothing at all. That is + the trade this option asks you to accept: far fewer rectangles out of place, and no way for + this instrument to prove it. Section 12 says what could. +

    +

    + The control that makes this a finding rather than a bigger model. Give every + printed line another line's correction, chosen at random, and grade it on the same + unseen half: ${d.heldOut.shuffle.toFixed(1)}% of rectangles are badly out — worse than + applying no per-line correction at all, and + ${(d.heldOut.shuffle / ho("tilt").far).toFixed(0)} times worse than the real one. So the + per-line figures are not what any correction with that many knobs would have given. The last + column says the same thing from the other side: the finest correction does flatter itself, by + ${(ho("tilt").far - ho("tilt").trained).toFixed(1)} points, and that is the whole of the + flattery — against a fall from ${ho("page").far.toFixed(1)}% to + ${ho("tilt").far.toFixed(1)}% that it cannot account for. +

    +
    +
    + +
    +

    8What does each one cost?

    +

    + Accuracy is one column of this and it is the column where two of the options are the same + number. Everything that separates them is here instead: what has to be worked out, what it + disturbs on the way in, and what has to be kept true forever afterwards. +

    + ${board} +
    +

    + Read F and G side by side, because that is the fork this page cannot settle by + measuring. Their first rows are identical, and not by coincidence — they are the same + correction, so no measurement on this page can tell them apart. Every figure here is taken on + marks, and G is the option that leaves the words alone, so the whole of its cost falls + somewhere nothing above is looking. It buys one rebuild it does not have to do, and pays for it + by keeping two different linings-up true to each other for as long as the app exists. +

    +
    +
    + +
    +

    9What does moving everything do to the verse-end circles?

    +

    + This is the check that costs us something, so it is the one worth trusting. The circles are the + one thing on the page that is not ink under a rectangle — the correction is worked out + without them, so where they end up is a free, independent opinion. +

    +

    + The opinion is unwelcome and it is the same at every option: correcting the text pulls the + circles off themselves, on every page measured. That is not a bug in the correction. It is the + two prints disagreeing about their text and agreeing about their ornaments, stated from the + other side — you cannot have both, and this is the cost of choosing the text. +

    +
    + ${specimen(layerOrnaments("shipped"), { crop: BAND, label: "A — the circles land on themselves." })} + ${specimen(layerOrnaments("tilt"), { crop: BAND, label: "C — the same circles, after the recommended correction. They no longer do." })} +
    +

    + Whether that matters depends entirely on what the circles are used for, and today they are used + for one thing: working out the lining-up in the first place. Nothing the reader touches is + placed by them. So the honest statement is that this option trades an error nobody sees for an + error everybody sees — in the right direction — and that trade should be made knowingly rather + than discovered later. +

    +
    + +
    +

    10What about the pages nothing could measure?

    +

    + ${d.unmeasured.length} pages of the mus'haf — ${d.unmeasured.join(", ")} — carry too little to + measure a correction from. They are the opening and closing pages, which are set differently + from every other page in the book. +

    +

    + This is the whole of the generalisation question, and it is worth being blunt about + it. Every option here is worked out on the very page it is applied to. There is no + version of this that generalises to a page it has not seen, because there is no model of a page + — there is a table. That is not a flaw to be fixed; a table over ${c.pages} pages and + ${c.lines.toLocaleString("en")} printed lines is exactly the right shape when each page really + does differ. But it means those ${d.unmeasured.length} pages get whatever fallback we choose, + and the fallback is the entire answer for them. The proposal is that they keep option A + unchanged, and that the app never claims a precision there it has not earned. +

    +
    + +
    +

    11What else could have been on this page?

    +
    +
    Letting every rectangle find its own nearest ink, with nothing holding it back
    +
    + This page was written twice on this point, and the first version was wrong, so the reasoning + is left here rather than quietly replaced. The objection was that two marks sit about as far + apart as the error is large, so a search for the nearest ink would sometimes find the + neighbour's and centre a rectangle neatly on the wrong mark — which is one of the + complaints that started all of this. It was worth measuring rather than asserting, and it did + not survive. The nearest mark of any kind is a median ${d.neighbour.anyP50} units away and the + nearest one that looks the same is ${d.neighbour.sameP50}, against a search that + reaches ${d.neighbour.radius} and an error near one. Across all + ${c.marks.toLocaleString("en")} marks the search ends up nearer some other mark + ${d.neighbour.jumpAny.toFixed(2)}% of the time and nearer an identical-looking one + ${d.neighbour.jumpSame.toFixed(2)}% — ${d.neighbour.jumpSameN} marks in the whole mus'haf. + What the measuring did refuse is different. Where the ink match is poor the + search is chasing noise rather than finding a mark, and a search that stops at the edge of the + distance it was allowed has not found anything at all — it has run out of room. Option H is + this idea with those two refusals built in, and the marks it refuses inherit the printed line. +
    +
    Drawing the marks but never letting anything point at one
    +
    + Kept in the written record as option D and not drawn here, because there is nothing to draw: + it puts the same rectangles on the page and forbids the feature that would use them. It is + the honest halfway house if none of the four below is good enough, and it is a decision about + the app rather than about placement. +
    +
    Stretching each page rather than sliding it
    +
    + Option C in the written record. Measured and dropped. The stretch is real — it is certain, statistically — and it is worth + about a fifth of the sliding, removes under two per cent of what is left, and makes a third of + pages slightly worse. It is a footnote, not an option. +
    +
    Correcting word by word
    +
    + Not usable. A word carries about four marks, and an average over four marks is mostly noise; + the correction would fit the noise and look excellent while being worse. The printed line is + the finest grain with enough marks in it to mean anything. +
    +
    Fixing the mark names instead
    +
    + Already ruled out by counting. Every one of the ${c.marks.toLocaleString("en")} marks was + checked against the name the printer gave it, and the number of genuinely wrong names is zero. + Every complaint about a rectangle being on the wrong thing is this page's problem, not a + naming problem, and the two are kept separate on purpose. +
    +
    +
    + +
    +

    12What would change the answer?

    +
      +
    • + A person preferring the old rectangle. Everything here is measured against + ink. One forced-choice sitting on pages the correction was never worked out from is the only + witness that is not, and it is owed before anything ships. If a reader cannot see the + difference, option B is the cheaper answer and this page over-argues. +
      • +
    • + Someone finding a use for the verse-end circles. If the circles ever place + something a reader touches, section 9 stops being a free trade and becomes a real cost. +
      • +
    • + The remainder turning out to have a cause of its own. After the best + correction here, the rectangles that are still badly out are more concentrated than the + general spread predicts — a minority are badly out for a reason, rather than everything being + slightly out. Nobody has looked for that reason yet, and finding it could make a sixth option. +
      • +
    • + Somebody checking option H by eye and finding it holds. H cannot be graded + the way the others are, and that is not a reason to refuse it — it is a statement about which + instrument is needed. The check it wants is not the same as the one above: rather than being + shown two rectangles and asked which sits better, a reader is shown a sample of rectangles H + has placed and asked, of each one, whether it is on the mark it is named after. If a sample + drawn at random comes back clean, the procedure that placed all + ${c.marks.toLocaleString("en")} of them has been vouched for, which is the only kind of + guarantee a record rather than a rule can be given. That sitting has not happened. Until it + does, H is the most accurate option on this page and the one with the least behind it. +
      • +
    +
    + +
    +

    13What is this not settling?

    +

    + Not whether the app should colour individual marks at all — that is + its own question, and it is open. Not which colours anything + is drawn in. Not the mark names, which were counted and are right. And not the four pages in + section 10, which need their own answer whichever option wins. +

    +
    + +
    + +
    +

    + Where the pictures come from. Nothing here is a mock-up. The page is the real + mus'haf image the app ships. Every rectangle is computed by the same code that would build the + shipped geometry, from the printer's own measured ink, and the colours are read out of the app's + stylesheet when this page is built. All the counts are recomputed from the measurements each + time. +

    +

    + Built by scripts/build-placement-options.mjs — edit that, not this. The technical + write-up behind it, including everything that is still unresolved, is + docs/design/mark-registration.md. Page ${d.page} was chosen because its numbers sit + at the middle of all ${c.pages} measured pages on every option, not at either end. The print is + assets/pages/hafs-kfqc/${d.page}.svg: outlined shapes only, no text. Measured + ${d.ran}. +

    +
    +`; + + writeFileSync(out, html); + const kb = (Buffer.byteLength(html) / 1024).toFixed(1); + console.log(`${basename(out)} — ${kb} KB, page ${d.page}, ${d.marks.length} marks, ${d.words.length} words`); + return () => { + for (const g of GRAINS) { + const h = ho(g); + console.log(` ${g.padEnd(8)} this page ${pctS(share(g)).padStart(6)} corpus ${pctS(c.pooled[g].far).padStart(6)}${h ? ` held out ${h.far.toFixed(2)}%` : ""}`); + } + }; +} + +const STYLE = ` +:root{ + --paper:#f4efe6; --ink:#26201a; --accent:#1f6f66; + --tint:#ede6da; --rule:#ddd2c2; --dim:#6f6559; --card:#fbf8f2; + --shadow:0 1px 0 rgba(38,32,26,.05); +} +@media (prefers-color-scheme: dark){ + :root:not([data-theme="light"]){ --paper:#191612; --ink:#ece4d8; --accent:#5fbfae; --tint:#221d18; + --rule:#3a322a; --dim:#9a8d7d; --card:#201b16; --shadow:none; } +} +:root[data-theme="dark"]{ --paper:#191612; --ink:#ece4d8; --accent:#5fbfae; --tint:#221d18; + --rule:#3a322a; --dim:#9a8d7d; --card:#201b16; --shadow:none; } +:root[data-theme="light"]{ --paper:#f4efe6; --ink:#26201a; --accent:#1f6f66; --tint:#ede6da; + --rule:#ddd2c2; --dim:#6f6559; --card:#fbf8f2; --shadow:0 1px 0 rgba(38,32,26,.05); } + +body{ background:var(--paper); color:var(--ink); margin:0; + font:16px/1.65 "Iowan Old Style","Charter","Palatino Linotype",Palatino,Georgia,serif; + -webkit-text-size-adjust:100%; } +main,.head,footer{ max-width:60rem; margin:0 auto; padding:0 1.5rem; } +h1,h2,h3,h4,.eyebrow,.tag,.n,code,table,.stats b,figcaption,.big{ + font-family:ui-sans-serif,"Helvetica Neue",Arial,system-ui,sans-serif; } +h1{ font-size:clamp(1.9rem,4.6vw,3rem); line-height:1.1; letter-spacing:-.02em; + margin:.2em 0 .5rem; text-wrap:balance; font-weight:650; } +h2{ font-size:1.35rem; letter-spacing:-.01em; margin:0 0 .9rem; display:flex; gap:.7rem; + align-items:baseline; font-weight:650; text-wrap:balance; } +h3{ font-size:1.08rem; margin:0; font-weight:650; } +h4{ font-size:.72rem; margin:0 0 .35rem; text-transform:uppercase; letter-spacing:.09em; + color:var(--dim); font-weight:650; } +p{ margin:0 0 1rem; max-width:64ch; } +code{ font-family:ui-monospace,SFMono-Regular,Menlo,monospace; font-size:.86em; + background:var(--tint); padding:.1em .35em; border-radius:3px; } +strong{ font-weight:650; } +.dim{ color:var(--dim); } .small{ font-size:.85rem; } +a{ color:var(--accent); text-underline-offset:2px; } +a:focus-visible{ outline:2px solid var(--accent); outline-offset:2px; border-radius:2px; } + +.head{ padding-top:3.5rem; padding-bottom:1rem; } +.eyebrow{ font-size:.72rem; letter-spacing:.16em; text-transform:uppercase; color:var(--accent); + margin:0; font-weight:650; } +.standfirst{ font-size:1.1rem; max-width:58ch; } +.stats{ display:flex; flex-wrap:wrap; gap:1px; background:var(--rule); border:1px solid var(--rule); + border-radius:6px; overflow:hidden; margin:1.75rem 0 0; } +.stats div{ flex:1 1 11rem; background:var(--card); padding:.9rem 1rem; } +.stats b{ display:block; font-size:1.7rem; font-variant-numeric:tabular-nums; letter-spacing:-.02em; + line-height:1.1; } +.stats span{ display:block; font-size:.8rem; color:var(--dim); margin-top:.15rem; } + +section{ padding:3rem 0 0; border-top:1px solid var(--rule); margin-top:3rem; } +section:first-child{ border-top:0; } +.num{ font-size:.72rem; color:var(--paper); background:var(--accent); width:1.5rem; height:1.5rem; + border-radius:50%; display:inline-grid; place-items:center; flex:0 0 auto; font-weight:650; + font-variant-numeric:tabular-nums; } +.lede{ font-size:1.06rem; } + +.spec{ margin:1.5rem 0; } +.win{ position:relative; width:100%; padding-top:var(--ar); overflow:hidden; background:#fdfaf4; + border:1px solid var(--rule); border-radius:5px; } +@media (prefers-color-scheme: dark){ .win{ background:#f4efe6; } } +.print{ position:absolute; display:block; max-width:none; } +.ov{ position:absolute; inset:0; width:100%; height:100%; } +figcaption{ font-size:.8rem; color:var(--dim); margin-top:.5rem; max-width:64ch; } +.two{ display:grid; grid-template-columns:repeat(auto-fit,minmax(15rem,1fr)); gap:1.25rem; } +.two .spec{ margin:0; } + +.opt{ border:1px solid var(--rule); border-radius:7px; overflow:hidden; margin:1.5rem 0 0; + background:var(--card); box-shadow:var(--shadow); } +.opt-h{ padding:1rem 1.15rem; display:flex; flex-wrap:wrap; gap:.2rem 1rem; align-items:baseline; + border-bottom:1px solid var(--rule); } +.tag{ margin:0; font-size:.78rem; color:var(--dim); letter-spacing:.02em; } +.opt .spec{ margin:0; } +.opt .win{ border:0; border-radius:0; border-bottom:1px solid var(--rule); } +.opt figcaption{ padding:.5rem 1.15rem 0; margin:0; } +.cost{ display:grid; grid-template-columns:repeat(auto-fit,minmax(13rem,1fr)); border-top:1px solid var(--rule); } +.cost>div{ padding:1rem 1.15rem; border-right:1px solid var(--rule); } +.cost>div:last-child{ border-right:0; } +.cost p{ margin:0; font-size:.9rem; } +.big{ font-size:1.5rem; font-variant-numeric:tabular-nums; letter-spacing:-.02em; font-weight:650; } + +table.grid{ width:100%; border-collapse:collapse; margin:1.25rem 0 1.5rem; font-size:.88rem; + display:block; overflow-x:auto; } +table.grid caption{ text-align:left; font-size:.82rem; color:var(--dim); padding-bottom:.55rem; + font-family:ui-sans-serif,system-ui,sans-serif; max-width:64ch; } +.grid th{ text-align:left; font-size:.7rem; text-transform:uppercase; letter-spacing:.07em; + color:var(--dim); font-weight:650; padding:0 .7rem .45rem 0; border-bottom:1px solid var(--rule); + white-space:nowrap; } +.grid td{ padding:.45rem .7rem .45rem 0; border-bottom:1px solid var(--rule); vertical-align:middle; } +.grid .n{ text-align:right; font-variant-numeric:tabular-nums; white-space:nowrap; } +/* H's row is a different kind of statement from the four above it, and four of + its six cells are honestly empty. Setting it off keeps a reader from scanning + down the column and reading those dashes as a worse score. */ +.grid tr.aside td{ border-top:2px solid var(--rule); background:var(--tint); } +.grid tr.aside td:first-child{ padding-left:.7rem; } + +/* The tradeoff board. Severity is its own hue set, kept away from --accent so a + column reads as good-or-costly before any of its words are read. */ +:root{ --t-good:#2f6b45; --t-warn:#8a5a12; --t-bad:#a03c28; } +@media (prefers-color-scheme: dark){ + :root:not([data-theme="light"]){ --t-good:#6fbf8e; --t-warn:#d9a441; --t-bad:#e08a72; } +} +:root[data-theme="dark"]{ --t-good:#6fbf8e; --t-warn:#d9a441; --t-bad:#e08a72; } +:root[data-theme="light"]{ --t-good:#2f6b45; --t-warn:#8a5a12; --t-bad:#a03c28; } + +/* Sized so all five columns fit inside the reading column on a desktop and the + board only scrolls where it must. It is the one element on the page where a + reader compares across, and a board that scrolls when it did not need to hides + a column from somebody who never thought to drag it. */ +.board{ display:grid; grid-template-columns:7rem repeat(5,minmax(10.4rem,1fr)); + gap:1px; background:var(--rule); border:1px solid var(--rule); border-radius:7px; + overflow-x:auto; margin:1.5rem 0 0; box-shadow:var(--shadow); + align-items:stretch; } +.bh{ background:var(--tint); padding:.85rem 1rem; } +.lbl{ position:sticky; left:0; z-index:1; background:var(--tint); font-size:.7rem; + text-transform:uppercase; letter-spacing:.07em; color:var(--dim); font-weight:650; + font-family:ui-sans-serif,"Helvetica Neue",Arial,system-ui,sans-serif; + border-left:0; box-shadow:1px 0 0 var(--rule); justify-content:center; } +.bh h3{ font-size:1.45rem; letter-spacing:-.02em; line-height:1; color:var(--accent); } +.bt{ margin:.3rem 0 0; font-size:.86rem; line-height:1.35; } +.bv{ margin:.35rem 0 0; font-size:.72rem; text-transform:uppercase; letter-spacing:.08em; + color:var(--dim); font-family:ui-sans-serif,system-ui,sans-serif; font-weight:650; } +.tc{ background:var(--card); padding:.8rem 1rem; display:flex; flex-direction:column; gap:.25rem; + border-left:3px solid transparent; } +.tc b{ font-family:ui-sans-serif,"Helvetica Neue",Arial,system-ui,sans-serif; font-size:.84rem; + font-weight:650; line-height:1.3; } +.tc span{ font-size:.83rem; line-height:1.45; color:var(--dim); } +.tone-good{ border-left-color:var(--t-good); } .tone-good b{ color:var(--t-good); } +.tone-warn{ border-left-color:var(--t-warn); } .tone-warn b{ color:var(--t-warn); } +.tone-bad{ border-left-color:var(--t-bad); } .tone-bad b{ color:var(--t-bad); } +.bar-cell b{ font-size:1.8rem; font-variant-numeric:tabular-nums; letter-spacing:-.03em; + line-height:1; color:var(--ink); } +.bar-cell .pc{ font-size:.9rem; color:var(--dim); margin-left:.1em; letter-spacing:0; } +.bar{ height:7px; border-radius:4px; background:var(--tint); overflow:hidden; + box-shadow:inset 0 0 0 1px var(--rule); } +.bar i{ display:block; height:100%; background:var(--t-bad); border-radius:4px; } + +.note{ border-left:3px solid var(--accent); background:var(--tint); padding:1rem 1.15rem; + border-radius:0 5px 5px 0; margin:1.5rem 0 0; } +.note p:last-child{ margin:0; } + +.prior{ margin:1.25rem 0 0; max-width:64ch; } +.prior dt{ font-family:ui-sans-serif,"Helvetica Neue",Arial,system-ui,sans-serif; font-size:.92rem; + font-weight:650; margin-top:1.1rem; } +.prior dt:first-child{ margin-top:0; } +.prior dd{ margin:.35rem 0 0; padding-left:1rem; border-left:2px solid var(--rule); + font-size:.94rem; color:var(--ink); } + +.wch{ margin:1.25rem 0 0; padding-left:1.15rem; max-width:64ch; } +.wch li{ margin-bottom:.7rem; } + +footer{ margin-top:3rem; padding-top:1.5rem; padding-bottom:4rem; border-top:1px solid var(--rule); } +footer p{ font-size:.85rem; color:var(--dim); } +footer strong{ color:var(--ink); } +@media (max-width:34rem){ .cost>div{ border-right:0; border-bottom:1px solid var(--rule); } } +`; + +// ──────────────────────────────────────────────────────────────── main ────── + +if (process.argv.includes("--extract")) await extract(); + +/** + * Both copies, always, from one read of the measurements — and the figures + * printed once at the end rather than once per copy, because the same numbers + * twice reads like two runs disagreeing about nothing. + */ +let figures; +for (const copy of COPIES) figures = render(copy); +figures(); diff --git a/scripts/lib/session-log.mjs b/scripts/lib/session-log.mjs index ac0f4b0..4ad37ab 100644 --- a/scripts/lib/session-log.mjs +++ b/scripts/lib/session-log.mjs @@ -143,7 +143,15 @@ export function summarise(events) { if (at >= 0) notes[at] = e; else notes.push(e); } else if (e.kind === "observation") observations.push(e); - else if (e.kind === "artifact") artifacts.push(e); + else if (e.kind === "report") { + // A sitting that names what is wrong posts `report` where the forced-choice + // tools post `observation`, and it can take one back. Both are answers, and + // counting only the older word left the page reading "0 answers" while its + // transcript filled up — which looks exactly like a sink that is not + // working, and the sink is the whole reason to open a tool from here. + if (e.payload?.kind === "retracted") observations.pop(); + else observations.push(e); + } else if (e.kind === "artifact") artifacts.push(e); else if (e.kind === "verdict") verdict = e; } diff --git a/scripts/session.mjs b/scripts/session.mjs index 2ef1a0d..7f34ad7 100644 --- a/scripts/session.mjs +++ b/scripts/session.mjs @@ -53,12 +53,26 @@ * node scripts/session.mjs --check # resume an unbanked session, or start one * node scripts/session.mjs --check --new # start a fresh transcript regardless * node scripts/session.mjs --check --port 4180 + * node scripts/session.mjs --check --tool + * + * ── Why --tool ─────────────────────────────────────────────────────────── + * + * The ledger pins one tool page per check, which is right for a check whose + * tool is one page. It is wrong for a check whose population was cut into + * parts so a person could finish it: sixteen files, one check, and the pinned + * path can only name one of them. Without an override the other fifteen get + * opened as plain files off a static server, which injects no sink — so every + * answer lives in that phone's browser store and nowhere else until somebody + * remembers to press save. The flag is what makes the phone just a screen. + * + * It overrides the path only. The check is still the check: same transcript, + * same ledger entry, same verdict at the end. */ import { createServer } from "node:http"; import { existsSync, readFileSync, writeFileSync } from "node:fs"; import { randomBytes } from "node:crypto"; import { spawnSync, spawn } from "node:child_process"; -import { extname, join, basename } from "node:path"; +import { extname, join, basename, resolve, sep } from "node:path"; import { readLedger, ROOT, SHOTS_DIR } from "./validation-ledger.mjs"; import { card, CSS, rich, attr } from "./lib/validation-render.mjs"; import { @@ -102,6 +116,40 @@ if (!steps.length) { process.exit(2); } +/* ── which page this sitting opens ─────────────────────────────────────── */ + +/** + * The one the ledger pinned, or the one named on the command line. + * + * An override keeps the pinned label and note where there are any, because the + * check has not changed: the same reasons still apply and they are usually the + * only prose saying what the page is for. Only the path moves, and both the + * card and the transcript say which file it moved to — otherwise sixteen + * sittings of one check produce sixteen identical-looking pages and sixteen + * transcripts that cannot say which was which, and that is not a thing anybody + * discovers until they are reading them months later. + */ +function chosenTool() { + const pinned = check.runbook?.tool ?? null; + const named = arg("tool"); + if (!named) return pinned; + + // Repo-relative and inside the repo. Not a security boundary — whoever runs + // this already has the shell — but the page it names is served through a + // route the token guards, and the guard is only worth having while this + // server's reach is as small as it claims. Both sides go through `resolve` + // because ROOT carries a trailing slash and a string comparison against it + // silently rejects every real path. + if (!resolve(ROOT, named).startsWith(resolve(ROOT) + sep) || extname(named) !== ".html") { + console.error(`\n session — --tool wants a .html path inside the repo, and got "${named}".\n`); + process.exit(2); + } + return { ...(pinned ?? {}), path: named, from: "command line" }; +} + +const tool = chosenTool(); +const toolReady = tool ? existsSync(join(ROOT, tool.path)) : false; + /* ── the transcript ────────────────────────────────────────────────────── */ const commit = @@ -125,15 +173,23 @@ if (resumed) { commit, on: `${process.platform} node ${process.versions.node}`, stepsTotal: steps.length, + tool: tool?.path ?? null, }); console.log(`\n new transcript → docs/validation/sessions/${basename(logPath)}`); } +// A resumed transcript already said which page it was opened against, and a +// resume that opens a different one has to say so or the record is wrong about +// itself. Nothing reads this yet; it is here because the alternative is a file +// that quietly stops describing the sitting it holds. +if (resumed && tool?.path && resumed.events.find((e) => e.kind === "session")?.tool !== tool.path) { + append(logPath, { kind: "tool", path: tool.path }); + console.log(` now against ${tool.path}`); +} + /* ── the page ──────────────────────────────────────────────────────────── */ const TOKEN = randomBytes(9).toString("base64url"); -const tool = check.runbook?.tool ?? null; -const toolReady = tool ? existsSync(join(ROOT, tool.path)) : false; function page() { return ` @@ -162,6 +218,7 @@ ${ ? `
    tool

    ${rich(tool.label ?? "This check has its own tool")}

    ${tool.note ? `

    ${rich(tool.note)}

    ` : ""} + ${tool.from ? `

    Opening ${attr(tool.path)} — asked for by name, not the page this check pins.

    ` : ""} ${ toolReady ? `

    Open it →

    @@ -578,6 +635,7 @@ server.listen(port, "0.0.0.0", () => { if (lan) console.log(` Same Wi-Fi: http://${lan}:${port}/`); console.log(`\n Ticks, notes and answers land in docs/validation/sessions/${basename(logPath)}`); console.log(` as you make them. Ctrl-C when you are done — nothing is lost by stopping.\n`); + if (tool?.from) console.log(` Tool: ${tool.path} (asked for by name)`); if (tool && !toolReady) { console.log(` ! ${tool.path} is not built yet — run the setup commands above first.\n`); } From edf9ce0f338071a94d331d17df8d377c5e00d46f Mon Sep 17 00:00:00 2001 From: Omar Eid Date: Fri, 14 Aug 2026 11:35:33 -0500 Subject: [PATCH 14/95] The registers learn where the marks go, and the question about it gets a page anyone can open MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The placement question is open and now has somewhere to send people: a page that draws each option on real pages of the print at the size it would actually be used, with the measurements beside it rather than only the argument. A wash you cannot see at that size is an answer, and it is one no paragraph would have given. The page names the script that rebuilds it, the record links both, and the row in the register carries the link and the copy because a link with no copy dies the day the host does. The design notes carry what the measurement actually found: which marks are placed from their own ink and which inherit the line's tilt, what the match threshold and the window-edge rule are for, and why the fallback population is defined the way it is — the placed set could not contain a gross error by construction, so a clean result there bounds visible error at about five per cent and not at zero. That caveat has to travel with the number or the number will be read as saying more than it can. The map gains the sitting page, the server that keeps its answers, and the scorer, and the sitting page's note is where the interaction lessons live: why the destructive controls are not in the thumb corner, why the affirming answer costs the same as a fault, why nothing drawn on the paper is themed, and why the deal and what is left of it are two lists. That last one has three load-bearing parts — the arithmetic has to match the builder's exactly or the two counts go on disagreeing, the transcript is not retired with the deck because it is written under one name, and the reader's place moves with the deck rather than resetting. The ledger's runbook is the reader's on-screen instructions, so it moves with what the page now shows, and two sittings already sat are banked beside it. The issue rows are only for the findings that distorted a measurement — the invisible rectangles and the scorer that printed zero — which is this repo's line for a review tool. The ergonomics are real and are not issues. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt --- .claude/plans/shimmering-wandering-lobster.md | 495 +- docs/decisions.json | 44 +- docs/decisions/README.md | 10 +- docs/design/mark-labels.md | 5 +- docs/design/mark-placement.data.json | 22300 ++++++++++++++++ docs/design/mark-placement.html | 823 + docs/design/mark-registration.md | 752 +- docs/issues.json | 45 +- docs/issues.md | 11 +- docs/map.json | 48 +- docs/validation/guide.html | 159 +- docs/validation/ledger.json | 159 +- ...-12T2228-placement-correction-by-eye.jsonl | 10 + ...13T2106-placement-what-kind-of-wrong.jsonl | 62 + 14 files changed, 24636 insertions(+), 287 deletions(-) create mode 100644 docs/design/mark-placement.data.json create mode 100644 docs/design/mark-placement.html create mode 100644 docs/validation/sessions/2026-08-12T2228-placement-correction-by-eye.jsonl create mode 100644 docs/validation/sessions/2026-08-13T2106-placement-what-kind-of-wrong.jsonl diff --git a/.claude/plans/shimmering-wandering-lobster.md b/.claude/plans/shimmering-wandering-lobster.md index 7bb8f5b..af16df3 100644 --- a/.claude/plans/shimmering-wandering-lobster.md +++ b/.claude/plans/shimmering-wandering-lobster.md @@ -1,267 +1,286 @@ -# The correction is confirmed in direction and unresolved in size +# The instrument that asks the question has to be trustworthy first ## Context -Sixty marks were placed by hand on 2026-08-12 to answer §⑦ of `docs/design/mark-registration.md` -— *how far is our correction still out*. The result was banked as: +Sixteen sittings are about to be sat. Between them they cover all **1,851** marks the machine +could not place from ink — every one seen, not sampled — at roughly an hour each. That is +sixteen hours of the scarcest thing this project has. + +Before spending it, the page that does the asking was audited against the frontend-design +brief, and three of its findings do not make the sitting *slower*. They make its answers +**mean something other than what they say**: + +1. **In dark mode the rectangles are drawn in near-invisible colours.** The crop's paper is + deliberately never re-themed — a mus'haf page stays on white — but the two rectangle stroke + colours *are* re-themed, and then drawn on that white. Measured: our box 5.05:1 → **2.49:1**, + the reader's box 4.89:1 → **1.70:1**, at a constant 1.64px stroke. A reader who cannot see the + box affirms it. This finding runs in exactly the direction that looks like success. +2. **A destructive control lands where the reader's thumb already is.** The answer list sits + directly above Back/Next, each answer adds a ~28px row, and the "take it back" button in that + row is right-aligned — into the corner Next occupied a moment earlier. Two taps of Next can + retract the answer just given. +3. **Every pointer frame re-parses the page.** The drag handler rebuilds the stage from + `c.svg` — 2.0 / 9.2 / **23.2 KB** of path data (min/median/max) — on every `pointermove`. A + correction that stutters is a correction the reader gives up on and affirms instead. + +Below those, four findings cost time and accuracy rather than truth: the affirm button looks +the same pressed as unpressed; it sits at y≈842 on a 393px phone, below the fold, behind a +~275px lede that never goes away; 91 of 115 cards carry the mark's containing word and throw it +away, while the sentence naming the mark omits the mark's own name and is the dimmest text on +the card; and four **destructive or load-bearing** controls are under 44px. + +**Decided with the reader, 2026-08-14:** the buttons get pinned to the bottom of the screen and +the picture keeps its own shape (nothing ever shrinks); the fixes land **before** part 1 is sat, +which costs nothing because the sixteen parts were re-dealt and **0 of 1,851** have been +answered on the new deal; scope is the three data-corrupting findings plus the four +time-and-accuracy ones. + +**Outcome:** all 1,851 marks are answered on one instrument, and the transcript means what it +says. + +### What the audit explicitly warns against, and why + +- **Do not put the fault buttons behind a disclosure.** Adding a tap to reporting a fault while + affirming stays free biases the exact ratio the sitting exists to measure. Change weight, not + cost. +- **Do not highlight the ligature's own ink in the crop.** Where the ink is, is the unknown + being measured. Also ruled out and not to be revisited here: centring the crop on the mark's + own ink; SVG resize handles on the rectangle. + +--- + +## Also in scope: the scorer is reporting the wrong number + +`score-mark-report.mjs` (~377-387) medians the `by` field across every placement **event**. Two +things are wrong with that, and they were verified against the banked transcript, not argued: + +**It medians increments, and increments cancel.** `by` is always an increment — both the nudge +pad (`flush`) and the drag (`pointerup`) send the burst, and `to` is the running total. Of 179 +consecutive pairs on the same mark, **133** chain exactly; the other 46 are marks where the +reader pressed *put it back where it was*, and a banked transcript has retractions already +applied, so the superseded placements are gone and the chain legitimately restarts. Nothing is +wrong with the field. What is wrong is medianing it: opposite-signed ±0.1 increments cancel, and +one mark with 44 events outvotes 25 marks with one each. The scorer prints **0.000 / 0.000**. + +**And `to` is not the reader's hand.** `to = at + total − box`, so it is measured from the +**shipped** box and already contains the correction the pipeline applied. Three different +numbers, all real, currently collapsed into one wrong one: + +| question | across | down | +|---|---|---| +| what the scorer prints today | 0.000 | 0.000 | +| where the reader put it, against what ships | −3.569 | −3.134 | +| **how far the reader's own hand moved it** | **−2.468** | **−2.010** | + +The middle row is what corroborates the ink measurement: the corpus sets its text lower and +further across than the ornament fit predicts, `dty` negative on **599 of 600** pages. Two +instruments, same direction, same rough size. That is worth printing correctly. + +--- + +## The work + +Fixes are ordered so each lands in one place rather than in a string another fix is about to +rewrite. **Do them in this order.** + +### ① Parse the paths once per card — `build-mark-report.mjs`, `paint()` ~1084-1122 + +Split `paint()` into `mount(c)` and `paint()`. + +- Module-level `let drawnFor = null, hitEl = null, boxEl = null, mineEl = null;` +- `mount(c)` writes `stage.innerHTML` **once per card**: `c.svg` plus the `.hit`, `.grab` and a + hidden reader's rect; caches the three nodes; sets `drawnFor = c.id`. +- `paint()` calls `mount(c)` only when `drawnFor !== c.id`, then `setAttribute`s x/y/width/height + on the cached nodes and toggles the reader's rect's `visibility`. The `viewBox` write stays. +- The `.hit` slack `t = vb[2]/30` stays per-frame — it is unit-based and framing-dependent. +- **Drop `sw = vb[2] / 220`** and use `vector-effect="non-scaling-stroke"` with `stroke-width` in + CSS pixels. That constant is 1.64px *only because* the stage is `width: 100%`; leaving it in + makes stroke width a function of each card's aspect the moment anything sizes the stage + differently. One fewer per-frame attribute write, too. + +This is what `crop()`'s two-viewBox design was always for — the framing toggle becomes one +attribute write instead of a 23 KB reparse — and it has never actually delivered it. + +**Correction to the audit:** pointer capture is on the **stage** (`stage.setPointerCapture`, +~1158), gated by `onRect(p)` (~1141) — not on a rect node. `docs/map.json:752` describes the +gate. So rebuilding mid-gesture never risked capture; do this fix for the parse cost alone. + +### ② Keep the rectangles legible on white paper, in both themes — CSS ~514-525 + +The rule to establish and then never break: **nothing drawn on the paper is themed.** + +- Add to bare `:root` **only**: `--ours-line`, `--ours-wash`, `--yours-line`, `--yours-wash` + (suggested `#c2410c` / `#c2410c1f` and `#15803d` / `#15803d24`). +- Leave `--ours` / `--yours` / `--ours-fill` / `--yours-fill` exactly as they are and keep them + re-themed — they are **chrome** (affirm button, pressed state, focus ring, the answer list, the + bank panel) and sit on `--field`, so they must stay themed. +- `mount()` draws with the four new tokens. Add one line of comment saying they join `--paper` + and `--ink` in the never-re-themed set, and *why*. +- Carry the distinction independently of hue at the same time: + `stroke-dasharray="0.6 0.4"` on the reader's rect. Two rectangles that differ in dash as well + as colour survive deuteranopia, and survive anyone re-theming the palette later. + +Light theme comes out byte-identical. Dark theme returns to 5.05:1 and 4.89:1. -> residual (-0.073, -0.110) units against a precision of 0.03 → **adopt with the residual -> applied; only the down component is distinguishable from nought** +### ③ Pin the buttons to the bottom — markup ~679-697, CSS + +Two independent halves; **③b must land whichever layout is chosen.** -Re-interrogating the same transcript shows that second clause does not hold, and that the -session had two structural blind spots nothing in its output mentions. None of this changes -the headline — the correction is confirmed, decisively — but it changes what may be done with -it, and the registers currently overstate the case. +**③a — the dock.** Move the affirm row and the Back/Next row into one `
    ` with +`position: sticky; bottom: 0; background: var(--field); border-top: 1px solid var(--edge); +padding: .5rem 0 calc(.5rem + env(safe-area-inset-bottom));` and bump the main column's bottom +padding. **Sticky, not fixed** — it keeps document flow and does not fight the iOS URL bar. +Requires `viewport-fit=cover` on the viewport meta (~511) or `env()` resolves to zero; that is +also the whole of the safe-area finding. -**Outcome:** the record says what the sitting actually established; the scorer stops making the -assumption that produced the overstatement; the answers a person gave come home from a downloads -folder into the repo; and the correction gets measured on all 604 pages instead of 40, which is -the input every remaining question needs. +The picture keeps its natural per-card shape (295–410px judging, 266–529px identifying, measured +across all 1,851 marks). Nothing shrinks, `ptIn()` is untouched, and on the tallest cards the +reader scrolls the picture behind a dock that never moves. **Explicitly not doing:** normalising +the two viewBoxes to a constant aspect inside `crop()` — it either shrinks the mark or makes +every card as tall as the tallest, and the ring clip would have to be re-ordered after the +normalisation or the new margin comes back empty of the ink that belongs in it. -### What the sitting established, corrected +**③b — get the destructive control out of the thumb corner.** In the answer list row, move "take +it back" to the **start** of the row (button, then the kind, then the detail) so nothing tappable +sits bottom-right, and give the list at least 44px of bottom margin. **This must ship before ⑦ +enlarges that button to 44px**, or the mis-tap target doubles. -| | | -|---|---| -| the correction points the right way | **yes, decisively** — 59/60, 98.3% [91.1–99.7] | -| the residual is a real distance | **not established** — see below | -| the correction is the right *size* | **unmeasurable from this session** | -| it generalises to unmeasured pages | **no evidence either way** | +### ④ Make the affirm button look pressed — CSS ~596 -**One.** The scorer treats 60 placements as 60 independent facts. They are 40 pages' worth of -fit, and two marks on one page share that page's error — which the data confirms (page -explains most of the across-axis spread, F ≈ 3.6 on 39,20 df). Clustering by page: +`button.affirm[aria-pressed="true"]` gets a solid fill: background `var(--ours)`, text +`var(--field)`, matching border. Inverts legibly against both themes and reads as *on* without a +second glance. Add a check glyph via `::before` on the pressed state only. **Do not change the +label text** — that would move the fold ⑤ just fixed. -``` -down mean -0.110 · as scored ±0.054 [-0.215, -0.005] - · clustered by page ±0.061 [-0.230, +0.009] crosses zero -``` +*(The audit's specificity claim was off by one rule: `button.affirm` and the generic pressed rule +are both (0,1,1) and settled by source order; the actual out-specifier is +`button.affirm[aria-pressed="true"]` at (0,2,1). Conclusion unchanged.)* -It was marginal to begin with — the upper bound was −0.005 — so this is *not established at -95%*, not *refuted*. Most of the interval still sits below zero. The 59/60 headline is -untouched; nothing that lopsided is reachable by a clustering adjustment. +### ⑤ Get the lede off the fold without touching the fault buttons — markup ~640-646 -The lesson already existed in this repo and did not travel: `probe-mark-ink.mjs:369-383` says -in as many words that two marks on one page are not independent and that a plain interval on -a rate is therefore narrower than the truth. The placing scorer was written without it. +Split the lede into the full text and a one-line brief, with a quiet toggle between them. The +brief shows once the reader has demonstrably read it: `seen > 0`, persisted through a +`keepRead`/`keptRead` pair written **exactly** like the existing `keepSeen`/`keptSeen` +(~733-736) — try/catch on every call, because a `file:` origin throws. -**Two.** `mark-shift.json` covers **40 pages of 604** (`sampled: 4000`, `minMarksPerPage: 20`, -`--pages-n` defaulting to 40 at `probe-mark-ink.mjs:387`). A trial needs a proposed move, so -all 60 placements necessarily came from those same 40 pages. **The session validated the -correction on its own training pages**, and neither the shift file nor the scorer says so. +Roughly 230px comes off the fold. **No answer costs a tap more than it does today**, so the +affirm/fault ratio the sitting measures is untouched. That constraint is the whole design. -**Three.** Across those 40 pages the proposed move barely varies — down spans −1.19…−0.81, -sd 0.085 on a mean of −1.0. With that little leverage the size of the correction is -unmeasurable: the gain came out −0.10 ± 0.68, so "exactly right" and "20% short" are -indistinguishable, and brute force will not fix it (~710 placements even sampling only the -extreme deciles). A ~11% gain error and a −0.11 constant are the *same number* on these pages. -They stop being the same number on a page whose correction is a different size — which is -precisely the 564 pages nobody has measured. +### ⑥ Say which mark, properly — `identify()` ~1240-1264 -The three findings are one finding: **we measured 40 pages and asked those same 40 pages.** +- The card already carries the containing word and throws it away for the 91 cards that have + both it and a single letter. Keep the big letters as they are, and append the word after them + at ~1rem, dim, `lang="ar" dir="rtl"`. +- Put the mark's **name** into all three branches. It reaches only the `of === 1` branch today. +- The sentence naming the mark is the question, not a footnote: `--dim` → `--text`, + .82rem → .9rem. -### What was ruled out, and is worth keeping +### ⑦ 44px on the four — CSS -These are real negative results and they narrow the problem usefully: +All four are omissions from a convention this file already keeps elsewhere (the note pad and the +chips are already 44/48): -- **Not the mark.** Splitting the residual by mark name leaves *more* spread than a single - number (0.425 vs 0.415 down; 0.545 vs 0.526 across). The rectangle is not anchored wrong - inside particular glyphs. An early read that shadda drove the residual did not survive the - model comparison. -- **Not a stretch.** No dependence on where the mark sits on the page (t = −0.5 down), so the - fit is off by a translation, not a scale. -- **Not the starting point** (slope 0.00) and **not fatigue** (slope 0.000 against trial - order). The evenly-spread starts and the interleaved repeats both did their job. +| control | today | why it matters | +|---|---|---| +| take it back | ~24px | destructive — **land ③b first** | +| the two view toggles | ~31px | pressed constantly, both framings | +| hand over what I have said | ~29px | the reader's safety net | +| put it back where it was | 34px | destructive; also give it `margin-left: auto` so it is not shoulder-to-shoulder with "Finer steps" | + +Land **after** ⑤ and ③, since each of those moves the fold. + +### ⑧ The scorer — `score-mark-report.mjs` ~377-387 + +Do 1-4 **before** 5. Steps 1-4 work on the two transcripts already on disk, which cannot be +re-recorded; once the headline reads `to`, the rename stops being load-bearing. -So it is a per-page frame error, which is where §⑦ already believed it was. +1. Collapse to **one row per mark** — walk in order, keep the last placement carrying a `to`. +2. Print the reader's **hand**: `to` minus the displacement already shipped + (`drawnAt(r) − r.box`, both already available). Median/p90/worst of the magnitude, plus signed + per-axis medians. **−2.468 / −2.010** on the banked sitting. +3. Print, under a **separately worded** sentence, where they landed **against what ships** — + median final `to`, **−3.569 / −3.134** — and say in the prose that this one includes the + correction the pipeline already applied, so no reader ever differences the two. +4. Say *n marks*, and say how many events those marks took. 26 marks / 205 events is itself a + finding about the nudge pad. +5. ~~Rename the drag's field.~~ **Dropped.** Both paths genuinely send an increment and `to` is + the running total, so one name is right, and a rename would only make the two transcripts + already on disk unreadable. (The drag path does not need `flush()`'s save-and-restore around + `dropVague` either: it captures the total at ~1190, before, and reassigns at ~1199, after.) -## Code audit +--- + +## The hazard, and the guard for it -| where | what is wrong | -|---|---| -| `score-mark-nudge.mjs:122` `meanCI` | independence across placements; needs a page-clustered standard error. Used at `:168-169` for both reported components. | -| `score-mark-nudge.mjs:161` `wilson(...)` | same assumption on the headline rate. Survives it here, but the output should say the denominator is pages-worth-of-fit, not independent trials. | -| `score-mark-nudge.mjs` (absent) | no estimate of whether the correction is the right *size*, and no statement that the sample had no leverage to find out. A silent blind spot is worse than a wide interval. | -| `score-mark-nudge.mjs` (absent) | never says the placements came from the same pages the correction was measured on. The largest limitation of the whole session appears nowhere in its report. | -| `probe-mark-ink.mjs:1136-1148` shift emit | records `sampled` and `minMarksPerPage` but not page coverage. Nothing downstream can tell the file describes 6.6% of the mushaf. | - -Nothing is wrong with the arithmetic. Every defect is a **claim the output does not qualify**, -which is the failure mode this project's whole validation shape exists to catch. - -## Work - -### A. `score-mark-nudge.mjs` — say what the sample can and cannot support - -Four additions. All reporting; the residual itself does not move. - -1. **Cluster the interval by page.** Keep `meanCI` for the naive number and add the clustered - one beside it, so the difference is visible rather than swapped in silently. The verdict - sentence at the foot reads from the clustered interval. -2. **Report the gain**, by regressing the residual on the proposed move, *with* the spread of - the proposed moves printed next to it — that spread is what says whether the estimate could - ever have meant anything. -3. **Report coverage**: how many distinct pages the placements came from, and how many of them - the shift file covers out of 604. One line. -4. **Print the negative results** — by-name, by-position and by-order — because "we looked and - found nothing" is the part a later reader will otherwise pay to rediscover. This is where - the throwaway analysis behind this plan gets a permanent home. - -Doc comments carry the reasons, in the register the file already uses. Extend -`lib/adjudication.test.mjs` with a case per new statistic against a hand-built fixture — the -clustered interval must be provably wider than the naive one on clustered input. - -### B. Re-score, then correct the record from the scorer's own output - -Not from the throwaway analysis behind this plan. `pnpm nudge:score` prints the corrected -numbers, and the registers quote what it printed. - -- `docs/validation/ledger.json` — the `result` line of `placement-residual-by-hand`. The check - stays **done**: it ran, it produced a result, and its result is the corrected reading. A done - check leaving its question open is a normal outcome and not a contradiction. -- `docs/issues.json` ⑦ `a-preference-does-not-say-how-far` — back to **open**. The question is - literally *how far*, and the magnitude is unresolved; leaving it `answered` is exactly the - quiet overstatement this repo's rules exist to prevent. The note records what the sitting - did establish, so re-opening reads as progress rather than a reversal. -- `docs/issues.json` — **a new row**: the correction covers 40 pages of 604 and has only ever - been checked on those 40. Distinct question, distinct row. -- `docs/design/mark-registration.md` §⑦ — heading back to **open**, with the sitting's numbers, - the clustering correction, the three ruled-out explanations, and an explicit statement of - what the session could not see. §⑩ ① is untouched — the forced choice is a separate - instrument and its row does not move. - -Then `pnpm issues:doc` && `pnpm gate:issues`; `pnpm guide` after the ledger edit. - -### C. Measure all 604 pages - -No code change — `probe-mark-ink.mjs` already takes `--pages-n` and `--shift-out`, and all four -downstream tools already take `--shift`. - -``` -node packages/etl/scripts/probe-mark-ink.mjs --pages-n 604 --sample \ - --shift-out packages/etl/out/mark-shift.604.json -``` - -- **Time a 5-page run first** and multiply. The runtime of a full pass is unknown and opening - a page is the expensive part. -- **`--sample` must hold `minMarksPerPage: 20`** across 604 pages, so ≥ ~15,000 marks, up from - 4,000. Confirm against the printed per-page counts rather than assuming. -- **Write to `mark-shift.604.json`, never over `mark-shift.json`.** The naming follows - `mark-exemplars.${N}.json`. This is not tidiness: a forced-choice session is live at 5 of 100 - answers and pinned to fingerprint `c8528da9`; rewriting the file in place makes - `adjudicate:score` exit 2 and throws those answers away. -- Add page coverage to the emitted file so nothing downstream can read it without knowing. - -**Then stop and read it.** The one question that decides everything after: *do the per-page -corrections vary across the full mushaf, or are they all alike?* If they vary, the size of the -correction becomes measurable and another session is worth someone's time. If they are all -alike, a single global number is the right model, the residual is moot, and mark-C can proceed. - -### D. The results live in the repo, not in `~/Downloads` - -The answers a person gave are the primary evidence for every number in §⑦, and they currently -sit in one person's downloads folder where a browser will eventually clear them. The transcripts -already come home to `docs/validation/sessions/`; the answer files do not, and nothing says -where they should go. - -**New directory: `docs/validation/rulings/`** — the raw answer files both scorers read. Named -for what the scorers themselves call them (`const ruling = ...` in both). Distinct from its two -neighbours on purpose, and the README says which is which: - -| directory | what it holds | +**The whole page — CSS, markup and JS — lives inside a template literal.** Backticks are +forbidden anywhere in the emitted region *including comments*, and `${` must not appear except +as a deliberate interpolation. This has broken the file three times. + +Add two assertions to `build-mark-report.test.mjs` that say so out loud: the emitted HTML +contains no backtick and no `${`. Two lines, and they document the trap. + +## Tests + +The suite today parses the head and the cards out of the emitted HTML and string-matches the +emitted script. Both styles extend cleanly; there is no visual coverage and none is being +invented here. + +| fix | what a test would actually assert | |---|---| -| `sessions/*.jsonl` | the transcript — what a person did, step by step, as it happened | -| `evidence/.json` | a machine run's exit code, written by `make validate-auto` | -| `rulings/` | **new** — the answers themselves, the input a scorer needs to reproduce a verdict | - -Move `~/Downloads/mark-placements-23.json` in as -`docs/validation/rulings/2026-08-12T1650-placement-residual-by-hand.seed23.json`, pairing with -the transcript already committed under the same stem. The seed is in the name because the seed -is load-bearing — it is what rebuilds the answer key. - -Audited before proposing it: 11.9 KB, no Arabic anywhere in the file, no NUL bytes, no field -that could carry anything personal — page numbers, mark indices, offsets and millisecond -timings. It is the same class of artifact as the transcript beside it, which -`sessions/README.md` already answers "yes, committed" for. - -**And the file it must be scored against.** `packages/etl/out/mark-shift.json` is gitignored, -so a committed ruling still cannot be re-scored on a clean checkout — the verdict is auditable -only by someone who can rebuild the shift file first. That is a real gap and it is worth -closing in the same move: the shift file is 40 rows of page number and offset, derived from the -gitignored ligature cache but containing no ink and no scripture. **Proposal: commit -`mark-shift.604.json` from C** into `docs/validation/rulings/` beside the ruling it explains, -so the banked verdict re-derives from committed bytes the way everything else here does. - -This runs against `probe-mark-ink.mjs:1129-1131`, which says the shift file "lands beside the -evidence page, which is not checked in" — but the reason given there is that the evidence page -draws the mus'haf's own artwork, and the shift file draws nothing. Flagged rather than assumed; -strike it and the rulings still come home, just less useful. - -### E. The links - -- `docs/design/mark-registration.md` "Where does this live?" — `score-mark-nudge.mjs` gains the - clustering and coverage statements; the coverage caveat gets named where the commands are. -- `docs/map.json` — hand-edited. The existing rows for `score-mark-nudge.mjs` and - `probe-mark-ink.mjs` gain the two facts a future reader needs before trusting either: - placements cluster by page, and the shift file describes a sample of pages. -- `packages/etl/data/pages/PROVENANCE.md` — the coverage number belongs beside the data it is - about, the way `probe-reference`'s 568/36/0 already does. -- `docs/validation/sessions/README.md` — points at `rulings/` so the pair is findable from - either end, the same reason `docs/decisions.json` insists a relation is stated in both rows. -- `docs/validation/ledger.json` — the `save-the-placements` runbook step of - `placement-residual-by-hand` currently ends at a downloads folder. It should end in the repo, - or the next session's answers go the same way these nearly did. - -## The next session, pre-registered but not built - -Captured here so the design survives; **built only after C is read.** - -- **Block 1, coverage** — ~40 marks, one each from 40 **held-out** pages: pages with a - correction from the 604-pass that were not among the original 40. Answers *does it work where - no eye has been?* -- **Block 2, structure** — ~20 marks over 5 pages, 4 each. Answers *how much of the spread is - the page and how much is the hand*, which is what the clustered interval needs and what 40 - pages × 1.5 marks could never give. -- **A second reader** — ~20 marks drawn from reader A's set, same build, placed independently. - This is the only thing that separates *the print is off by this much* from *this reader - places boxes this way*. Needs a reader field in the session and the scorer. Agreement within - ~0.05 says it is the print; a larger gap says a global residual should not be applied at all. -- Repeats stay as they are — the existing 15% mechanism measured 0.03 and worked. -- **The prediction goes in the ledger before the sitting**, so it can fail: applying - (−0.073, −0.110) should return a residual of 0 ± 0.06 with "nearer ours" ≥ 90%. The same - −0.11 again means it was applied to the wrong thing. +0.11 means it was fitted to one hand. - -## What this means for mark-C - -**Unblocked, with one constraint.** The per-page correction is confirmed in direction at 98.3% -and may be applied as measured. The extra residual is **not** applied — it is unresolved, and -0.11 units spread over 326,515 rectangles is a number that would have to be bought again. -mark-C needs the 604-page file from C first regardless, since it ships shards for every page. +| ② **highest value** | the dark block re-themes **none** of the paper, the ink, or the four new rectangle tokens; and both rectangle line colours clear 3.0:1 against the paper. Twelve lines, no browser, and it is precisely the invariant that broke. | +| ① | the paint function contains no `innerHTML`, and `c.svg` appears exactly once in the emitted script, inside `mount`. Coarse, in the same spirit as the existing replay test, and it catches the regression that matters. | +| ⑥ | data, not layout: a single-letter card always carries a longer containing word — this is what makes the fix possible and what would silently stop being true. Plus that the mark's name reaches all three branches. | +| ⑤ | the brief lede exists and the collapse is keyed off the stored flag; a regression deletes the second copy. | +| ⑧ | in `score-mark-report.test.mjs`, which already builds synthetic transcripts and asserts on stdout: one mark nudged +0.5, −0.4, +0.4 (final `to` 0.5) and a second moved once by 0.5 — assert the printed median is **0.5, not 0.4**, and that the line says **2 marks**, not 4 events. That fixture would have caught this. | +| ③④⑦ | not testable here beyond CSS-text presence. Fold position, per-frame cost and thumb geometry need a browser, and there is no in-repo harness for one. | + +## Registers + +- **`docs/map.json:752`** — the `build-mark-report.mjs` note is the canonical home and already + records *why* each control sits where it does, including two prior interaction regressions. + Fixes ①②③ belong there. Hand-edited. +- **`docs/map.json`** — `serve-sittings.mjs` has **no row anywhere in `docs/`**. It needs one; + it is the thing that makes an answer survive the browser losing it. +- **`docs/validation/ledger.json` → `placement-what-kind-of-wrong`** — its 13 runbook steps + **are** the reader's on-screen instructions. ⑤ and ⑥ change what the reader sees; the steps + change with them. Then `pnpm guide`. +- **`docs/issues.json`** — only for the findings that **distorted a measurement**, which is this + repo's convention for a review tool (precedent: the "wrong instrument" paragraph inside + `nobody-has-looked-at-the-placement-verdicts`). That is finding ② and the scorer ⑧ — not the + ergonomics. Then `pnpm issues:doc && pnpm gate:issues`. +- **A design doc is not warranted yet.** `docs/design/encoding-inspector.md` is the template if + it ever is. + +### Owed from the sitting already completed, and still unrecorded + +`make record CHECK=placement-what-kind-of-wrong RESULT='…'` — sixty marks from the placed set, +every one explicitly affirmed, no faults. Bounds visible placement error at about **5%, not at +zero**. The caveat that must survive into the wording: the placed population is defined by a +match of 0.55 or better and a displacement under 3 units on a mark 5.6 × 3.6, so **gross errors +were structurally impossible on those cards**. The check also owes its `tunes` step — a manual +result must tighten something automated, and the gate fails if it tunes nothing. ## Verification -1. `pnpm nudge:score docs/validation/rulings/2026-08-12T1650-placement-residual-by-hand.seed23.json` - — the corrected numbers, from the shipped tool, read from the repo rather than from a - downloads folder. Every figure quoted into a register comes from this output, and the run - itself is the proof the moved file is intact. -2. `pnpm --filter @hifth/etl test` (or the repo's vitest path) — the new statistics against the - fixture, including the clustered-vs-naive assertion. -3. `pnpm issues:doc && pnpm gate:issues`, `pnpm gate:validation` after `pnpm guide`. -4. `git add -A && make ci`. -5. `adjudicate:score` on the live forced-choice ruling still exits without a fingerprint - complaint — proof the 604-pass did not disturb it. -6. Commit code and docs separately. +1. Rebuild all sixteen and confirm the deal is unchanged — same slice fingerprint, 1,851 marks + across sixteen parts, 0 answered marks re-asked: + `for n in 1..16; node scripts/build-mark-report.mjs --rows out/mark-rows.line-tilt.json + --set fallback --seed 23 --part $n/16 --answered … --out out/sit.fallback-$n-of-16.html` +2. `pnpm --filter @hifth/etl test` — the new assertions above, including the two hazard guards. +3. Re-run the scorer over the two banked transcripts and confirm it prints the hand and the + against-what-ships figures as **two** numbers under two sentences. +4. Open part 1 on the phone over the tailnet, in **dark mode**, and confirm by eye: both boxes + visible on white paper; the buttons do not move between cards; the drag does not stutter on + the largest card; affirm is above the fold and obviously pressed when pressed. +5. `pnpm issues:doc && pnpm gate:issues`; `pnpm guide` after the ledger edit; + `pnpm gate:validation`. +6. `git add -A && make ci`. Commit code and docs **separately**. ## Operational - `cd /Users/omareid/Workspace/git/hifth && ./scripts/with-lock.sh
    {isOpen && fromKey && (
    - +
    )} diff --git a/package.json b/package.json index 1346fd0..520fb8b 100644 --- a/package.json +++ b/package.json @@ -29,6 +29,7 @@ "gate:verified-edges": "node scripts/gate-verified-edges.mjs", "gate:edges": "node scripts/gate-edges.mjs", "gate:text-sources": "node scripts/gate-text-sources.mjs", + "gate:scripture": "node scripts/gate-scripture.mjs", "gate:ci-artifacts": "node scripts/gate-ci-artifacts.mjs", "gate:golden-env": "node scripts/gate-golden-env.mjs", "gate:golden-size": "node scripts/gate-golden-size.mjs", @@ -47,7 +48,7 @@ "gate:i18n": "node scripts/gate-i18n.mjs", "gate:params": "node scripts/gate-params.mjs", "gate:gates": "node scripts/gate-gates.mjs", - "gates": "pnpm gate:text-sources && pnpm gate:notext && pnpm gate:license && pnpm gate:license-copy && pnpm gate:notices && pnpm gate:validation && pnpm gate:verified-edges && pnpm gate:edges && pnpm gate:gates && pnpm gate:ci-artifacts && pnpm gate:golden-env && pnpm gate:golden-size && pnpm gate:assets && pnpm gate:pages && pnpm gate:words && pnpm gate:align && pnpm gate:map && pnpm gate:use-cases && pnpm gate:issues && pnpm gate:tasks && pnpm gate:decisions && pnpm gate:quran-meta && pnpm gate:tajweed && pnpm gate:revision-privacy && pnpm gate:i18n && pnpm gate:params && pnpm gate:budget", + "gates": "pnpm gate:text-sources && pnpm gate:scripture && pnpm gate:notext && pnpm gate:license && pnpm gate:license-copy && pnpm gate:notices && pnpm gate:validation && pnpm gate:verified-edges && pnpm gate:edges && pnpm gate:gates && pnpm gate:ci-artifacts && pnpm gate:golden-env && pnpm gate:golden-size && pnpm gate:assets && pnpm gate:pages && pnpm gate:words && pnpm gate:align && pnpm gate:map && pnpm gate:use-cases && pnpm gate:issues && pnpm gate:tasks && pnpm gate:decisions && pnpm gate:quran-meta && pnpm gate:tajweed && pnpm gate:revision-privacy && pnpm gate:i18n && pnpm gate:params && pnpm gate:budget", "align": "node packages/etl/scripts/build-alignment.mjs", "probe:ligature-print": "node packages/etl/scripts/probe-ligature-print.mjs", "probe:word-registration": "node packages/etl/scripts/probe-word-registration.mjs", diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 556569b..6278713 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -101,14 +101,7 @@ export { type StageFit, } from "./view.js"; -export { - VERSE_TEXT, - verseTokens, - diffPair, - type DiffClass, - type DiffToken, - type DiffSide, -} from "./verse-text.js"; +export { wordDiff, divergentRuns, type DiffSide, type WordDiff } from "./verse-diff.js"; export { serializeState, diff --git a/packages/core/src/verse-diff.test.ts b/packages/core/src/verse-diff.test.ts new file mode 100644 index 0000000..6bc689b --- /dev/null +++ b/packages/core/src/verse-diff.test.ts @@ -0,0 +1,75 @@ +import { describe, expect, it } from "vitest"; +import type { Edge } from "./adjacency.js"; +import { divergentRuns, wordDiff } from "./verse-diff.js"; + +/** The 2:48 ↔ 2:123 edge as it ships, which is the pair the old table existed for. */ +const EDGE: Edge = { + type: "mutashabih", + to: "quran/hafs-kfqc/2:123", + page: 19, + dir: { dSurah: 0, dPage: 12, sameJuz: true }, + span: { from: [1, 13] }, + toSpan: { from: [1, 13] }, +}; + +describe("wordDiff", () => { + it("reads both sides' shared run off the edge", () => { + const d = wordDiff(EDGE, "quran/hafs-kfqc/2:48"); + expect(d).toEqual({ + from: { key: "2:48", page: 7, shared: [1, 13] }, + to: { key: "2:123", page: 19, shared: [1, 13] }, + }); + }); + + it("derives the source page by undoing the edge's own page delta", () => { + // 19 − 12 = 7, and 2:48 is printed on 7. The edge is self-describing, so + // this cannot drift out of step with the page it names. + expect(wordDiff(EDGE, "2:48")?.from.page).toBe(7); + expect(wordDiff({ ...EDGE, page: 100, dir: { dSurah: 0, dPage: -5 } }, "2:48")?.from.page).toBe( + 105, + ); + }); + + it("strips an edition prefix and a word anchor from the source key", () => { + expect(wordDiff(EDGE, "quran/hafs-kfqc/2:48#w3-7")?.from.key).toBe("2:48"); + }); + + it("declines an edge that names no words on either side", () => { + // The common case, not a failure: 452 of 2,996 look-alike edges match in + // more than one place and so name none of them. The caller falls back. + const { span: _span, ...noSpan } = EDGE; + const { toSpan: _toSpan, ...noToSpan } = EDGE; + expect(wordDiff(noSpan, "2:48")).toBeNull(); + expect(wordDiff(noToSpan, "2:48")).toBeNull(); + }); + + it("declines an inverted range rather than painting it backwards", () => { + expect(wordDiff({ ...EDGE, span: { from: [9, 4] } }, "2:48")).toBeNull(); + }); +}); + +describe("divergentRuns", () => { + it("names the tail when the two ayahs share their opening", () => { + // 2:48 is 23 words on its page and shares its first 13 — so 14..23 differ. + expect(divergentRuns({ from: 1, to: 23 }, [1, 13])).toEqual([[14, 23]]); + expect(divergentRuns({ from: 1, to: 22 }, [1, 13])).toEqual([[14, 22]]); + }); + + it("names both ends when the shared run is in the middle", () => { + expect(divergentRuns({ from: 1, to: 20 }, [5, 15])).toEqual([ + [1, 4], + [16, 20], + ]); + }); + + it("names nothing when the whole of what is here is shared", () => { + expect(divergentRuns({ from: 1, to: 13 }, [1, 13])).toEqual([]); + }); + + it("clamps to what this page holds, for an ayah that runs onto the next", () => { + // The shared run is recorded over the whole ayah; only 1..8 is printed here. + expect(divergentRuns({ from: 1, to: 8 }, [1, 13])).toEqual([]); + // And where the ayah *starts* on this page part-way through its numbering. + expect(divergentRuns({ from: 9, to: 20 }, [1, 13])).toEqual([[14, 20]]); + }); +}); diff --git a/packages/core/src/verse-diff.ts b/packages/core/src/verse-diff.ts new file mode 100644 index 0000000..3e46c75 --- /dev/null +++ b/packages/core/src/verse-diff.ts @@ -0,0 +1,102 @@ +/** + * Which words two look-alike ayahs share, and which ones they do not. + * + * WHAT THIS REPLACED, AND WHY. Until now the "why are these two confusable" + * panel read from a hand-typed table of twelve ayahs' text, carried in this + * package as source. Three things were wrong with that. It was running scripture + * in a code repository, which is the shape takedown notices are routinely scoped + * to. It was typed in a plainer spelling than the mus'haf the reader is looking + * at, so the panel showed different letters from the page underneath it. And it + * covered twelve ayahs out of six thousand. + * + * None of that was necessary, because the corpus already ships the answer. Every + * look-alike edge carries the target's page and the matching word run on *both* + * sides — `span` and `toSpan`, in the print's own word numbering, the same + * numbering the word boxes use. So the shared stretch is a lookup, not a + * judgement, and what diverges is simply what is left over at either end: + * + * 2:48 → 2:123 page 19 span [1,13] toSpan [1,13] + * 2:48 words 1..23 on page 7 → 1–13 shared, 14–23 differ + * 2:123 words 1..22 on page 19 → 1–13 shared, 14–22 differ + * + * which is exactly what the typed table's per-token 0/1/2 classes encoded, for + * 2,544 pairs rather than twelve. The panel draws the page's own ink through + * those boxes, so the reader compares the printing rather than a transcription + * of it. + * + * This module holds no text and reaches no network: it is arithmetic on word + * indices. Fetching the pages and painting the ink is the view's job. + */ +import type { Edge } from "./adjacency.js"; +import type { WordSpanRange } from "./words.js"; + +/** `"quran/hafs-kfqc/2:48#w3-7"` → `"2:48"`. */ +function bareKey(key: string): string { + const tail = key.slice(key.lastIndexOf("/") + 1); + const hash = tail.indexOf("#"); + return hash === -1 ? tail : tail.slice(0, hash); +} + +/** One side of the comparison: an ayah, the page it is printed on, and the run it shares. */ +export interface DiffSide { + /** Bare `"2:48"`, ready for {@link WordIndex}. */ + readonly key: string; + /** The page whose word shard and artwork this side is drawn from. */ + readonly page: number; + /** Inclusive print word indices the two ayahs have in common. */ + readonly shared: readonly [number, number]; +} + +/** The two sides of a look-alike comparison. */ +export interface WordDiff { + readonly from: DiffSide; + readonly to: DiffSide; +} + +/** + * The comparison an edge describes, or `null` when it describes none. + * + * `null` is the common case and not a failure: `build-adjacency.mjs` emits a + * span only where the shared run occurs in exactly one place on *both* sides, + * because naming one of several occurrences would be a guess. 2,544 of 2,996 + * look-alike edges carry one; the rest make no claim about where they match, and + * a caller shows its plain note instead — the same fallback the twelve-ayah + * table left in place for every pair it did not cover. + * + * The source page is read off `dir.dPage`, which is the target's page minus the + * source's, rather than passed in. The edge is self-describing that way, and the + * arithmetic cannot drift out of step with the page the edge actually names. + */ +export function wordDiff(edge: Edge, fromKey: string): WordDiff | null { + if (!edge.span || !edge.toSpan) return null; + const from = edge.span.from; + const to = edge.toSpan.from; + if (from[1] < from[0] || to[1] < to[0]) return null; + return { + from: { key: bareKey(fromKey), page: edge.page - edge.dir.dPage, shared: from }, + to: { key: bareKey(edge.to), page: edge.page, shared: to }, + }; +} + +/** + * The runs of an ayah that are *not* shared, given everything of it that is on + * this page. + * + * Returned as ranges rather than as a set of indices because the caller paints + * them with {@link WordIndex.bandsFor}, which collapses a run to one rectangle + * per line of the print and wants the run whole. + * + * Both ends can be empty — two ayahs that differ only in their tail share + * everything before it — and an ayah that continues onto the next page is + * clamped to what this page holds, so the leftover named here is the leftover + * the reader can actually see. + */ +export function divergentRuns( + present: WordSpanRange, + shared: readonly [number, number], +): Array { + const runs: Array = []; + if (shared[0] > present.from) runs.push([present.from, Math.min(shared[0] - 1, present.to)]); + if (shared[1] < present.to) runs.push([Math.max(shared[1] + 1, present.from), present.to]); + return runs; +} diff --git a/packages/core/src/verse-text.test.ts b/packages/core/src/verse-text.test.ts deleted file mode 100644 index e36ecd1..0000000 --- a/packages/core/src/verse-text.test.ts +++ /dev/null @@ -1,41 +0,0 @@ -import { describe, expect, it } from "vitest"; -import { diffPair, VERSE_TEXT, verseTokens } from "./verse-text.js"; - -describe("verse-text · fixture", () => { - it("carries the 12 mock ayahs", () => { - expect(Object.keys(VERSE_TEXT)).toHaveLength(12); - for (const k of ["2:40", "2:48", "2:123", "82:19"]) { - expect(VERSE_TEXT[k]).toBeDefined(); - } - }); - - it("2:48 vs 2:123 carries the شفاعة/عدل swap as class-1/2 tokens", () => { - // The signature diff: 2:48 marks شَفَاعَةٌ divergent, 2:123 marks عَدْلٌ. - const t48 = VERSE_TEXT["2:48"]!; - const t123 = VERSE_TEXT["2:123"]!; - expect(t48.some((tok) => tok.text.includes("شَفَاعَةٌ") && tok.cls !== 0)).toBe(true); - expect(t123.some((tok) => tok.text.includes("عَدْلٌ") && tok.cls !== 0)).toBe(true); - }); -}); - -describe("verse-text · lookup", () => { - it("verseTokens accepts bare and canonical keys", () => { - expect(verseTokens("2:48")).toBe(VERSE_TEXT["2:48"]); - expect(verseTokens("quran/hafs-kfqc/2:48")).toBe(VERSE_TEXT["2:48"]); - expect(verseTokens("quran/hafs-kfqc/9:99")).toBeNull(); - }); -}); - -describe("verse-text · diffPair", () => { - it("pairs two vendored ayahs into from/to sides", () => { - const d = diffPair("quran/hafs-kfqc/2:48", "quran/hafs-kfqc/2:123"); - expect(d).not.toBeNull(); - expect(d!.from.tokens).toBe(VERSE_TEXT["2:48"]); - expect(d!.to.tokens).toBe(VERSE_TEXT["2:123"]); - }); - - it("returns null when either ayah has no vendored text", () => { - expect(diffPair("2:48", "9:99")).toBeNull(); - expect(diffPair("9:99", "2:48")).toBeNull(); - }); -}); diff --git a/packages/core/src/verse-text.ts b/packages/core/src/verse-text.ts deleted file mode 100644 index a021d70..0000000 --- a/packages/core/src/verse-text.ts +++ /dev/null @@ -1,120 +0,0 @@ -/** - * L3 verse-text token table for the diff view (spec §3 diff), keyed by bare - * `surah:ayah`. Each token carries a diff class: 0 = common/unchanged, while - * 1 and 2 mark the two divergent readings being compared. This is Loop-3 demo - * data — the mock's hand-verified tokens — that Loop 4's ETL will replace with - * a real per-ayah token source. The Arabic text is copied EXACTLY, including - * all harakat/diacritics: it is scripture and must not be normalized. - */ - -export type DiffClass = 0 | 1 | 2; - -export interface DiffToken { - readonly text: string; - readonly cls: DiffClass; -} - -export const VERSE_TEXT: Readonly> = { - '2:40': [ - { text: 'يَا بَنِي إِسْرَائِيلَ اذْكُرُوا نِعْمَتِيَ الَّتِي أَنْعَمْتُ عَلَيْكُمْ', cls: 0 }, - { text: 'وَأَوْفُوا بِعَهْدِي أُوفِ بِعَهْدِكُمْ وَإِيَّايَ فَارْهَبُونِ', cls: 1 }, - ], - '2:47': [ - { text: 'يَا بَنِي إِسْرَائِيلَ اذْكُرُوا نِعْمَتِيَ الَّتِي أَنْعَمْتُ عَلَيْكُمْ', cls: 0 }, - { text: 'وَأَنِّي فَضَّلْتُكُمْ عَلَى الْعَالَمِينَ', cls: 2 }, - ], - '2:122': [ - { text: 'يَا بَنِي إِسْرَائِيلَ اذْكُرُوا نِعْمَتِيَ الَّتِي أَنْعَمْتُ عَلَيْكُمْ', cls: 0 }, - { text: 'وَأَنِّي فَضَّلْتُكُمْ عَلَى الْعَالَمِينَ', cls: 2 }, - ], - '2:48': [ - { text: 'وَاتَّقُوا يَوْمًا لَا تَجْزِي نَفْسٌ عَنْ نَفْسٍ شَيْئًا وَلَا يُقْبَلُ مِنْهَا', cls: 0 }, - { text: 'شَفَاعَةٌ', cls: 1 }, - { text: 'وَلَا', cls: 0 }, - { text: 'يُؤْخَذُ مِنْهَا عَدْلٌ', cls: 2 }, - { text: 'وَلَا هُمْ يُنْصَرُونَ', cls: 0 }, - ], - '2:123': [ - { text: 'وَاتَّقُوا يَوْمًا لَا تَجْزِي نَفْسٌ عَنْ نَفْسٍ شَيْئًا وَلَا يُقْبَلُ مِنْهَا', cls: 0 }, - { text: 'عَدْلٌ', cls: 2 }, - { text: 'وَلَا', cls: 0 }, - { text: 'تَنْفَعُهَا شَفَاعَةٌ', cls: 1 }, - { text: 'وَلَا هُمْ يُنْصَرُونَ', cls: 0 }, - ], - '2:58': [ - { text: 'وَإِذْ', cls: 0 }, - { text: 'قُلْنَا ادْخُلُوا', cls: 1 }, - { text: 'هَذِهِ الْقَرْيَةَ', cls: 0 }, - { text: 'فَكُلُوا', cls: 2 }, - { text: 'مِنْهَا حَيْثُ شِئْتُمْ', cls: 0 }, - { text: 'رَغَدًا وَادْخُلُوا الْبَابَ سُجَّدًا وَقُولُوا حِطَّةٌ', cls: 1 }, - { text: 'نَغْفِرْ لَكُمْ', cls: 0 }, - { text: 'خَطَايَاكُمْ', cls: 2 }, - { text: 'وَسَنَزِيدُ الْمُحْسِنِينَ', cls: 0 }, - ], - '7:161': [ - { text: 'وَإِذْ', cls: 0 }, - { text: 'قِيلَ لَهُمُ اسْكُنُوا', cls: 1 }, - { text: 'هَذِهِ الْقَرْيَةَ', cls: 0 }, - { text: 'وَكُلُوا', cls: 2 }, - { text: 'مِنْهَا حَيْثُ شِئْتُمْ', cls: 0 }, - { text: 'وَقُولُوا حِطَّةٌ وَادْخُلُوا الْبَابَ سُجَّدًا', cls: 1 }, - { text: 'نَغْفِرْ لَكُمْ', cls: 0 }, - { text: 'خَطِيئَاتِكُمْ', cls: 2 }, - { text: 'سَنَزِيدُ الْمُحْسِنِينَ', cls: 0 }, - ], - '2:60': [ - { text: 'فَقُلْنَا', cls: 1 }, - { text: 'اضْرِبْ بِعَصَاكَ الْحَجَرَ', cls: 0 }, - { text: 'فَانْفَجَرَتْ', cls: 2 }, - { text: 'مِنْهُ اثْنَتَا عَشْرَةَ عَيْنًا', cls: 0 }, - ], - '7:160': [ - { text: 'أَنِ', cls: 1 }, - { text: 'اضْرِبْ بِعَصَاكَ الْحَجَرَ', cls: 0 }, - { text: 'فَانْبَجَسَتْ', cls: 2 }, - { text: 'مِنْهُ اثْنَتَا عَشْرَةَ عَيْنًا', cls: 0 }, - ], - '2:45': [ - { text: 'وَاسْتَعِينُوا بِالصَّبْرِ وَالصَّلَاةِ', cls: 0 }, - { text: 'وَإِنَّهَا لَكَبِيرَةٌ إِلَّا عَلَى الْخَاشِعِينَ', cls: 1 }, - ], - '2:153': [ - { text: 'يَا أَيُّهَا الَّذِينَ آمَنُوا', cls: 2 }, - { text: 'اسْتَعِينُوا بِالصَّبْرِ وَالصَّلَاةِ', cls: 0 }, - { text: 'إِنَّ اللَّهَ مَعَ الصَّابِرِينَ', cls: 1 }, - ], - '82:19': [ - { text: 'يَوْمَ لَا تَمْلِكُ نَفْسٌ لِنَفْسٍ شَيْئًا وَالْأَمْرُ يَوْمَئِذٍ لِلَّهِ', cls: 0 }, - ], -}; - -/** - * Look up the diff-token array for an ayah. Accepts either a bare `"2:48"` key - * or a canonical `"quran/hafs-kfqc/2:48"` key — everything up to and including - * the last `/` is stripped before lookup. Returns `null` if the key is absent. - */ -export function verseTokens(key: string): readonly DiffToken[] | null { - const bare = key.slice(key.lastIndexOf('/') + 1); - return VERSE_TEXT[bare] ?? null; -} - -/** One side of a diff: the ayah's key and its pre-classified tokens. */ -export interface DiffSide { - readonly key: string; - readonly tokens: readonly DiffToken[]; -} - -/** - * The two rows of a token diff (spec §3): the source ayah ("here") and the hop - * target. Each token is already class-tagged in the fixture (the mock does no - * runtime alignment — the divergent readings are pre-marked), so this just pairs - * the two token arrays. Returns `null` when either ayah has no vendored text, so - * a caller can fall back to the plain note rather than render an empty diff. - */ -export function diffPair(fromKey: string, toKey: string): { from: DiffSide; to: DiffSide } | null { - const from = verseTokens(fromKey); - const to = verseTokens(toKey); - if (!from || !to) return null; - return { from: { key: fromKey, tokens: from }, to: { key: toKey, tokens: to } }; -} diff --git a/packages/etl/scripts/build-roots.mjs b/packages/etl/scripts/build-roots.mjs index cf3c5ad..c54660c 100644 --- a/packages/etl/scripts/build-roots.mjs +++ b/packages/etl/scripts/build-roots.mjs @@ -83,10 +83,24 @@ const BUCKETS = 32; const GZ_LIMIT = 50 * 1024; /* ------------------------------------------------------------------ */ -/* Buckwalter → Arabic (corpus.quran.com/java/buckwalter.jsp, verbatim).*/ +/* Buckwalter → Arabic. */ +/* */ +/* The scheme is Tim Buckwalter's — a one-character-per-letter Arabic */ +/* transliteration published with the LDC's Arabic morphological */ +/* analyser and in general use since. The rows below the letters are its */ +/* "XML-safe" variants plus the extra codepoints the Quranic Arabic */ +/* Corpus needs for a mus'haf's pause, madd and small-letter marks, */ +/* which its own transliteration page sets out. */ +/* */ +/* This comment used to cite that page and say "verbatim", which read as */ +/* if a table had been copied out of the corpus rather than as a */ +/* statement about which scheme is in use. What is here is a character */ +/* correspondence — the alphabet on one side and the alphabet on the */ +/* other — and it has to match theirs exactly or the roots come out */ +/* wrong. See the corpus's entry in SOURCES.md for its terms. */ /* ------------------------------------------------------------------ */ -/** The corpus's own transliteration table — roots and lemmas arrive in it. */ +/** The scheme the corpus writes its roots and lemmas in, read back to Arabic. */ const BUCKWALTER = { "'": "ء", ">": "أ", "&": "ؤ", "<": "إ", "}": "ئ", A: "ا", b: "ب", p: "ة", t: "ت", v: "ث", diff --git a/packages/etl/scripts/build-tajweed.mjs b/packages/etl/scripts/build-tajweed.mjs index 26d0e1b..ed95b90 100644 --- a/packages/etl/scripts/build-tajweed.mjs +++ b/packages/etl/scripts/build-tajweed.mjs @@ -32,8 +32,9 @@ * `word-alignment.pin.json` joins the print's word index to the Quranic Arabic * Corpus's. Both were the named blockers and both are discharged. What is left * is specific to this file: the annotations below are codepoint offsets into - * each ayah's Tanzil Uthmani text, and this repo holds no Quran text and will - * not (see `morphology.mjs`). Turning an offset into a word means joining a + * each ayah's Tanzil Uthmani text, and nothing vendored here is that text (see + * `morphology.mjs` for the rule and `gate:scripture` for the check). Turning an + * offset into a word means joining a * third segmentation to the two we now have. `docs/design/word-indexing.md` ⑤ * carries what would answer it and what it would cost — an ETL measurement, * not the rendering change this comment used to promise. diff --git a/packages/etl/scripts/lib/tajweed-fold.test.mjs b/packages/etl/scripts/lib/tajweed-fold.test.mjs index a3f461d..9073a42 100644 --- a/packages/etl/scripts/lib/tajweed-fold.test.mjs +++ b/packages/etl/scripts/lib/tajweed-fold.test.mjs @@ -9,9 +9,10 @@ * absolute rather than relative to the prefix. Every one of those is a way the * inspector could show a confident wrong screen. * - * The words below are the ordinary ones — al-Fatiha's opening, which is also - * the basmala the fold prepends — because a correction is easiest to read as a - * difference in a string a reader recognises. + * The words below are bare letters wherever the assertion is arithmetic, and a + * real one only where the codepoint under test *is* the subject — a correction + * that fires on a small high mark cannot be tested without that mark. Keeping + * the two apart is what lets this file hold single specimens and no passage. */ import { describe, expect, it } from "vitest"; import { @@ -38,8 +39,18 @@ import { /** `{ hafs, waw, mark }` without the ceremony. */ const w = (hafs, extra = {}) => ({ hafs, waw: false, mark: false, ...extra }); -/** 1:1 — «بِسۡمِ ٱللَّهِ ٱلرَّحۡمَٰنِ ٱلرَّحِيمِ», as the print numbers it. */ -const BASMALA = [w("بِسۡمِ"), w("ٱللَّهِ"), w("ٱلرَّحۡمَٰنِ"), w("ٱلرَّحِيمِ")]; +/** + * A four-word stand-in for the prefix the fold prepends. Four because that is + * how many words the print numbers there, and the host count below asserts it. + * + * Bare letters rather than the real opening: `foldAyah` never reads a word's + * content when it builds the prefix — it joins, spaces, and counts — so the real + * text would be decoration on an arithmetic test, and `gate:scripture` refuses a + * running passage in source wherever it finds one. The corrections that *are* + * content-sensitive are tested on single words, below, where the codepoint under + * test is the point. + */ +const PREFIX = [w("ا"), w("ب"), w("ج"), w("د")]; const str = (cps) => cps.join(""); const all = new Set(ALL_CORRECTIONS); @@ -192,11 +203,11 @@ describe("hasBasmala", () => { }); describe("foldAyah", () => { - const args = { surah: 2, ayah: 1, words: [w("الٓمٓ")], basmala: BASMALA, indices: [1] }; + const args = { surah: 2, ayah: 1, words: [w("الٓمٓ")], basmala: PREFIX, indices: [1] }; it("prepends the basmala and one space when `basmala` is on", () => { const { cps, prefix } = foldAyah({ ...args, on: all }); - expect(str(cps)).toBe("بِسۡمِ ٱللَّهِ ٱلرَّحۡمَٰنِ ٱلرَّحِيمِ الٓمٓ"); + expect(str(cps)).toBe("ا ب ج د الٓمٓ"); expect(prefix).toBe(str(cps).indexOf("الٓمٓ")); }); @@ -209,7 +220,7 @@ describe("foldAyah", () => { it("folds the prefix under the same corrections as the body", () => { // A mark inside the basmala must vanish from the prefix too, or the ayah's // own offsets shift by two and every span in it is wrong. - const basmala = [w("بِسۡمِ"), w("ۖ", { mark: true }), w("ٱللَّهِ")]; + const basmala = [w("ا"), w("ۖ", { mark: true }), w("ب")]; const on = new Set(ALL_CORRECTIONS); const withMarks = foldAyah({ ...args, basmala, on: new Set(["basmala", "waw"]) }); const without = foldAyah({ ...args, basmala, on }); diff --git a/packages/etl/scripts/morphology.mjs b/packages/etl/scripts/morphology.mjs index bc6ce7c..9dd48f7 100644 --- a/packages/etl/scripts/morphology.mjs +++ b/packages/etl/scripts/morphology.mjs @@ -11,8 +11,18 @@ * computed a different way, and the first time the two disagreed the reader * would be the one assumed wrong. * - * There is no Quran text in this repo and there will not be. What is vendored - * is the Quranic Arabic Corpus *morphology*: one row per segment, carrying the + * Nothing vendored for this file, and nothing shipped from it, is Quran text. + * + * That rule was stated for years as *"there is no Quran text in this repo and + * there will not be"*, and the unscoped half of it was false: twelve verses had + * been typed into a source file and shipped in the bundle, and a pipeline test + * held a four-word phrase. Nothing was checking — `gate:notext` sounds as though + * it would and does not; it forbids `` elements in page artwork, for a + * rendering reason. `gate:scripture` is the check the sentence was standing in + * for, and it fails on any run of three consecutive fully-vowelled words + * anywhere in the tree. + * + * What is vendored is the Quranic Arabic Corpus *morphology*: one row per segment, carrying the * segment's Buckwalter form and its features. Concatenating a word's segments * reconstructs that word — which is enough to ask "do these two ayahs share * phrasing" and not enough to be a mushaf. The roots live in the shipped diff --git a/packages/etl/scripts/probe-encodings.mjs b/packages/etl/scripts/probe-encodings.mjs index 13bc8c0..56c796a 100644 --- a/packages/etl/scripts/probe-encodings.mjs +++ b/packages/etl/scripts/probe-encodings.mjs @@ -36,12 +36,18 @@ * * ## The one rule this script exists under * - * **There is no Quran text in this repo and there will not be.** The report is - * full of Arabic; every codepoint of it is read at runtime from the gitignored - * `.cache/words/`, and the output goes to `packages/etl/out/`, which is - * gitignored. Committing a generated report would vendor the mus'haf through - * the back door. `gate:notext` and `gate:text-sources` are the check; the - * output path is the reason they stay green. + * **Nothing this script writes is Quran text, and nothing it writes is + * committed.** The report is full of Arabic; every codepoint of it is read at + * runtime from the gitignored `.cache/words/`, and the output goes to + * `packages/etl/out/`, which is gitignored. Committing a generated report would + * vendor the mus'haf through the back door. + * + * `gate:scripture` is the check, and it is newer than this comment, which named + * two gates that do not do this. `gate:notext` forbids `` elements in page + * artwork — a rendering requirement, for a Safari paint bug, wearing a name that + * reads like a scripture check. `gate:text-sources` asks that rendered text + * traces to a declared edition, which is a different question again. The output + * path is still the reason all three stay green. * * Not a gate, and never will be: no cache, nothing to read. Named `probe-` for * exactly the reason `probe-tajweed-words.mjs` is. diff --git a/packages/etl/scripts/probe-mark-ink.mjs b/packages/etl/scripts/probe-mark-ink.mjs index 10857e8..743d2c9 100644 --- a/packages/etl/scripts/probe-mark-ink.mjs +++ b/packages/etl/scripts/probe-mark-ink.mjs @@ -62,7 +62,9 @@ * * ## The one rule this script exists under * - * **There is no Quran text in this repo and there will not be.** Marks are + * **Nothing this script writes is Quran text, and nothing it writes is + * committed** — the rule as `morphology.mjs` now states it, and as + * `gate:scripture` enforces it. Marks are * reported by page, word index and drawn name. The word's own letters are read * from the cache at runtime and never written to a file outside * `packages/etl/out/`, which is gitignored — the same arrangement diff --git a/packages/etl/scripts/probe-mark-labels.mjs b/packages/etl/scripts/probe-mark-labels.mjs index c16827c..a2c9d8b 100644 --- a/packages/etl/scripts/probe-mark-labels.mjs +++ b/packages/etl/scripts/probe-mark-labels.mjs @@ -73,9 +73,11 @@ * * ## The one rule this script exists under * - * **There is no Quran text in this repo and there will not be.** Marks are - * reported by page, word index and drawn name. Nothing is written outside - * `packages/etl/out/`, which is gitignored. + * **Nothing this script writes is Quran text, and nothing it writes is + * committed.** Marks are reported by page, word index and drawn name, and + * nothing is written outside `packages/etl/out/`, which is gitignored. The rule + * used to be stated about the whole repository, which was not true of it; see + * `morphology.mjs` and `gate:scripture`. * * Usage: * node packages/etl/scripts/probe-mark-labels.mjs # every mark in the cache diff --git a/scripts/gate-notices.mjs b/scripts/gate-notices.mjs index a62f2d0..2158068 100644 --- a/scripts/gate-notices.mjs +++ b/scripts/gate-notices.mjs @@ -18,7 +18,7 @@ * colophon, which is a real check on whether a quotation drifted and no check at * all on whether the SET of quotations is still complete. * - * WHAT IT CHECKS. Three things that could each drift alone, tied together: + * WHAT IT CHECKS. Four things that could each drift alone, tied together: * * 1. THE TABLE AGAINST THE BYTES. Every row of LICENSES.md's bucket table * whose terms say "(inherited)" must be declared below, and vice versa — @@ -38,6 +38,14 @@ * a real `### ` entry in SOURCES.md, so a typo or a renamed project * fails instead of quietly matching nothing. * + * 4. THE DECLARATIONS AGAINST THE FOLDER. Everything that actually ships under + * apps/web/public/assets/ is either an inherited bucket or a named entry + * below, and LICENSES.md mentions it either way. Added 2026-08-16, because + * checks 1–3 compared a declaration against a table and never looked in the + * directory — so two trees that were in neither, 604 files of word geometry + * and the manifest, passed this gate every time it ran. A check that reads + * only what it was told about cannot report what it was not told about. + * * THE VERDICT VOCABULARY. Three words, and the third is the interesting one. * * named this input's terms are on the row and in the notice. @@ -62,7 +70,10 @@ * the exact shape of the original drift, a builder quietly reaching one more * upstream: "reads packages/etl/data/tajweed/… and gate-notices.mjs has no verdict * for it". Dropping Waqar144 from the bucket row: "does not name Waqar144". The - * middle one is the one that matters; the other two are cheap. + * middle one is the one that matters; the other two are cheap. Check 4 was broken + * the same way when it was added: an empty directory under assets/ — "ships and + * nothing declares it" — and the declaration for the word geometry removed from + * LICENSES.md — "never mentions assets/words, which ships". Both restored. * * WHAT IT DOES NOT SEE, said out loud rather than discovered later. The trace * follows *relative* imports inside packages/etl and reads string literals; it @@ -97,10 +108,15 @@ const BUCKETS = [ sources: ["quranic-arabic-corpus"], reads: { "quranic-corpus-morphology-0.4.txt": "named", - // Every root-ayah pair is filed by ayah, never by page; the table is read - // here only to validate that the ayah numbering is the one this edition - // uses. No page number reaches a root shard. - "ayah-pages.json": "ours", + // This read was declared "ours" on the grounds that "no page number + // reaches a root shard". That was false, and measurably so: + // `build-roots.mjs` writes `pageOf(abs)` into every one of 44,431 + // occurrence tuples. It is the same table, reaching the same kind of + // output, as the deferral the adjacency bucket already carries — so it + // defers to the same question, in the row that widened it to cover this + // tree. Declaring one bucket's read of a file "ours" while another + // bucket's read of that same file is an open question was the drift. + "ayah-pages.json": "open:the-pagination-question-covers-three-outputs", }, }, { @@ -131,6 +147,37 @@ const BUCKETS = [ }, ]; +/** + * Everything else that ships under `apps/web/public/assets/`. + * + * The buckets above are the trees whose terms are *inherited*, and until this + * list existed that was the only part of the folder anything read. The check was + * therefore blind by construction: it compared a declaration against a table, + * and never once asked what was actually in the directory. Two shipped trees sat + * outside both for months — 604 files of word geometry and a 24 KB manifest — + * and no file in the repository mentioned either. + * + * So every entry of that folder must now appear here or in `BUCKETS`, and every + * entry here must name where a reader goes to find out what it is. The verdicts + * are the same three words the buckets use, with the same meanings. + */ +const UNBUCKETED = { + // KFGQPC's artwork, which is not ours to relicense and is deliberately not a + // bucket: an inherited bucket promises a NOTICE.txt travelling with the data, + // and this tree's terms are the Complex's own rather than a grant we can + // restate. LICENSES.md says so under "What we do not license". + pages: { verdict: "named" }, + // Rectangles this project measured onto its own page frame. No byte of the + // print they were measured from ships, and the grant behind that print obliges + // no attribution — so the tree carries no inherited terms and the source entry + // is a courtesy pointer, not a condition. + words: { verdict: "ours", source: "word-geometry-mushafdatabase" }, + // Ships the whole 6,236-entry page table verbatim, which is the third of the + // three outputs the pagination question covers. Having a row in the licence + // table is what it was missing; what the row cannot yet say is settled. + "manifest.json": { verdict: "open:the-pagination-question-covers-three-outputs" }, +}; + /** Vendored third-party inputs: a real file under a data directory that has a * PROVENANCE.md, excluding this project's own pins and probe results. */ function vendoredInputs() { @@ -295,6 +342,49 @@ for (const b of BUCKETS) { } } +/* 6 ─ The folder against the declarations. Everything shipped is spoken for. */ +const shipped = existsSync(ASSETS) ? readdirSync(ASSETS) : []; +if (shipped.length === 0) { + problems.push("apps/web/public/assets/ is empty or missing — the app ships no data"); +} +for (const entry of shipped) { + if (declared.has(entry) || entry in UNBUCKETED) continue; + problems.push( + `apps/web/public/assets/${entry} ships and nothing declares it — it is neither an ` + + `inherited bucket nor a named entry in gate-notices.mjs. Say what its terms are.`, + ); +} +for (const [entry, decl] of Object.entries(UNBUCKETED)) { + if (!existsSync(join(ASSETS, entry))) { + problems.push( + `gate-notices.mjs names apps/web/public/assets/${entry}, which does not ship — ` + + `a stale declaration hides the next real one`, + ); + continue; + } + // The licence table is what a reader opens, so every shipped tree has to be + // findable there. This is the check the two undeclared trees would have failed. + if (!licenses.includes(`assets/${entry}`)) { + problems.push(`LICENSES.md never mentions assets/${entry}, which ships`); + } + if (decl.source && !documented.has(decl.source)) { + problems.push(`assets/${entry} names "${decl.source}", which SOURCES.md has no entry for`); + } + if (decl.verdict.startsWith("open:")) { + const id = decl.verdict.slice(5); + if (!knownIssues.has(id)) { + problems.push(`assets/${entry} defers to issue "${id}", which docs/issues.json does not have`); + } else if (!openIssues.has(id)) { + problems.push( + `assets/${entry} still defers to issue "${id}", which is now closed — ` + + `the question was answered, so the declaration has to say what the answer was`, + ); + } else { + notes.push(`assets/${entry} — open, tracked as ${id}`); + } + } +} + if (problems.length) { console.error("gate:notices — FAIL:"); for (const p of problems) console.error(` - ${p}`); @@ -305,6 +395,7 @@ const trees = BUCKETS.length; const traced = BUCKETS.reduce((n, b) => n + Object.keys(b.reads).length, 0); console.log( `gate:notices — OK (${trees} inherited trees, every notice shipped and naming its upstreams; ` + - `${traced} vendored inputs traced, all accounted for)`, + `${traced} vendored inputs traced, all accounted for; ` + + `${shipped.length} entries under assets/, each one declared)`, ); for (const n of notes) console.log(` ${n}`); diff --git a/scripts/gate-scripture.mjs b/scripts/gate-scripture.mjs new file mode 100644 index 0000000..685dc6e --- /dev/null +++ b/scripts/gate-scripture.mjs @@ -0,0 +1,241 @@ +#!/usr/bin/env node +/** + * CI gate: no source file may hold a running passage of scripture. + * + * WHY THIS EXISTS. Twenty-two times across twenty files this repo asserts some + * version of "there is no Quran text here" — including in three shipped + * NOTICE.txt files. When it was finally measured, the unscoped form of that + * claim was false in two places, and one of them shipped: 48 fully-vowelled + * token strings in `packages/core/src/verse-text.ts`, every one byte-present in + * the web bundle, sitting there since Loop 3 as "demo data Loop 4 will replace". + * + * Nothing caught it, and the reason is worth stating where somebody will read + * it: `gate:notext` sounds like it would. It does not. That gate forbids `` + * elements inside page SVGs because such an element under `content-visibility: + * auto` can fail to paint in Safari. It is a rendering requirement wearing a + * name that reads like a scripture check, and for two loops it was mistaken for + * one. Hence this file rather than a widening of that one. + * + * WHY IT MATTERS MORE THAN TIDINESS. Takedown notices over scripture are + * routinely path-scoped *into* code repositories — at a data directory, and in + * at least one case at a test-fixtures directory. The blast radius tracks the + * directory holding the text, not what the project is for. A test fixture is + * therefore not a safe place to keep a passage, which is exactly the assumption + * the second site here was resting on. + * + * WHAT IT MEASURES, AND WHY THAT AND NOT "IS THERE ARABIC". About a hundred + * tracked source files carry Arabic and nearly all of them should: the Arabic + * locale is our own translation, surah names are names, and the pipeline keeps + * single vowelled words as specimens because tajweed and encoding edge cases are + * *about* particular orthography. Flagging Arabic would flag all of it and be + * switched off inside a week. + * + * What separates scripture from the rest is not the script and not even the + * vowelling — it is how much vowelled text runs *consecutively*. One vowelled + * word is a specimen; a phrase is a passage a recipient can recover. Measured + * across the whole tree, longest run of consecutive fully-vowelled words: + * + * packages/core/src/verse-text.ts 11 + * packages/etl/scripts/lib/tajweed-fold.test.mjs 5 + * ... everything else 1 or 2 + * + * Nothing sits at 3 or 4. The threshold below is not a taste call — it is placed + * in a gap that the tree itself provides, and if that gap ever closes the number + * needs re-deriving rather than nudging. + */ +import { execSync } from "node:child_process"; +import { existsSync, readFileSync } from "node:fs"; +import { join } from "node:path"; + +const ROOT = new URL("..", import.meta.url).pathname; + +/* ------------------------------------------------------------------ */ +/* The two rules */ +/* ------------------------------------------------------------------ */ + +/** + * A run of this many consecutive fully-vowelled words fails, and no list can + * excuse it. Three, because the tree's own measurement leaves 3 and 4 empty. + */ +const PHRASE = 3; + +/** + * A word counts as fully vowelled when at least this share of its letters carry + * a mark. Vowelled scripture sits at 0.85–1.0; Arabic prose that uses the + * occasional shadda for disambiguation sits at 0.43 and below on the same + * measure, so the midpoint separates them with room on both sides. + */ +const VOWELLED = 0.5; + +/** + * A file holding this many vowelled words is a *collection* — somebody + * assembled it — and has to say so in SPECIMENS below. Below it, a vowelled + * word is incidental: a domain term in a UI string, one label in a test. + */ +const COLLECTION = 5; + +/** + * The specimen collections, each with the reason it is one. This list cannot + * excuse a phrase — PHRASE is checked first and independently. It exists so the + * lone vowelled words scattered through the pipeline read as reviewed rather + * than as nobody having looked. + */ +const SPECIMENS = { + "packages/etl/scripts/lib/tajweed-fold.mjs": + "per-word tajweed edge cases; the fold is about particular orthography and cannot be tested without it", + "packages/etl/scripts/lib/tajweed-fold.test.mjs": "the cases above, asserted", + "packages/etl/scripts/probe-tajweed-words.mjs": + "the same edge cases, reached from the print's own per-word text", + "packages/etl/scripts/probe-diacritics.mjs": "one word per diacritic shape the print uses", + "packages/etl/scripts/lib/mark-join.mjs": "per-word join cases; a join is a fact about one word", + "packages/etl/scripts/lib/mark-join.test.mjs": "the cases above, asserted", + "packages/etl/scripts/lib/segmentation.mjs": + "the named print-to-corpus alignment exceptions, each one word", + "apps/web/src/messages/ar.gen.ts": "our own Arabic interface strings, generated from our own catalog", + "apps/web/src/components/RootLens.test.tsx": "one root, vowelled, as a lens fixture", + "packages/core/src/roots.test.ts": "the same root", +}; + +/* ------------------------------------------------------------------ */ +/* Character classes, written as escapes on purpose */ +/* ------------------------------------------------------------------ */ + +// This file is one of the ones the gate reads. Arabic literals here would be +// self-referential noise at best and a false positive at worst, so every range +// below is an escape. +const ARABIC = "\\u0600-\\u06FF\\u0750-\\u077F\\uFB50-\\uFDFF\\uFE70-\\uFEFF"; +// Letters only: tatweel (U+0640) is a connector that carries marks without being +// one, and counting it as a letter would dilute the ratio on exactly the words +// that use it. +const LETTER = new RegExp("[\\u0621-\\u064A\\u0671-\\u06D3]", "g"); +// Harakat, superscript alef, the hamza/madda companions, and the small high +// marks the Uthmani print uses. +const MARK = new RegExp("[\\u064B-\\u0652\\u0670\\u0653-\\u0655\\u06D6-\\u06ED]", "g"); +const HAS_ARABIC = new RegExp("[" + ARABIC + "]"); +const WORD = new RegExp("[" + ARABIC + "]+", "g"); +// Anything that is neither Arabic nor whitespace ends a run — see scan(). +const BREAK = new RegExp("[^\\s" + ARABIC + "]+"); + +/** Whether a token is fully vowelled in the sense defined above. */ +function isVowelled(word) { + const letters = (word.match(LETTER) || []).length; + if (letters < 2) return false; + return (word.match(MARK) || []).length / letters >= VOWELLED; +} + +/** + * The longest run of consecutive vowelled words, and how many there are in all. + * + * Runs break on anything that is not Arabic or whitespace — a quote, a comma in + * the source, a `+`. That is deliberate: two adjacent string literals are two + * fragments, and gluing them would let the gate be defeated by pressing Enter. + * It also means the count is a floor, never an overstatement. + */ +function scan(text) { + let longest = 0; + let total = 0; + let sample = ""; + for (const line of text.split("\n")) { + for (const chunk of line.split(BREAK)) { + if (!HAS_ARABIC.test(chunk)) continue; + let run = 0; + for (const word of chunk.match(WORD) || []) { + if (!isVowelled(word)) { + run = 0; + continue; + } + total++; + run++; + if (run > longest) { + longest = run; + sample = chunk.trim().slice(0, 60); + } + } + } + } + return { longest, total, sample }; +} + +/* ------------------------------------------------------------------ */ +/* Which files */ +/* ------------------------------------------------------------------ */ + +// Same enumeration as gate-text-sources.mjs, for the reason recorded there: a +// gate that lists only tracked files has a different scope on the machine that +// writes the code than on the machine that gates it, and the difference is +// invisible from the machine that gates it. `make ci` is the mirror people run +// before staging, and that is the run this gate most needs to be part of. +const SOURCE_RE = /^(packages|apps|scripts)\/.*\.(ts|tsx|mjs|js|jsx)$/; +const EXCLUDE_RE = /^(packages\/etl\/data|.*\/(dist|coverage|playwright-report|test-results))\//; + +const files = [ + ...new Set( + execSync("git ls-files --cached --others --exclude-standard", { cwd: ROOT, encoding: "utf8" }) + .trim() + .split("\n"), + ), +].filter((f) => SOURCE_RE.test(f) && !EXCLUDE_RE.test(f)); + +if (files.length === 0) { + console.error("gate:scripture — no source files matched; the filter is stale"); + process.exit(1); +} + +const passages = []; +const unlisted = []; +const listed = new Set(); + +for (const rel of files) { + const abs = join(ROOT, rel); + // `git ls-files` reads the index, so a file deleted in the working tree but + // not yet staged is listed and no longer on disk. Same reasoning as + // gate-text-sources.mjs: a gate that crashes names the wrong problem. + if (!existsSync(abs)) continue; + const { longest, total, sample } = scan(readFileSync(abs, "utf8")); + if (total === 0) continue; + if (longest >= PHRASE) passages.push({ rel, longest, total, sample }); + else if (total >= COLLECTION && !(rel in SPECIMENS)) unlisted.push({ rel, total }); + if (rel in SPECIMENS) listed.add(rel); +} + +// A list that names files which no longer hold specimens is a list nobody has +// read. Same insistence the other registers make. +const stale = Object.keys(SPECIMENS).filter((rel) => !listed.has(rel)); + +const problems = []; + +if (passages.length > 0) { + problems.push("running scripture in source — a passage a recipient can recover:"); + for (const p of passages) { + problems.push(` ${p.rel} — ${p.longest} consecutive vowelled words (${p.total} in all)`); + problems.push(` ${p.sample}`); + } + problems.push( + " This is not excusable by SPECIMENS. Draw what the reader needs from the page artwork\n" + + " and the word shards, which ship already, or hold the case as single words.", + ); +} + +if (unlisted.length > 0) { + problems.push( + `specimen collections not named in SPECIMENS (${COLLECTION}+ vowelled words in one file):`, + ); + for (const u of unlisted) problems.push(` ${u.rel} — ${u.total} vowelled words`); + problems.push(" Add each with the reason it is a collection, or remove the words."); +} + +if (stale.length > 0) { + problems.push("SPECIMENS names files that no longer hold vowelled words:"); + for (const rel of stale) problems.push(` ${rel}`); +} + +if (problems.length > 0) { + console.error("gate:scripture — FAIL:"); + for (const line of problems) console.error(line.startsWith(" ") ? line : " " + line); + process.exit(1); +} + +console.log( + `gate:scripture — OK (${files.length} sources, no passage of ${PHRASE}+ vowelled words, ` + + `${listed.size} specimen collection(s) named)`, +); diff --git a/scripts/probe-reference.mjs b/scripts/probe-reference.mjs index 5b58a1a..18a5b41 100644 --- a/scripts/probe-reference.mjs +++ b/scripts/probe-reference.mjs @@ -31,10 +31,11 @@ * * WHAT IT DOES NOT READ. `--page-table` asks api.quran.com for verse *keys* and * nothing else: no `fields` parameter, so no `text_uthmani`, no translation, no - * transliteration. The repo's standing rule is "There is no Quran text in this - * repo and there will not be" (packages/etl/scripts/morphology.mjs), and the - * cheapest way to keep a probe honest is for the text never to cross the wire in - * the first place. Nothing here writes a file. + * transliteration. The repo's standing rule is that nothing it vendors and + * nothing it ships is Quran text (packages/etl/scripts/morphology.mjs states it, + * `gate:scripture` enforces it), and the cheapest way to keep a probe honest is + * for the text never to cross the wire in the first place. Nothing here writes a + * file. * * Usage: * node scripts/probe-reference.mjs # reachability, ~15s From 7fe43be3285b0998a4e997c500257a44fad922ae Mon Sep 17 00:00:00 2001 From: Omar Eid Date: Sun, 16 Aug 2026 20:03:51 -0500 Subject: [PATCH 36/95] One run writes both copies, and the measurement is recomputed each time MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cropping the ayahs out of the printed page fixed the panel and opened a smaller question: an ayah almost never begins and ends at the edge of a printed line, so the rectangle cut around it carries whatever else sits on those lines. The wash marks the words the two ayahs do not share. It cannot mark the difference between the rest of this ayah, which the two have in common, and a different ayah that happens to share a line — so a reader can take unmarked for shared, in an app whose whole purpose is to stop exactly that. build-crop-options.mjs draws five answers on the real artwork and measures the cost of the sixth, which is doing nothing. The sweep walks every shipped adjacency shard and computes, for both sides of all 2,544 drawable pairs, how much of the rectangle the panel draws is the ayah it names. 5,088 crops: mean 69.8%, median 70.2%, 820 of them more neighbour than ayah, 220 under a third, worst 10.3% on 80:36. It runs on every build rather than being pasted in, so the page cannot quietly stop matching the tree. Two facts in that distribution matter more than the average. 584 crops sit on one line and are already perfect, so whatever is chosen must not disturb them. And the worst case is not the longest ayah but the shortest multi-line one — both its lines are partial and there is no full line in the middle to dilute the edges, so two-line crops average 60.7%, well below three-line ones. The sharpest statement of it is not a number, and only the drawing gives it: on the app's own signature pair, the crop labelled 2:123 has its entire first line occupied by 2:122, ending in a printed ayah marker, with nothing marking it. The ayah the label names does not start until line two. The one structural change against the three sibling generators: `COPIES`. They took an --artifact flag, which meant the checked-in page and the published one were two hand-run commands that could disagree with nobody able to notice — and they did disagree once, and the published page lost its subject entirely. Here one run writes both, and the only difference between them is where the print comes from. The checked-in copy points at the page SVG with a relative URL and stays 44 KB; the published copy inlines each leaf once into a hidden defs and uses it, because the publishing host blocks every external request and a relative URL there does not error, it silently draws blank paper. Both leaves share one defs safely: neither page SVG contains a use, a url(#…) or an href="#…", and every id is stripped anyway. It shares nothing with its siblings' stylesheet, on purpose. They argue about marks on a page; this one argues about the app's own panel, so it borrows the app's tokens. The specimens keep the app's paper in both themes — the printer's ink is a fixed dark that would vanish on a dark ground — and the page says so, rather than letting it read as an oversight. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt --- scripts/build-crop-options.mjs | 857 +++++++++++++++++++++++++++++++++ 1 file changed, 857 insertions(+) create mode 100644 scripts/build-crop-options.mjs diff --git a/scripts/build-crop-options.mjs b/scripts/build-crop-options.mjs new file mode 100644 index 0000000..0db46f0 --- /dev/null +++ b/scripts/build-crop-options.mjs @@ -0,0 +1,857 @@ +#!/usr/bin/env node +/** + * Draws the options for `comparison-crop`: what the look-alike panel should + * show *around* an ayah, now that it cuts both ayahs out of the printed page + * instead of retyping them. + * + * Every specimen on the page is the real artwork with real geometry over it. + * Nothing here is a mock-up and nothing is transcribed: the bands, the washes + * and the shared range all come out of the same `WordIndex`/`divergentRuns` + * the app itself calls, and the measurement in the prose is computed by this + * script over every edge in the shipped adjacency shards. An options page that + * argued from a hand-drawn diagram would be arguing about a different picture + * than the one the reader is being asked to judge. + * + * Two copies, and one run writes both — the same rule as the placement page, + * for the same reason. They differ in exactly one thing: where the picture of + * the mus'haf comes from. + * + * The checked-in copy points at the print with a relative URL, which is right + * for somebody who opens the file inside a checkout and keeps it to a few tens + * of KB. The published copy cannot do that — it is served from claude.ai under + * a CSP that blocks every external host, so a relative URL resolves to nothing + * and every specimen is an empty overlay over blank paper. That copy inlines + * each page ONCE into a hidden `` and ``s it per specimen. Writing + * both from one pass removes the only state in which the repo can lie about + * what was published. + * + * node scripts/build-crop-options.mjs + * + * Registered in docs/decisions.json as the `builtBy` for comparison-crop; the + * reasons live in docs/decisions/comparison-crop.md. + */ +import { readFileSync, readdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { ROOT } from "./code-pointers.mjs"; +import { WordIndex, divergentRuns } from "../packages/core/dist/index.js"; + +const ASSETS = join(ROOT, "apps/web/public/assets"); +const OUT = join(ROOT, "docs/design/comparison-crop.html"); +const OUT_ARTIFACT = join(ROOT, "docs/design/comparison-crop.artifact.html"); + +/** Where the checked-in copy reaches the print from, relative to docs/design/. */ +const PRINT_HREF = (page) => `../../apps/web/public/assets/pages/hafs-kfqc/${page}.svg`; + +const COPIES = [ + { artifact: false, out: OUT }, + { artifact: true, out: OUT_ARTIFACT }, +]; + +/** The pair every specimen is drawn on, and the words the two have in common. */ +const SHARED = [1, 13]; +const SIDES = [ + { key: "2:48", page: 7, label: "البقرة · ٢:٤٨ · هنا", wash: "a" }, + { key: "2:123", page: 19, label: "البقرة · ٢:١٢٣", wash: "b" }, +]; + +/** The panel's own padding around a crop, in page units — mirrors DiffView. */ +const PAD = 2; + +// ---------------------------------------------------------------- geometry + +const shardCache = new Map(); +function indexOf(page) { + if (!shardCache.has(page)) { + try { + const raw = readFileSync(join(ASSETS, `words/hafs-kfqc/${page}.json`), "utf8"); + shardCache.set(page, new WordIndex(JSON.parse(raw))); + } catch { + shardCache.set(page, null); + } + } + return shardCache.get(page); +} + +const areaOf = (rs) => rs.reduce((n, r) => n + r.width * r.height, 0); + +function unionOf(rs) { + let l = Infinity; + let t = Infinity; + let r = -Infinity; + let b = -Infinity; + for (const q of rs) { + l = Math.min(l, q.x); + t = Math.min(t, q.y); + r = Math.max(r, q.x + q.width); + b = Math.max(b, q.y + q.height); + } + return { x: l, y: t, width: r - l, height: b - t }; +} + +/** + * The measurement the page is built on: for every side of every look-alike pair + * the panel can draw, how much of the rectangle it actually draws belongs to + * the ayah it names. + * + * Clamped at 1: a handful of sides score fractionally over 100% because two + * adjacent lines' bands overlap slightly where letters descend, and a number + * above 100 in a share column reads as a bug rather than as the rounding it is. + * The page says so in its own prose rather than hiding it here. + */ +function sweep() { + const shares = []; + let pairs = 0; + for (const file of readdirSync(join(ASSETS, "adj/hafs-kfqc"))) { + if (!file.endsWith(".json")) continue; + const surah = file.replace(".json", ""); + const shard = JSON.parse(readFileSync(join(ASSETS, "adj/hafs-kfqc", file), "utf8")); + for (const [ayah, entry] of Object.entries(shard)) { + for (const e of entry.edges ?? []) { + // No matching run on both sides means no comparison to draw at all — + // the row keeps its plain note, and there is no crop to measure. + if (!e.span?.from || !e.toSpan?.from) continue; + pairs++; + const sides = [ + { key: `${surah}:${ayah}`, page: e.page - (e.dir?.dPage ?? 0) }, + { key: e.to.split("/").pop(), page: e.page }, + ]; + for (const s of sides) { + const idx = indexOf(s.page); + if (!idx?.has(s.key)) continue; + const present = idx.span(s.key); + const bands = idx.bandsFor(s.key, present.from, present.to); + const drawn = unionOf(bands); + const box = drawn.width * drawn.height; + if (!box) continue; + shares.push({ share: Math.min(1, areaOf(bands) / box), lines: bands.length, key: s.key, page: s.page }); + } + } + } + } + shares.sort((a, b) => a.share - b.share); + return { shares, pairs }; +} + +/** Everything a specimen needs for one side of the pair. */ +function measureSide(side) { + const idx = indexOf(side.page); + const present = idx.span(side.key); + const bands = idx.bandsFor(side.key, present.from, present.to); + const u = unionOf(bands); + return { + ...side, + bands, + shared: idx.bandsFor(side.key, SHARED[0], Math.min(SHARED[1], present.to)), + washes: divergentRuns(present, SHARED).flatMap(([f, t]) => idx.bandsFor(side.key, f, t)), + frame: { x: u.x - PAD, y: u.y - PAD, width: u.width + PAD * 2, height: u.height + PAD * 2 }, + share: areaOf(bands) / (u.width * u.height), + }; +} + +/** + * The page's own markup, with every `id` stripped. + * + * Two leaves go into one document, and both declare `verse-N` ids. Nothing in + * either page references an id — no ``, no `url(#…)`, no `href="#…"`, all + * three checked — so dropping them costs nothing and removes the only way the + * two could collide. + */ +function printMarkup(page) { + return readFileSync(join(ASSETS, `pages/hafs-kfqc/${page}.svg`), "utf8") + .replace(/^[\s\S]*?]*>/, "") + .replace(/<\/svg>\s*$/, "") + .replace(/ id="[^"]*"/g, ""); +} + +function printSize(page) { + const vb = readFileSync(join(ASSETS, `pages/hafs-kfqc/${page}.svg`), "utf8").match(/viewBox="([^"]+)"/); + const [, , w, h] = vb[1].split(/\s+/).map(Number); + return { w, h }; +} + +// ----------------------------------------------------------------- drawing + +const n = (v) => Number(v.toFixed(2)); +const vb = (r) => `${n(r.x)} ${n(r.y)} ${n(r.width)} ${n(r.height)}`; +const esc = (s) => String(s).replace(/&/g, "&").replace(/ { + if (ARTIFACT) return ``; + const { w, h } = printSize(page); + return ``; + }; + + const washes = (side, cls = side.wash) => + side.washes + .map( + (r) => + ``, + ) + .join(""); + + /** A — the whole rectangle the lines occupy, exactly as it ships today. */ + const A = (side) => + ``; + + /** B — one strip per line, each cut to that line's own words, kept in place. */ + const B = (side) => { + const f = side.frame; + const strips = side.bands + .map((b) => { + const r = { x: b.x - 1.5, y: b.y - 1.5, width: b.width + 3, height: b.height + 3 }; + const style = [ + `left:${n(((r.x - f.x) / f.width) * 100)}%`, + `top:${n(((r.y - f.y) / f.height) * 100)}%`, + `width:${n((r.width / f.width) * 100)}%`, + `height:${n((r.height / f.height) * 100)}%`, + ].join(";"); + return ( + `` + ); + }) + .join(""); + return `
    ${strips}
    `; + }; + + /** C — the whole rectangle, with everything that is not this ayah faded back. */ + const C = (side) => { + const f = side.frame; + const box = (r) => `M${n(r.x)} ${n(r.y)}H${n(r.x + r.width)}V${n(r.y + r.height)}H${n(r.x)}Z`; + const holes = side.bands.map((b) => box({ x: b.x - 1, y: b.y - 1, width: b.width + 2, height: b.height + 2 })); + return ( + `` + ); + }; + + /** D — the shared opening marked too, so nothing meaningful is left plain. */ + const D = (side) => { + const same = side.shared + .map( + (r) => + ``, + ) + .join(""); + return ( + `` + ); + }; + + /** E — a bracket at the first word and the last, and nothing else added. */ + const E = (side) => { + const first = side.bands[0]; + const last = side.bands[side.bands.length - 1]; + const T = 5; // tick length, page units + // The print runs right to left, so an ayah opens at the right edge of its + // first line's band and closes at the left edge of its last. + const open = first.x + first.width; + const close = last.x; + const marks = [ + `M${n(open - T)} ${n(first.y - 1)}H${n(open)}V${n(first.y + first.height + 1)}H${n(open - T)}`, + `M${n(close + T)} ${n(last.y - 1)}H${n(close)}V${n(last.y + last.height + 1)}H${n(close + T)}`, + ] + .map((d) => ``) + .join(""); + return ( + `` + ); + }; + + const DRAW = { A, B, C, D, E }; + + /** The panel exactly as the app builds it: label, crop, label, crop. */ + return (which, sides) => + `
    ` + + sides.map((s) => `
    ${s.label}${DRAW[which](s)}
    `).join("") + + `
    `; +} + +// --------------------------------------------------------------- the chart + +const BINS = 10; + +function chartOf(shares) { + const bins = Array.from({ length: BINS }, () => 0); + for (const s of shares) bins[Math.min(BINS - 1, Math.floor(s.share * BINS))]++; + const top = Math.max(...bins); + + const CW = 640; + const CH = 208; + const LEFT = 34; + const BOT = 48; + const TOP = 12; + const RIGHT = 22; + const plotW = CW - LEFT - RIGHT; + const plotH = CH - TOP - BOT; + const barW = plotW / BINS; + + const grid = [0, 0.5, 1] + .map((g) => { + const y = TOP + plotH - g * plotH; + return ( + `` + + `${Math.round(g * top)}` + ); + }) + .join(""); + + const bars = bins + .map((count, i) => { + const h = (count / top) * plotH; + const low = i / BINS < 0.5; + return ( + `` + ); + }) + .join(""); + + const ticks = [0, 2, 4, 6, 8, 10] + .map((i) => `${i * 10}%`) + .join(""); + + const under = shares.filter((s) => s.share < 0.5).length; + const label = + `Distribution: how much of each drawn rectangle is the verse it names. ` + + `Most crops sit between 50 and 90 per cent; ${((under / shares.length) * 100).toFixed(1)} per cent fall below half.`; + + return ( + `${grid}${bars}` + + `${ticks}` + + `` + + `share of the drawn rectangle that is the verse it names` + ); +} + +// ---------------------------------------------------------------- the copy + +const OPTIONS = [ + { + id: "A", + family: "Change nothing", + name: "Leave it as it is", + lede: + "The rectangle stays as it is drawn today: every line the verse touches, edge to edge, whatever else " + + "is printed on those lines.", + for: [ + "It ships. It is already a large improvement on what it replaced, and it is the only option with no work behind it.", + "The context is not worthless — seeing what runs before and after is how a hafiz places a verse on the page in the first place.", + "The verse numbers are printed in the artwork, so the boundaries are visible to anyone who looks for them.", + ], + against: [ + "Unmarked does not mean shared, and the panel gives the reader no way to know that.", + "On the lower crop here, the whole first line is the previous verse — the labelled one does not start until line two, and nothing says so.", + "On one crop in six, more of what is drawn belongs to other verses than to the one being named.", + "It is the one option with no precedent behind it: nothing else that marks a run of text on a page marks it this way.", + ], + }, + { + id: "B", + family: "Take the neighbours away", + name: "Cut each line down to the verse's own words", + lede: + "Instead of one rectangle over all the lines, one strip per line, each cut to exactly the words of this " + + "verse — kept in its true place, so the shape of the verse on the page survives.", + for: [ + "The problem disappears by construction. Nothing is drawn that is not the verse, so unmarked can only mean shared.", + "It is what everything else already does. Selecting three wrapped lines of text in any browser gives three shapes cut to the words, never a rectangle around them — and the formats that describe scanned pages put their geometry on the line for the same reason.", + "It keeps the printer's own line breaks and the verse's real position, so it still looks like the page it came from.", + "It costs nothing to read: no new colour, no new mark, no legend.", + ], + against: [ + "It is ragged. A verse that opens with three words at the end of a line gets a stub floating above a full line, which looks broken until you understand why.", + "None of the precedents above has to look like a page of a mus'haf — a selection is transient and a scanned newspaper is not a place of reverence.", + "It removes the context argument entirely — including for the readers who were using it.", + "It is the largest change of the five: the crop stops being one picture and becomes several.", + ], + }, + { + id: "C", + family: "Push the neighbours back", + name: "Fade everything that is not this verse", + lede: + "One rectangle still, but a translucent veil over every part of it that belongs to another verse, so this " + + "one sits forward and the rest recedes.", + for: [ + "Context survives and stops competing. You can still read what runs before and after, but only if you go looking.", + "It is a small change — the same single picture, one shape drawn over it.", + "It answers the actual confusion directly: faded is visibly a different category from washed.", + ], + against: [ + "It puts a veil over printed Quran, which is a heavier thing to do than tinting a few words, and it does it to most of the crop.", + "Three visual states now — faded, plain, washed — where there were two.", + "On a small screen a faint veil may not read as a veil at all; it may just look like uneven printing.", + ], + }, + { + id: "E", + family: "Push the neighbours back", + name: "Mark where the verse begins and ends", + lede: + "The rectangle is untouched. Two small brackets are drawn instead — one at the first word, one at the " + + "last — so the verse's extent is stated without anything being hidden or dimmed.", + for: [ + "The lightest possible intervention: two marks, no colour over the ink, nothing obscured.", + "It states the boundary exactly, which is the fact the reader is missing.", + "It reads the same at any size, unlike a faint veil.", + ], + against: [ + "It tells you where the verse ends without making the neighbours any quieter — a reader skimming still sees one block of ink.", + "Two more marks in a panel that already carries a coloured wash, on a surface where restraint is the whole aesthetic.", + "A bracket is a convention, and conventions have to be learnt.", + ], + }, + { + id: "D", + family: "Say what the marks mean", + name: "Mark the words the two share, as well as the ones they do not", + lede: + "Nothing is removed or dimmed. A third, neutral tint goes over the shared opening — so the two states " + + "that carry meaning are both stated, and anything left plain is simply not part of the comparison.", + for: [ + "It fixes the actual error rather than its cause: the reader's wrong inference was that plain means shared. Here, shared is marked, so plain means nothing at all.", + "The neighbours keep whatever context value they have, at no cost.", + "It is the smallest change to what is drawn — one more tint, using machinery already there.", + ], + against: [ + "Almost the whole verse is now under colour, which is close to the opposite of the panel's intent — the wash exists so the eye lands on a few words.", + "Three tints on one crop needs a legend, and there is no room for a legend under a row in a list.", + "It is the only option that makes the crop busier rather than quieter.", + ], + }, +]; + +// ---------------------------------------------------------------- assembly + +function render({ artifact: ARTIFACT, out }) { + const panel = makeRenderer(ARTIFACT); + const sides = SIDES.map(measureSide); + const { shares, pairs } = sweep(); + + const N = shares.length; + const pct1 = (x) => (x * 100).toFixed(1); + const pct0 = (x) => Math.round(x * 100); + const mean = shares.reduce((a, s) => a + s.share, 0) / N; + const median = shares[Math.floor(N / 2)].share; + const oneLine = shares.filter((s) => s.lines === 1).length; + const underHalf = shares.filter((s) => s.share < 0.5).length; + const underThird = shares.filter((s) => s.share < 1 / 3).length; + const worst = shares[0]; + const twoLine = shares.filter((s) => s.lines === 2); + const twoLineMean = twoLine.reduce((a, s) => a + s.share, 0) / twoLine.length; + const num = (v) => v.toLocaleString("en"); + + const optionCard = (o) => ` +
    +
    + ${o.id} +
    +

    ${o.family}

    +

    ${esc(o.name)}

    +
    +
    +
    +
    +

    ${esc(o.lede)}

    +
    +
    +

    What it gets you

    +
      ${o.for.map((x) => `
    • ${esc(x)}
      • `).join("")}
    +
    +
    +

    What it costs

    +
      ${o.against.map((x) => `
    • ${esc(x)}
      • `).join("")}
    +
    +
    +
    +
    + ${panel(o.id, sides)} +
    Drawn on 2:48 and 2:123, at the width the panel really is on a phone. ${ + o.id === "A" + ? "This is a picture of the live app." + : "Everything here is the printed page — only what is drawn over it changes." + }
    +
    +
    +
    `; + + const glance = ` +
    + ${OPTIONS.map( + (o) => + `
    ${o.id} ${esc(o.name)}
    ` + + `${panel(o.id, [sides[1]])}
    `, + ).join("")} +
    `; + + const defs = ARTIFACT + ? `` + : ""; + + const html = `What should the panel show around the ayah? + + +${defs} + +
    +
    +

    Hifth · open question

    +

    What should the panel show around the ayah?

    +

    When you open a pair of look-alike verses, the app now cuts both of them straight out of the printed page instead of retyping them. That fixed a real problem and introduced a smaller one: a verse almost never begins and ends at the edge of a line, so the rectangle cut around it brings its neighbours along for the ride.

    +

    Open Nobody has decided. Option A is what ships today.

    +
    + +
    +

    The words on this pageWhat do these words mean?

    +
    +
    ayah
    A verse.
    +
    mus'haf
    The printed Quran the app draws. Every page is the printer's own artwork, vendored as-is.
    +
    look-alikes
    Pairs of verses that read almost the same — the thing a hafiz has to keep apart. Opening one is called a hop.
    +
    the crop
    The rectangle the app cuts out of a printed page to show one verse.
    +
    the wash
    The translucent colour laid over the words the two verses do not share.
    +
    +
    + +
    +

    The questionWhat is being decided?

    +

    What the app draws around a verse when it shows it beside its look-alike — not what it marks. Which words get the wash is read off the pair itself and is correct; that part is not in question.

    +

    What is in question is everything else inside the frame. Because a verse starts and stops mid-line, the rectangle that holds it also holds whatever else is printed on those lines. The reader is given no way to tell the difference between this is the other half of the verse, and the two agree here and this is a different verse that happens to sit on the same line.

    +
    + +
    +

    TimingWhy is this being asked now?

    +

    Until this week the panel showed the two verses as retyped text. That was wrong on its own terms — the transcription used a plainer spelling than the printed page, so a reader comparing the panel against the page underneath saw one set of letters in each. It was also the larger of the only two places in this project holding running scripture, which is the reason it went.

    +

    The replacement cuts the verses out of the page itself, so there is exactly one set of letters now, and it covers ${num(pairs)} pairs where the retyped table covered twelve. The neighbours arrived with the ink. This question exists because the fix worked.

    +
    + +
    +

    Doing nothingWhat happens if nobody decides?

    +

    Option A stands, because option A is live. Nothing breaks, and the panel remains a clear improvement on what it replaced.

    +

    The risk is quiet rather than loud: a reader takes an unmarked neighbour for shared wording and memorises a difference that is not there — in an app whose entire purpose is to stop exactly that.

    +
    + +
    +
    +

    Measured, not estimatedWhat does the app do today, and what does that cost?

    +

    Every look-alike pair that the panel can draw was measured, both sides — ${num(N)} crops across ${num(pairs)} pairs. For each one: how much of the rectangle actually drawn belongs to the verse it names.

    +
    +
    +
    ${pct1(mean)}%of the average crop is the verse it names
    +
    ${pct1(median)}%is the middle crop — half are worse
    +
    ${pct1(underHalf / N)}%are more neighbour than verse (${num(underHalf)} crops)
    +
    ${pct1(worst.share)}%is the worst of them — ${worst.key} on page ${worst.page}
    +
    ${pct1(oneLine / N)}%sit on one line and have no neighbours at all
    +
    +
    ${chartOf(shares)}
    +
    +

    Two facts in that distribution matter more than the average. The first is that ${num(oneLine)} crops — the verses that fit on a single line — are perfect already: the strip drawn for them contains nothing but the verse. Whatever is decided here should not disturb them.

    +

    The second is that the worst case is not the longest verse but the shortest multi-line one. Verses spanning two lines average just ${pct1(twoLineMean)}%, because both of their lines are partial and there is no full line in the middle to dilute the edges. ${num(underThird)} crops are under a third. At the very bottom sits ${worst.key} on page ${worst.page}, where nine tenths of what the reader is shown is other verses.

    +

    The pair drawn on every specimen below is the app's own signature example, and it straddles the problem neatly: 2:48 is ${pct0(sides[0].share)}% its own verse, and 2:123 is ${pct0(sides[1].share)}%.

    +

    Look at what that means on the lower crop, in option A below. It is labelled 2:123, and its entire first line is 2:122 — the verse before it, ending in a printed verse number, none of it marked in any way. The verse the label names does not begin until the second line. A reader who takes plain ink for shared wording has just been told that a whole line the two verses do not have in common is a line they agree on.

    +

    A note on the arithmetic: a few crops score fractionally over 100% because two adjacent lines' strips overlap slightly where letters descend. They are counted as 100% rather than more.

    +
    +
    + +
    +

    Prior artWhat do other people do about this?

    +

    The whole thing, nobody. I did not find anyone cutting a verse out of a printed page and setting it beside its look-alike. The largest public library of Quran data publishes the two halves as separate downloads and has not joined them: printed-page layouts in one place, and look-alike phrase data in another — 5,277 look-alike phrase entries and 4,001 similar-verse links, alongside twenty approved page layouts. The most-used Quran reader shows no look-alike feature on a verse's own page at all.

    +

    The hard part of it, everybody — and they all do the same thing. Marking a run of text that wraps across several lines of a page is a solved problem, and in four independent traditions it is solved per line, never by drawing a box around the whole run.

    +
      +
    • Every text selection you have ever made. The specification that governs how a highlight is painted says it is not one shape over a range: there is a single overlay for the document, and Each box owns the piece of the overlay corresponding to the text directly inside it. Drag across three wrapped lines in any browser and you get three shapes cut to the words. CSS Pseudo-Elements 4, §3.4 Area of a Highlight
      • +
    • The web platform gives the two shapes different names. You can ask a range for its geometry as a list — one for each box fragment — or as a single rectangle, the smallest rectangle that includes all of the rectangles in list. Two methods, because they are two different answers to two different questions. The app is currently calling the second one. CSSOM View Module Level 1
      • +
    • The standard for annotating part of an image. Its ordinary rectangle selector can say only origin, width and height, and the spec is blunt about the limit — even a simple circular region of an image, or a diagonal line across it, are not possible — so it defines a second, shape-based selector and points you at it whenever the region is not a rectangle. W3C Web Annotation Data Model
      • +
    • The standard for serving a crop of a scanned page can cut only rectangles — the whole image, a square, or a box given in pixels or per cent. There is no multi-part or non-rectangular request in it. That is worth naming, because it explains why our crop is a rectangle: a rectangle is what the tooling hands you, not what the content is. Viewers that need a real shape draw it over the image instead of asking for it. IIIF Image API 3.0, §4.1 Region
      • +
    • The formats that describe scanned pages put the geometry on the line. Both of the widely used ones carry a box per text line, while logical units like a paragraph are containers that group lines rather than shapes of their own. One goes further and defines an element for exactly our case, documented as describing the bounding shape of a block, if it is not rectangular, holding a polygon. A verse crossing three lines is that block. hOCR 1.2 · ALTO 4.4 schema
      • +
    +

    What this changes. Option B stops being the boldest of the five and becomes the conventional one. Its remaining objection is not correctness but appearance — none of the precedents above has to look like a page of a mus'haf — and that is a question about taste and reverence, which is exactly the sort a stranger to this code is better placed to answer than its author.

    +
    +

    What I could not confirm. I believe a multi-line highlight in a PDF is stored as several quadrilaterals rather than one rectangle — the same convention a fifth time — but both sources I tried for that specification returned nothing, and this session's search budget was spent, so it is not counted above.

    +

    I also could not open the layout or look-alike data files themselves to see whether they carry word positions. The finding about them rests on the library's own descriptions.

    +
    +
    + +
    +

    ConstraintsWhat have we already decided that limits the answer?

    +
      +
    • The printed page is never edited. Whatever the panel does, it does by drawing over the artwork. Moving, re-flowing, or re-setting the letters is not available — that was the whole point of cropping the page instead of retyping it.
      • +
    • The page is paper and stays paper. It is never re-themed, because the printer's ink is a fixed dark that would vanish on a dark ground. Every specimen below is on paper even if you are reading this at night, and that is not an oversight.
      • +
    • The wash must not hide the words it points at. It is translucent for that reason, and a solid fill is not on the table for anything drawn over the ink.
      • +
    • When the comparison cannot be drawn, nothing is drawn and the row keeps its plain note. Any option has to keep that fallback.
      • +
    • Whether a rectangle should line up with a printed line or with a whole page is already an open question here, asked about the marks the app draws over recitation rules. This is the same tension in a second place, and the two should not be answered in opposite directions without somebody saying why.
      • +
    +
    + +
    +
    +

    Side by sideHow do the five compare at a glance?

    +

    The hard side of the pair — 2:123, where a little under half the crop is other verses — drawn five ways at the panel's real width. Scroll sideways. Each one is the actual printed page with the actual measured geometry over it.

    +
    + ${glance} +
    + +
    +
    +

    The optionsWhat could be done instead?

    +

    Five, grouped by what they actually do about the neighbours: take them away, push them back, or leave them and say plainly what the marks mean. They are not exclusive — B and D compose, and so do C and E — but each is drawn on its own so it can be judged on its own.

    +
    + ${OPTIONS.map(optionCard).join("")} +
    + +
    +

    RejectedWhat else was considered, and why is it not here?

    +
      +
    • Show the whole page and point at the verse. The panel is a strip underneath a row in a list. A whole page at that size is unreadable, and the reader is already looking at one.
      • +
    • Re-flow the verse's words onto a line of their own. It would move the printer's letters, and showing the reader letters arranged differently from the page is the exact defect that cropping the page was brought in to fix.
      • +
    • Blur the neighbours' ink. Same objection as re-flowing, and a blur laid over printed Quran is a harder thing to defend than a translucent tint of the same strength.
      • +
    • Draw only the words that differ. A difference with no context is not memorable. The shared opening is the reason the pair is confusable in the first place — remove it and the panel stops answering its own question.
      • +
    +
    + +
    +

    SensitivityWhat would change the answer?

    +
      +
    • A hafiz reading the panel and saying whether the neighbours help or hurt. The case against A rests on a guess about someone else's habit — that context is a distraction rather than an aid. That guess has not been tested on a single person who memorises.
      • +
    • Whether the raggedness actually reads as broken. After the prior art above, this is the only real objection left to B, and it is an appearance claim nobody has tested. Two people and fifteen minutes would settle it.
      • +
    • The panel moving out of a list row. B's raggedness is only a problem in a cramped strip. On a full screen, most of the objection to it evaporates.
      • +
    • Any change to how the page is themed. C is a veil in the page's own paper colour; if the artwork's ground ever moves, C moves with it.
      • +
    +
    + +
    +

    ScopeWhat is this not settling?

    +
      +
    • Which words get washed. Read off the pair itself, verified, and correct.
      • +
    • The two colours. Terracotta for the verse you are on, verdigris for the one you are looking at — settled elsewhere, and not reopened here.
      • +
    • Whether the panel should exist at all. It should.
      • +
    • Anything about the printed page. It is vendored unmodified apart from three declared transforms, and nothing here touches that.
      • +
    +
    +
    +`; + + writeFileSync(out, html); + return { html, N, pairs, mean, underHalf, worst, sides }; +} + +let last; +for (const copy of COPIES) { + const r = render(copy); + console.log(`${copy.out.replace(ROOT, "")} ${(r.html.length / 1024).toFixed(0)} KB`); + last = r; +} +console.log( + ` measured ${last.N} crops over ${last.pairs} pairs — mean ${(last.mean * 100).toFixed(1)}%, ` + + `${last.underHalf} under half, worst ${(last.worst.share * 100).toFixed(1)}% (${last.worst.key} p${last.worst.page})`, +); From c7525ee7b014eadf2b38d718ae4852340d9e2276 Mon Sep 17 00:00:00 2001 From: Omar Eid Date: Sun, 16 Aug 2026 20:04:15 -0500 Subject: [PATCH 37/95] The registers catch up, and a fourth question goes on the board MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prose the two code commits made false or newly provable, and the decision they opened. LICENSES.md said the app code is not a derivative of the corpus data because it reads the shards at runtime. The premise is true — every asset the web app reads is fetched by URL, nothing imports the pipeline's data directory — but the conclusion did not follow, because the content that was there arrived by typing, not across that boundary. It now claims what it can prove. It also said the page artwork is reproduced unmodified; svgo runs at one decimal place, so it now says what is applied. SOURCES.md said the artwork pipeline applies two declared transforms against code that says three: 23 polygon repairs across 19 pages and 2 id repairs, the description of which omitted the polygon repairs entirely. The code was right and the register was stale. what-we-distribute.md predicted the scripture fix and now records what was done, including the second site, which it did not know about. what-we-depend-on.md is new and carries the dependency and licensing survey it had no home for. The new decision is comparison-crop: when the app shows an ayah cut out of the printed page, what should it do about the neighbouring verses that come with it. Five options, each drawn on the real artwork at the panel's real width, and the cost of choosing none of them measured across all 5,088 crops. The prior art moved the recommendation, which is the point of looking. The composite — cropping an ayah out of a page and setting it beside its look-alike — nobody appears to have attempted; the largest public library of Quran data publishes mushaf layouts and mutashabihat as separate downloads and has not joined them. But the hard part of it, marking a run of text that wraps across lines, is solved, and four independent traditions solve it the same way: per line, never by a box around the whole run. CSS says a highlight is one overlay per box fragment, which is why every text selection you have ever made looks the way it does. CSSOM View gives the two shapes two different methods and we picked the union one. The Web Annotation model is explicit that its rectangle selector cannot describe a non-rectangular region and points at an SVG selector instead. hOCR and ALTO both put geometry on the text line, and ALTO has an element documented for exactly a block whose bounding shape is not a rectangle. The IIIF image API can only cut rectangles, which is worth naming because it explains why our crop is one: a rectangle is what the tooling hands you, not what the content is. So option B — cut each line down to the ayah's own words — stops being the boldest of the five and becomes the conventional one, and its remaining objection is appearance rather than correctness. That is a question about taste and reverence, which a stranger to this code is better placed to answer than its author. Recorded as open, with what would change the answer stated: a hafiz reading the panel, and fifteen minutes finding out whether B's raggedness reads as broken to anyone but me. One search caveat is on the page rather than hidden here: WebSearch was exhausted at 200/200, so all of the above is primary-source fetching against known URLs. The PDF highlight convention, which I believe is the same one a fifth time, could not be confirmed with a link and is not counted. The registers: comparison-crop is in decisions.json with related named in both directions on mark-placement, word-selection and loop-4a; issues.json moves the four rows this work closes and gains one for the second scripture site, which no row covered; map.json gains rows for the new gate and the new generator, and records what the notices gate now enumerates; the validation ledger and use cases follow the panel's rewrite. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt --- .claude/plans/shimmering-wandering-lobster.md | 446 ++++++------- .claude/plans/sitting-instrument.md | 286 +++++++++ .claude/skills/mushaf-reference/SKILL.md | 7 +- LICENSES.md | 51 +- SOURCES.md | 32 +- docs/decisions.json | 48 +- docs/decisions/README.md | 12 +- docs/decisions/comparison-crop.md | 148 +++++ docs/decisions/loop-3.md | 7 + docs/design/comparison-crop.html | 484 +++++++++++++++ docs/design/etl-pipeline.md | 12 +- docs/design/what-we-depend-on.md | 586 ++++++++++++++++++ docs/design/what-we-distribute.md | 392 ++++++++++++ docs/design/word-indexing.md | 5 +- docs/issues.json | 163 ++++- docs/issues.md | 25 +- docs/map.json | 13 +- docs/tasks.md | 23 +- docs/use-cases.json | 14 +- docs/use-cases.md | 28 +- docs/validation/ledger.json | 2 +- 21 files changed, 2474 insertions(+), 310 deletions(-) create mode 100644 .claude/plans/sitting-instrument.md create mode 100644 docs/decisions/comparison-crop.md create mode 100644 docs/design/comparison-crop.html create mode 100644 docs/design/what-we-depend-on.md create mode 100644 docs/design/what-we-distribute.md diff --git a/.claude/plans/shimmering-wandering-lobster.md b/.claude/plans/shimmering-wandering-lobster.md index e87121e..d10db81 100644 --- a/.claude/plans/shimmering-wandering-lobster.md +++ b/.claude/plans/shimmering-wandering-lobster.md @@ -1,286 +1,218 @@ -# The instrument that asks the question has to be trustworthy first +# The repo's most-repeated claim is false, and nothing was checking it + +> **This file previously held the sitting-instrument plan**, which is unfinished and is *not* +> superseded. It is recoverable in full at `git show 83041f3:.claude/plans/shimmering-wandering-lobster.md` +> and should be restored to its own file before that work resumes. ## Context -Sixteen sittings are about to be sat. Between them they cover all **1,877** marks the machine -could not place from ink — every one seen, not sampled — at roughly an hour each. That is -sixteen hours of the scarcest thing this project has. - -Before spending it, the page that does the asking was audited against the frontend-design -brief, and three of its findings do not make the sitting *slower*. They make its answers -**mean something other than what they say**: - -1. **In dark mode the rectangles are drawn in near-invisible colours.** The crop's paper is - deliberately never re-themed — a mus'haf page stays on white — but the two rectangle stroke - colours *are* re-themed, and then drawn on that white. Measured: our box 5.05:1 → **2.49:1**, - the reader's box 4.89:1 → **1.70:1**, at a constant 1.64px stroke. A reader who cannot see the - box affirms it. This finding runs in exactly the direction that looks like success. -2. **A destructive control lands where the reader's thumb already is.** The answer list sits - directly above Back/Next, each answer adds a ~28px row, and the "take it back" button in that - row is right-aligned — into the corner Next occupied a moment earlier. Two taps of Next can - retract the answer just given. -3. **Every pointer frame re-parses the page.** The drag handler rebuilds the stage from - `c.svg` — 2.0 / 9.2 / **23.2 KB** of path data (min/median/max) — on every `pointermove`. A - correction that stutters is a correction the reader gives up on and affirms instead. - -Below those, four findings cost time and accuracy rather than truth: the affirm button looks -the same pressed as unpressed; it sits at y≈842 on a 393px phone, below the fold, behind a -~275px lede that never goes away; 91 of 115 cards carry the mark's containing word and throw it -away, while the sentence naming the mark omits the mark's own name and is the dimmest text on -the card; and four **destructive or load-bearing** controls are under 44px. - -**Decided with the reader, 2026-08-14:** the buttons get pinned to the bottom of the screen and -the picture keeps its own shape (nothing ever shrinks); the fixes land **before** part 1 is sat, -which costs nothing because the sixteen parts were re-dealt and **0 of 1,877** have been -answered on the new deal; scope is the three data-corrupting findings plus the four -time-and-accuracy ones. - -**Outcome:** all 1,877 marks are answered on one instrument, and the transcript means what it -says. - -### What the audit explicitly warns against, and why - -- **Do not put the fault buttons behind a disclosure.** Adding a tap to reporting a fault while - affirming stays free biases the exact ratio the sitting exists to measure. Change weight, not - cost. -- **Do not highlight the ligature's own ink in the crop.** Where the ink is, is the unknown - being measured. Also ruled out and not to be revisited here: centring the crop on the mark's - own ink; SVG resize handles on the rectangle. +Twenty-two times across twenty files, in three shipped `NOTICE.txt` files and in the sentence +`docs/decisions/loop-5.md:119` leans on, this repo asserts some version of *"there is no Quran +text here."* Measured against the tree, the unscoped form of that claim is false. ---- +Two sites hold running scripture. Measurement is the **longest run of consecutive fully-vowelled +words** — one vowelled word is a specimen, a phrase is a passage somebody can recover: -## Also in scope: the scorer is reporting the wrong number +| file | longest phrase | vowelled words | ships to the site? | +| --- | --- | --- | --- | +| `packages/core/src/verse-text.ts` | **11 words** | 160 | **yes** — all 48 token strings are byte-present in `apps/web/dist/assets/index-*.js` | +| `packages/etl/scripts/lib/tajweed-fold.test.mjs` | **5 words** | 29 | no — pipeline test, repository only | -`score-mark-report.mjs` (~377-387) medians the `by` field across every placement **event**. Two -things are wrong with that, and they were verified against the banked transcript, not argued: +Every other Arabic-bearing file in the tree scores **1 or 2**: our own interface Arabic +(`ar.gen.ts`, `format.ts`), domain terms (*madd*, *tamm*), and single-word pipeline specimens +(tajweed edge cases, join fixtures, encoding probes). **Nothing sits at 3 or 4.** That gap is +what makes a gate possible rather than a matter of taste. -**It medians increments, and increments cancel.** `by` is always an increment — both the nudge -pad (`flush`) and the drag (`pointerup`) send the burst, and `to` is the running total. Of 179 -consecutive pairs on the same mark, **133** chain exactly; the other 46 are marks where the -reader pressed *put it back where it was*, and a banked transcript has retractions already -applied, so the superseded placements are gone and the chain legitimately restarts. Nothing is -wrong with the field. What is wrong is medianing it: opposite-signed ±0.1 increments cancel, and -one mark with 44 events outvotes 25 marks with one each. The scorer prints **0.000 / 0.000**. +Nothing caught either site. `gate:notext` sounds like it would and does not — it forbids `` +elements in page SVGs because of a Safari paint bug, which is a rendering requirement wearing a +name that reads like a scripture check. -**And `to` is not the reader's hand.** `to = at + total − box`, so it is measured from the -**shipped** box and already contains the correction the pipeline applied. Three different -numbers, all real, currently collapsed into one wrong one: +**Why it matters more than tidiness.** The Crossway/ESV takedown campaign of July–August 2025 +ran eleven notices, roughly half of them **path-scoped into code repositories** — a data +directory in one, a **test-fixtures directory** in another. Blast radius tracks the directory +holding the text, not the project's purpose. Both of our sites are that shape. Unlike everything +else in `docs/design/what-we-distribute.md`, this needs no licensing opinion from anybody. -| question | across | down | -|---|---|---| -| what the scorer prints today | 0.000 | 0.000 | -| where the reader put it, against what ships | −3.569 | −3.134 | -| **how far the reader's own hand moved it** | **−2.468** | **−2.010** | +**And the shipped panel is also wrong on its own terms.** `verse-text.ts` is typed in a plainer +spelling than the mus'haf on screen. A reader comparing 2:48 and 2:123 sees one spelling in the +panel and different letters on the page underneath it. -The middle row is what corroborates the ink measurement: the corpus sets its text lower and -further across than the ornament fit predicts, `dty` negative on **599 of 600** pages. Two -instruments, same direction, same rough size. That is worth printing correctly. +**Outcome:** no running scripture in the tree, a gate that refuses the next one, the licensing +map covering everything that actually ships, and the panel showing the reader the same ink the +page does. --- ## The work -Fixes are ordered so each lands in one place rather than in a string another fix is about to -rewrite. **Do them in this order.** - -### ① Parse the paths once per card — `build-mark-report.mjs`, `paint()` ~1084-1122 - -Split `paint()` into `mount(c)` and `paint()`. - -- Module-level `let drawnFor = null, hitEl = null, boxEl = null, mineEl = null;` -- `mount(c)` writes `stage.innerHTML` **once per card**: `c.svg` plus the `.hit`, `.grab` and a - hidden reader's rect; caches the three nodes; sets `drawnFor = c.id`. -- `paint()` calls `mount(c)` only when `drawnFor !== c.id`, then `setAttribute`s x/y/width/height - on the cached nodes and toggles the reader's rect's `visibility`. The `viewBox` write stays. -- The `.hit` slack `t = vb[2]/30` stays per-frame — it is unit-based and framing-dependent. -- **Drop `sw = vb[2] / 220`** and use `vector-effect="non-scaling-stroke"` with `stroke-width` in - CSS pixels. That constant is 1.64px *only because* the stage is `width: 100%`; leaving it in - makes stroke width a function of each card's aspect the moment anything sizes the stage - differently. One fewer per-frame attribute write, too. - -This is what `crop()`'s two-viewBox design was always for — the framing toggle becomes one -attribute write instead of a 23 KB reparse — and it has never actually delivered it. - -**Correction to the audit:** pointer capture is on the **stage** (`stage.setPointerCapture`, -~1158), gated by `onRect(p)` (~1141) — not on a rect node. `docs/map.json:752` describes the -gate. So rebuilding mid-gesture never risked capture; do this fix for the parse cost alone. - -### ② Keep the rectangles legible on white paper, in both themes — CSS ~514-525 - -The rule to establish and then never break: **nothing drawn on the paper is themed.** - -- Add to bare `:root` **only**: `--ours-line`, `--ours-wash`, `--yours-line`, `--yours-wash` - (suggested `#c2410c` / `#c2410c1f` and `#15803d` / `#15803d24`). -- Leave `--ours` / `--yours` / `--ours-fill` / `--yours-fill` exactly as they are and keep them - re-themed — they are **chrome** (affirm button, pressed state, focus ring, the answer list, the - bank panel) and sit on `--field`, so they must stay themed. -- `mount()` draws with the four new tokens. Add one line of comment saying they join `--paper` - and `--ink` in the never-re-themed set, and *why*. -- Carry the distinction independently of hue at the same time: - `stroke-dasharray="0.6 0.4"` on the reader's rect. Two rectangles that differ in dash as well - as colour survive deuteranopia, and survive anyone re-theming the palette later. - -Light theme comes out byte-identical. Dark theme returns to 5.05:1 and 4.89:1. - -### ③ Pin the buttons to the bottom — markup ~679-697, CSS - -Two independent halves; **③b must land whichever layout is chosen.** - -**③a — the dock.** Move the affirm row and the Back/Next row into one `
    ` with -`position: sticky; bottom: 0; background: var(--field); border-top: 1px solid var(--edge); -padding: .5rem 0 calc(.5rem + env(safe-area-inset-bottom));` and bump the main column's bottom -padding. **Sticky, not fixed** — it keeps document flow and does not fight the iOS URL bar. -Requires `viewport-fit=cover` on the viewport meta (~511) or `env()` resolves to zero; that is -also the whole of the safe-area finding. - -The picture keeps its natural per-card shape (295–410px judging, 266–529px identifying, measured -across all 1,877 marks). Nothing shrinks, `ptIn()` is untouched, and on the tallest cards the -reader scrolls the picture behind a dock that never moves. **Explicitly not doing:** normalising -the two viewBoxes to a constant aspect inside `crop()` — it either shrinks the mark or makes -every card as tall as the tallest, and the ring clip would have to be re-ordered after the -normalisation or the new margin comes back empty of the ink that belongs in it. - -**③b — get the destructive control out of the thumb corner.** In the answer list row, move "take -it back" to the **start** of the row (button, then the kind, then the detail) so nothing tappable -sits bottom-right, and give the list at least 44px of bottom margin. **This must ship before ⑦ -enlarges that button to 44px**, or the mis-tap target doubles. - -### ④ Make the affirm button look pressed — CSS ~596 - -`button.affirm[aria-pressed="true"]` gets a solid fill: background `var(--ours)`, text -`var(--field)`, matching border. Inverts legibly against both themes and reads as *on* without a -second glance. Add a check glyph via `::before` on the pressed state only. **Do not change the -label text** — that would move the fold ⑤ just fixed. - -*(The audit's specificity claim was off by one rule: `button.affirm` and the generic pressed rule -are both (0,1,1) and settled by source order; the actual out-specifier is -`button.affirm[aria-pressed="true"]` at (0,2,1). Conclusion unchanged.)* - -### ⑤ Get the lede off the fold without touching the fault buttons — markup ~640-646 - -Split the lede into the full text and a one-line brief, with a quiet toggle between them. The -brief shows once the reader has demonstrably read it: `seen > 0`, persisted through a -`keepRead`/`keptRead` pair written **exactly** like the existing `keepSeen`/`keptSeen` -(~733-736) — try/catch on every call, because a `file:` origin throws. - -Roughly 230px comes off the fold. **No answer costs a tap more than it does today**, so the -affirm/fault ratio the sitting measures is untouched. That constraint is the whole design. - -### ⑥ Say which mark, properly — `identify()` ~1240-1264 - -- The card already carries the containing word and throws it away for the 91 cards that have - both it and a single letter. Keep the big letters as they are, and append the word after them - at ~1rem, dim, `lang="ar" dir="rtl"`. -- Put the mark's **name** into all three branches. It reaches only the `of === 1` branch today. -- The sentence naming the mark is the question, not a footnote: `--dim` → `--text`, - .82rem → .9rem. - -### ⑦ 44px on the four — CSS - -All four are omissions from a convention this file already keeps elsewhere (the note pad and the -chips are already 44/48): - -| control | today | why it matters | -|---|---|---| -| take it back | ~24px | destructive — **land ③b first** | -| the two view toggles | ~31px | pressed constantly, both framings | -| hand over what I have said | ~29px | the reader's safety net | -| put it back where it was | 34px | destructive; also give it `margin-left: auto` so it is not shoulder-to-shoulder with "Finer steps" | - -Land **after** ⑤ and ③, since each of those moves the fold. - -### ⑧ The scorer — `score-mark-report.mjs` ~377-387 - -Do 1-4 **before** 5. Steps 1-4 work on the two transcripts already on disk, which cannot be -re-recorded; once the headline reads `to`, the rename stops being load-bearing. - -1. Collapse to **one row per mark** — walk in order, keep the last placement carrying a `to`. -2. Print the reader's **hand**: `to` minus the displacement already shipped - (`drawnAt(r) − r.box`, both already available). Median/p90/worst of the magnitude, plus signed - per-axis medians. **−2.468 / −2.010** on the banked sitting. -3. Print, under a **separately worded** sentence, where they landed **against what ships** — - median final `to`, **−3.569 / −3.134** — and say in the prose that this one includes the - correction the pipeline already applied, so no reader ever differences the two. -4. Say *n marks*, and say how many events those marks took. 26 marks / 205 events is itself a - finding about the nudge pad. -5. ~~Rename the drag's field.~~ **Dropped.** Both paths genuinely send an increment and `to` is - the running total, so one name is right, and a rename would only make the two transcripts - already on disk unreadable. (The drag path does not need `flush()`'s save-and-restore around - `dropVague` either: it captures the total at ~1190, before, and reassigns at ~1199, after.) +Ordered so the gate lands last and passes on arrival. + +### ① Draw the diff panel from the page artwork — deletes `verse-text.ts` + +**The classification is already shipped, for every edge.** This was the discovery that made the +option cheap. Adjacency edges already carry the target's page and the matching word range on +both sides: + +``` +2:48 → 2:123 page 19 span {from:[1,13]} toSpan {from:[1,13]} +``` + +and the word shards already carry per-ayah boxes in the same index space: + +``` +2:48 page 7 from 1 23 boxes → indices 1..23 +2:123 page 19 from 1 22 boxes → indices 1..22 +``` + +Words 1–13 are the shared opening; 14–23 against 14–22 is where they diverge. That is exactly +what the hand-typed `cls 0 / 1 / 2` encoded — **already present, for all 44,431 links rather +than twelve ayah pairs.** + +So the panel becomes: read the edge's `span`/`toSpan`, load each side's page SVG and word shard, +crop each word's box out of the page, and wash the words that fall outside the shared range. + +- **Reuse, do not build:** `loadPageSvg` and `loadWordShard` (`apps/web/src/assets.ts:90,197`) + and `WordIndex` (`packages/core/src/words.ts:120`), which already keys bare `"2:48"` to rects + in reading order and holds the print's start index. +- **Rewrite** `apps/web/src/components/DiffView.tsx` — `TokenRow` becomes a row of cropped word + SVGs. Keep `lang="ar" dir="rtl"` and the existing `styles.dA`/`dB` washes. +- **Replace** `diffPair` in `packages/core/src/verse-text.ts` with a pure function over an edge — + no text, no table. Rename the module to say what it now is; drop `VERSE_TEXT`, `verseTokens`, + `DiffToken`, `DiffClass` from `packages/core/src/index.ts:105-111`. +- **First step before any of it:** confirm on 2:48/2:123 that `span.from` and the shard's `from` + share an index space across a page boundary. The whole design rests on that one assumption and + it is ten minutes to check. +- **The target's page must be fetched** when the panel expands — it is inside a popover opened on + demand, so lazy on expand. Both call sites: `HighlightMenu.tsx:213`, `HopPopover.tsx:166`. +- **Falls back exactly as today:** no edge, no shard, or no page → render nothing, and the hop row + keeps its plain note. That path already exists for the 6,224 pairs the table never covered. +- **While in there:** edge `note` fields carry bare Arabic (`"شفاعة ↔ عدل order swapped"`) and + ship in the adjacency assets. Two unvowelled words is a specimen, not a phrase — but check + where the note text originates before leaving it. + +### ② The second site — `packages/etl/scripts/lib/tajweed-fold.test.mjs` + +Five running words. Replace the phrase fixtures with per-word cases drawn from the same +edge-case set the rest of the file already uses, or with escaped code points where a word is +genuinely the thing under test. **Do not weaken what the test asserts** — it is checking fold +behaviour on real orthography and that coverage has to survive. + +### ③ A gate that refuses the next one — `scripts/gate-scripture.mjs` + +Model it on `scripts/gate-text-sources.mjs` exactly: same +`git ls-files --cached --others --exclude-standard` enumeration, same `SOURCE_RE`/`EXCLUDE_RE` +split with the reasoning stated in the file, same untracked-files rationale (its comments at +lines 42–56 explain why tracked-only is a real blind spot; that argument applies here unchanged). + +**The rule:** fail on any run of **three or more consecutive fully-vowelled Arabic words**, where +a word counts as vowelled at half or more of its letters carrying a mark. Measured margin — 11 +and 5 on the two offenders, 1 or 2 everywhere else, nothing between. + +**The allowlist:** a named list of single-specimen files with a one-line reason each, so the +~90 lone vowelled words in `probe-tajweed-words.mjs`, `tajweed-fold.mjs`, `mark-join*.mjs`, +`probe-diacritics.mjs` and `segmentation.mjs` read as reviewed rather than unnoticed. **The +phrase rule is not allowlistable** — a run of three fails regardless of what the list says. + +Wire into `package.json` beside the other gates, into `make ci`, and into `gate:gates`. + +State in the header what `gate:notext` is actually about, so the next reader does not assume that +one covers this. + +### ④ Make the licensing map see the whole assets folder — `scripts/gate-notices.mjs` + +Three defects, all in one file: + +- **Blind by construction.** `BUCKETS` declares three paths (`roots`, `skins`, `adj`) and the + cross-check at line 187 compares that declaration against `LICENSES.md` in both directions — + but it never enumerates `apps/web/public/assets/` itself. Add that enumeration: every entry + must be either a declared bucket or a named entry in an "ours, no inherited terms" list. +- **Two shipped trees are in no bucket.** `words/hafs-kfqc/` — 604 files, 2.8 MB — and + `manifest.json` at 24,471 bytes. The string `words` appears in neither `SOURCES.md` nor + `LICENSES.md` nor this gate. (`pages` is deliberate: it is covered by `LICENSES.md`'s "What we + do not license" — keep it out of the buckets and put it in the named list with that reason.) +- **A false declaration.** Line 103 asserts `"ayah-pages.json": "ours"` for the roots bucket. + Confirmed false. Line 129 already declares the honest form for the same file in the `adj` + bucket — mirror that. + +### ⑤ Say where the transliteration table came from — `packages/etl/scripts/build-roots.mjs:86` + +The comment reads *"Buckwalter → Arabic (corpus.quran.com/java/buckwalter.jsp, verbatim)"* — a +table copied from the GPL corpus's own page. Buckwalter is a published transliteration **scheme** +and a forty-odd entry character mapping is thin ground for anyone to claim, but the comment as +written says we copied it from them. Cite the scheme's original publication rather than the +corpus's rendering of it, and add a row to `SOURCES.md` if the citation does not resolve to +something already listed there. + +### ⑥ The prose that is wrong + +Four sites. **The distinction that matters:** *scoped* claims stay — "our shards carry no Quran +text", "no Quran text crosses the wire", "this file contains no Quran text" are all still true +and all still worth saying. Only the **unscoped repo-wide** form is false. + +- **Narrow the standing rule** where it is stated without scope: `morphology.mjs:14`, + `probe-mark-labels.mjs:76`, `probe-mark-ink.mjs:65`, `probe-encodings.mjs:39`, + `build-tajweed.mjs:35`, `scripts/probe-reference.mjs:34`, `docs/design/etl-pipeline.md:241`, + `docs/design/word-indexing.md:467`, `.claude/skills/mushaf-reference/SKILL.md:110`. It becomes + a rule about what is *vendored and shipped* — which is the thing it was always defending — and + it names the gate that now enforces it. + **Leave `docs/decisions/loop-5.md:119` alone.** It says *"our shards emit no Quran text"*, and + that is scoped and true; the CC BY-ND argument standing on it is not damaged. An earlier note + of mine overstated this. +- **`LICENSES.md:56-58`** — *"The app code is not a derivative of the corpus data — it reads the + shards at runtime — so nothing here reaches our source by way of the data."* The premise is + true (every asset read in `apps/web/src/assets.ts` is a `fetch` by URL; no `readFileSync` or + `import` targets `packages/etl/data/`). The conclusion does not follow, because content + arrived by **typing**, not across that boundary. Rewrite so it claims what it can prove. +- **`LICENSES.md:131`** — the page artwork is *"reproduced here unmodified"*. False: svgo runs at + `floatPrecision 1`. Say what is applied. +- **`SOURCES.md:86`** — *"4b applies exactly two declared transforms"*, against + `vendor-pages.mjs:25-26` which says three. Measured: `POLYGON_REPAIRS` is **23 entries across + 19 distinct pages**, `ID_REPAIRS` is **2**. The code is right and the register is stale; also + correct the two-`id`-repairs description, which omits the polygon repairs entirely. --- -## The hazard, and the guard for it - -**The whole page — CSS, markup and JS — lives inside a template literal.** Backticks are -forbidden anywhere in the emitted region *including comments*, and `${` must not appear except -as a deliberate interpolation. This has broken the file three times. - -Add two assertions to `build-mark-report.test.mjs` that say so out loud: the emitted HTML -contains no backtick and no `${`. Two lines, and they document the trap. - -## Tests - -The suite today parses the head and the cards out of the emitted HTML and string-matches the -emitted script. Both styles extend cleanly; there is no visual coverage and none is being -invented here. - -| fix | what a test would actually assert | -|---|---| -| ② **highest value** | the dark block re-themes **none** of the paper, the ink, or the four new rectangle tokens; and both rectangle line colours clear 3.0:1 against the paper. Twelve lines, no browser, and it is precisely the invariant that broke. | -| ① | the paint function contains no `innerHTML`, and `c.svg` appears exactly once in the emitted script, inside `mount`. Coarse, in the same spirit as the existing replay test, and it catches the regression that matters. | -| ⑥ | data, not layout: a single-letter card always carries a longer containing word — this is what makes the fix possible and what would silently stop being true. Plus that the mark's name reaches all three branches. | -| ⑤ | the brief lede exists and the collapse is keyed off the stored flag; a regression deletes the second copy. | -| ⑧ | in `score-mark-report.test.mjs`, which already builds synthetic transcripts and asserts on stdout: one mark nudged +0.5, −0.4, +0.4 (final `to` 0.5) and a second moved once by 0.5 — assert the printed median is **0.5, not 0.4**, and that the line says **2 marks**, not 4 events. That fixture would have caught this. | -| ③④⑦ | not testable here beyond CSS-text presence. Fold position, per-frame cost and thumb geometry need a browser, and there is no in-repo harness for one. | - ## Registers -- **`docs/map.json:752`** — the `build-mark-report.mjs` note is the canonical home and already - records *why* each control sits where it does, including two prior interaction regressions. - Fixes ①②③ belong there. Hand-edited. -- **`docs/map.json`** — `serve-sittings.mjs` has **no row anywhere in `docs/`**. It needs one; - it is the thing that makes an answer survive the browser losing it. -- **`docs/validation/ledger.json` → `placement-what-kind-of-wrong`** — its 13 runbook steps - **are** the reader's on-screen instructions. ⑤ and ⑥ change what the reader sees; the steps - change with them. Then `pnpm guide`. -- **`docs/issues.json`** — only for the findings that **distorted a measurement**, which is this - repo's convention for a review tool (precedent: the "wrong instrument" paragraph inside - `nobody-has-looked-at-the-placement-verdicts`). That is finding ② and the scorer ⑧ — not the - ergonomics. Then `pnpm issues:doc && pnpm gate:issues`. -- **A design doc is not warranted yet.** `docs/design/encoding-inspector.md` is the template if - it ever is. - -### Owed from the sitting already completed, and still unrecorded - -`make record CHECK=placement-what-kind-of-wrong RESULT='…'` — sixty marks from the placed set, -every one explicitly affirmed, no faults. Bounds visible placement error at about **5%, not at -zero**. The caveat that must survive into the wording: the placed population is defined by a -match of 0.55 or better and a displacement under 3 units on a mark 5.6 × 3.6, so **gross errors -were structurally impossible on those cards**. The check also owes its `tunes` step — a manual -result must tighten something automated, and the gate fails if it tunes nothing. +- **`docs/issues.json`** — four rows are already written for this work + (`twelve-verses-of-scripture-ship-in-the-bundle`, + `the-transliteration-table-is-copied-into-our-code`, + `the-artwork-pipeline-declares-two-transforms-and-applies-three`, + `the-asset-manifest-is-in-no-licence-bucket`). Move each to `fixed` with `closedBy` as it + lands, and **add one row for the second scripture site**, which no row covers. Hand-edited. + Then `pnpm issues:doc && pnpm gate:issues && make tasks-doc`. +- **`docs/map.json`** — `gate-scripture.mjs` needs a row, and the `gate-notices.mjs` row needs + the assets-folder enumeration recorded. Hand-edited. +- **`docs/design/what-we-distribute.md`** — item ④ carries the finding and predicts this fix; + update it to what was actually done, and record the second site, which the document does not + yet know about. +- **`LICENSES.md` / `SOURCES.md`** — the two new rows from ④ and any row from ⑤. +- **No new decision record.** Nothing here is a choice between defensible options; it is a set of + claims that do not match the tree. The open licensing decisions are untouched. ## Verification -1. Rebuild all sixteen and confirm the deal is unchanged — same slice fingerprint, 1,877 marks - across sixteen parts, 0 answered marks re-asked: - `for n in 1..16; node scripts/build-mark-report.mjs --rows out/mark-rows.line-tilt.json - --set fallback --seed 23 --part $n/16 --answered … --out out/sit.fallback-$n-of-16.html` -2. `pnpm --filter @hifth/etl test` — the new assertions above, including the two hazard guards. -3. Re-run the scorer over the two banked transcripts and confirm it prints the hand and the - against-what-ships figures as **two** numbers under two sentences. -4. Open part 1 on the phone over the tailnet, in **dark mode**, and confirm by eye: both boxes - visible on white paper; the buttons do not move between cards; the drag does not stutter on - the largest card; affirm is above the fold and obviously pressed when pressed. -5. `pnpm issues:doc && pnpm gate:issues`; `pnpm guide` after the ledger edit; - `pnpm gate:validation`. -6. `git add -A && make ci`. Commit code and docs **separately**. +1. `node scripts/gate-scripture.mjs` on the tree **before** ① and ② — it must name exactly the + two files, and nothing else. A gate that cannot reproduce the finding that motivated it is + not measuring what it claims. +2. After ① and ②, the same run passes, and + `grep -c $'َ' packages/core/src/verse-text.ts` finds no file. +3. `pnpm --filter @hifth/core test && pnpm --filter @hifth/web test` — `verse-text.test.ts` is + deleted with its subject; `DiffView` gets a test that a pair with a known span washes the + right word indices on both sides. +4. Rebuild the web bundle and confirm **zero** of the 48 token strings survive in + `apps/web/dist/assets/index-*.js`. That byte-presence check is how the finding was made and + it is how the fix is proved. +5. Open a hop from 2:48 to 2:123 in the browser and confirm by eye: the panel draws real printed + words, the two divergent endings are washed, and the letters match the page underneath. +6. `pnpm gate:notices` — it must now fail if a tree is added to `apps/web/public/assets/` with + no row. Test that by adding an empty directory and confirming the failure, then removing it. +7. `pnpm issues:doc && pnpm gate:issues`, `make tasks-doc`, `git add -A && make ci`. +8. **Commit code and docs separately.** ## Operational - `cd /Users/omareid/Workspace/git/hifth && ./scripts/with-lock.sh
    diff --git a/docs/map.json b/docs/map.json index 1157c24..a67dffa 100644 --- a/docs/map.json +++ b/docs/map.json @@ -1367,12 +1367,29 @@ { "file": "scripts/build-decision-board.mjs", "symbol": "arcs", - "note": "The register drawn rather than tabulated, and the only page that shows the whole of it at once. README.md answers 'where is the row I came for'; this answers the two questions people actually arrive with — what is still open, and what is this one leaning on — and the second is the one a table structurally cannot answer, because relatedness is a fact about a PAIR and a table has to pick one row to write it in. `arcs` is that fix: every `related` edge once (`r.id >= other` drops the reciprocal half the gate insists on), drawn as a curve between two dots on a time line, terracotta if either end is still open. Nothing on the page is typed twice — the question comes from the register, the answer-in-one-line is the record's own H1 read at build time for the reason stated beside decisions.mjs's titleOf(), and the counts, dates, arcs and option strips are all derived; if it ever disagrees with the register the register is right and this page has not been rebuilt. Undated decisions are PARKED to the right of a dashed break rather than given a position, because a time line that placed an unanswered question on a day would be the one lie this picture could tell. The month labels are placed below the deepest stack rather than at a fixed drop — four decisions share 25 July 2026 and a constant looked right until the day that got a fifth. Unlike the four options pages there is one copy, not two: it carries no page artwork and no external asset, so the published copy is the checked-in file." + "note": "The register drawn rather than tabulated, and the only page that shows the whole of it at once. README.md answers 'where is the row I came for'; this answers the two questions people actually arrive with — what is still open, and what is this one leaning on — and the second is the one a table structurally cannot answer, because relatedness is a fact about a PAIR and a table has to pick one row to write it in. `arcs` is that fix: every `related` edge once (`r.id >= other` drops the reciprocal half the gate insists on), drawn as a curve between two dots on a time line, terracotta if either end is still open. Nothing on the page is typed twice — the question comes from the register, the answer-in-one-line is the record's own H1 read at build time for the reason stated beside decisions.mjs's titleOf(), and the counts, dates, arcs and option strips are all derived; if it ever disagrees with the register the register is right and this page has not been rebuilt. Undated decisions are PARKED to the right of a dashed break rather than given a position, because a time line that placed an unanswered question on a day would be the one lie this picture could tell. The month labels are placed below the deepest stack rather than at a fixed drop — four decisions share 25 July 2026 and a constant looked right until the day that got a fifth. Unlike the four options pages there is one copy, not two: it carries no page artwork and no external asset, so the published copy is the checked-in file. Its last section is the shelf — docs/artifacts.json, read the same way, drawn as cards with the pages that have no checked-in copy first." + }, + { + "file": "docs/artifacts.json", + "symbol": "artifacts", + "note": "Every page published to the web, and the only place that knows they exist. Written after counting: nine had gone out and the tree named five, because gate:decisions only ever asks the question of a page attached to a decision. The other four were built in the scratchpad, published, discussed, and the scratchpad was later cleared — so a diagnosis, a comparison carrying a recommendation, a plan and a finding survive only as an address on a host we do not own, which is exactly what 'a link with no copy dies the day the host does' was written about. Stores two facts nothing else here can hold — that a public address exists at all, and what the page calls itself there — plus one sentence of its own prose, `shows`. For the four that belong to a decision it does NOT repeat the checked-in path or the build command; decisions.json owns those. `url` IS repeated, deliberately: it is the identity, and an inventory that omits four of its nine rows is not an inventory. Hand-edited, never generated." + }, + { + "file": "scripts/artifact-sweep.mjs", + "symbol": "PUBLISHED", + "note": "Answers 'has anything been published that no register names?' and CANNOT be a gate — which is why it is not called one. A published page's address is minted by the publish and never written back into the tree; the only record that a publish happened is the session log it happened in, and those live under ~/.claude/projects on one laptop. CI cannot see them, so a gate here would pass by being unable to look, and `pnpm gates` must not grow a check like that. `PUBLISHED` is the confirmation line it reads, and it captures the source path as well as the address because that is the one moment anything records whether the page came from the tree or from a scratch directory. `--hook` reads a PostToolUse payload instead of the logs — at hook time the publish may not be flushed to disk yet, and the payload carries the same sentence. It reports and never edits: a register this repo writes for itself is one nobody reads." + }, + { + "file": "scripts/hook-artifact.sh", + "symbol": "command -v node", + "note": "The reminder, fired by the PostToolUse hook in .claude/settings.json the moment a page is published — while the page, its subject and the reason for it are all still in hand. Reminding later is asking somebody to reconstruct. It exists as a shell wrapper rather than a bare `node …` in the settings file because settings.json is checked in and shared, and a hook runs in whatever environment the editor has: where node lives under nvm and nothing sourced it, the bare form fails with 'command not found' and is reported as an error against a publish that succeeded, which teaches people to delete the hook. So it looks for node, and if there is genuinely none it says nothing and leaves. `set -e` is deliberately absent for the same reason." } ], "extend": [ "A new decision is a record in docs/decisions/ (or a section of the design doc that owns it), a row in docs/decisions.json, and `make decisions-doc`. The gate refuses a record with no row, which is the direction nothing else checks.", "The board redraws itself from the register — a new decision appears on it with no edit here. It is only two commands because it is not a decision page: nothing gates it, so rebuild it in the same commit that moves a row or it silently shows last week's board.", + "Publishing a page is not finished until docs/artifacts.json has a row for it and the board is rebuilt. The hook will say so within seconds of the publish; nothing in CI ever will, because the evidence that a publish happened does not exist inside this repository. Treat the reminder as the mechanism, not as a nag on top of one.", + "A page built in the scratchpad and published is one `rm -rf` away from existing only as a link. If it is worth publishing it is worth a checked-in copy and a script that rebuilds it — and if it turns out to be a decision, it wants a row in the decision register rather than a row on the shelf.", "One run writes both copies of an options page — the checked-in one and the published one. A flag that emits one at a time is two commands, and the repo cannot tell you when the second was never run.", "Write the question in plain words or the build says so. No file names, no symbols, no commands — the record's body is where those belong. See CLAUDE.md for why this is a tenet and not a style note.", "An open decision needs two or more options and something to look at, published AND checked in. If there is nothing to look at yet, the decision is not open, it is unstarted.", diff --git a/scripts/build-decision-board.mjs b/scripts/build-decision-board.mjs index cbab941..4f2c671 100644 --- a/scripts/build-decision-board.mjs +++ b/scripts/build-decision-board.mjs @@ -12,6 +12,16 @@ * and the winner marked. The three that nobody has chosen yet come first, * because they are the only ones anybody can still act on. * + * IT ALSO CARRIES THE SHELF. The last section is docs/artifacts.json — every + * page this project has published to the web, which turned out to be nine when + * the tree knew about five. Four of them were built in a scratch directory, + * published, and then the directory was cleared, so the address is the only + * copy left. Those four lead the section rather than being tucked under it: an + * inventory whose worst rows are at the bottom is one nobody scrolls to. The + * board is where they belong because four of the nine ARE the decision pages, + * and because this page is the one thing that gets sent to somebody who has no + * repository — so it should be able to hand them everything else too. + * * NOTHING HERE IS TYPED TWICE. The question comes from the register, which is * the only place it is stored; the answer-in-one-line is the record's own H1, * read at build time for the reason decisions.mjs states beside titleOf() — a @@ -38,6 +48,7 @@ import { ROOT } from "./code-pointers.mjs"; import { readDecisions, titleOf, splitDoc } from "./decisions.mjs"; const OUT = join(ROOT, "docs/design/decision-board.html"); +const ARTIFACTS = join(ROOT, "docs/artifacts.json"); const esc = (s) => String(s).replace(/&/g, "&").replace(//g, ">").replace(/"/g, """); @@ -243,6 +254,56 @@ const card = (r, big) => ` ${links(r)}
    `; +/* ── The shelf ─────────────────────────────────────────────────────────────── + * Every page published to the web. Four of the nine belong to a decision and + * take their whereabouts from that decision's row rather than repeating it; + * one is this page; and four have no copy anywhere at all. Those four sort + * first, because they are the only rows on the shelf that anybody has to do + * something about. + */ +const shelf = JSON.parse(readFileSync(ARTIFACTS, "utf8")).artifacts.map((a) => { + const d = a.decision ? byId.get(a.decision) : null; + return { ...a, d, kept: Boolean(d?.page ?? a.page), page: a.page ?? d?.page ?? null }; +}); +const loose = shelf.filter((a) => !a.kept); +const held = shelf.filter((a) => a.kept).sort((x, y) => y.published.localeCompare(x.published)); + +/* One row on the shelf is this page. Saying "kept here too" of it, and offering + * a link to the file the reader already has open, is the kind of small wrongness + * that makes somebody distrust the rest of the list. */ +const SELF = OUT.replace(ROOT, ""); + +const pub = (a) => { + const self = a.page === SELF; + const bits = []; + if (a.d) { + bits.push( + `

    Drawn for` + + `${esc(a.d.question)}

    `, + ); + } + if (a.note) bits.push(`

    ${esc(a.note)}

    `); + const out = [`${self ? "The published copy" : "Open the page"}`]; + if (a.page && !self) out.push(`the same page, kept here`); + bits.push(`

    ${out.join('·')}

    `); + return ` +
    +
    + ${ + self + ? `You are reading it` + : a.kept + ? `Kept here too` + : `No copy — the link is all there is` + } + ${longDate(a.published)} +
    +

    ${esc(a.title)}

    +

    ${esc(a.shows)}

    + ${bits.join("")} +
    `; +}; + /* ── The page ─────────────────────────────────────────────────────────────── */ const counts = { open: open.length, settled: settled.length, living: living.length }; @@ -441,6 +502,20 @@ h2 .n { .links .sep { color: var(--rule); margin: 0 0.5rem; } .go { font-weight: 600; } +/* The shelf. A page with no copy anywhere gets the same terracotta the open + questions get, because it is the same kind of fact: something here needs a + person, and nothing will happen on its own. */ +.pub h3 { margin-bottom: 0.5rem; } +.pub-title { text-decoration: none; } +.pub-title:hover { text-decoration: underline; } +.pub-loose { border-color: var(--terra); } +.shows { font-size: 0.97rem; color: var(--soft); margin: 0 0 0.9rem; } +.belongs { font-size: 0.92rem; margin: 0 0 0.9rem; } +.orphan-note { + font-size: 0.9rem; color: var(--soft); margin: 0 0 0.9rem; + padding: 0.6rem 0.8rem; border-radius: 3px; background: var(--terra-soft); +} + .note { margin-top: 4.6rem; padding-top: 1.5rem; border-top: 1px solid var(--rule); font-size: 0.9rem; color: var(--faint); max-width: 46rem; @@ -459,7 +534,9 @@ h2 .n {

    Every question this project has had to answer, on one page: the ones that are settled, the ones nobody has chosen yet, and the lines between them that mean two answers have to agree. Each card carries the question in plain words, the options as they - were actually put, and a way through to the drawing and to the reasons.

    + were actually put, and a way through to the drawing and to the reasons. The last section is + the shelf: every page this project has published for somebody to look at, including the ones + that are not attached to any decision at all.

    • ${counts.open}still open
      • @@ -467,6 +544,7 @@ h2 .n {
    • ${counts.living}never finished
    • ${arcs.length}constraints, across ${linked} of them
    • ${(totalWords / 1000).toFixed(1)}kwords of reasons behind it
      • +
    • ${shelf.length}pages published${loose.length ? `, ${loose.length} with no copy` : ""}
    ${spine} @@ -505,10 +583,28 @@ h2 .n {
    ${settled.map((r) => card(r, false)).join("")}
    +
    +
    +

    The shelfWhat has been published, and can anybody still see it?

    +

    Every page this project has put on the web, in one place. Of the ${shelf.length}, + ${shelf.filter((a) => a.d).length} are the drawings the decisions above were made from; the + rest were made to settle something in the moment — a diagnosis, a comparison, a finding, a + plan — and were never attached to anything.

    +

    And ${loose.length} of them exist nowhere but at the address on the card. They were drawn + in a working folder, published, talked about, and the folder was later emptied. Nobody chose + that; it is what happens when the last step of publishing something is remembering to. They + come first here for the same reason the open questions do — they are the rows that need a + person.

    +
    +
    ${loose.map(pub).join("")}
    +
    ${held.map(pub).join("")}
    +
    +

    Drawn from this project's decision register on every build. The questions, the options, the dates, the arcs and the counts are all read from it — nothing on this page is typed a second time, so if it ever disagrees with the register, the register is right and this -page has not been rebuilt. Rebuild with node scripts/build-decision-board.mjs.

    +page has not been rebuilt. The shelf is read the same way from its own register. Rebuild with +node scripts/build-decision-board.mjs.

    `; @@ -516,5 +612,6 @@ writeFileSync(OUT, html); console.log( `${OUT.replace(ROOT, "")} ${(html.length / 1024).toFixed(0)} KB — ` + `${rows.length} decisions (${counts.open} open, ${counts.settled} decided, ${counts.living} living), ` + - `${arcs.length} constraints across ${linked}, ${totalWords.toLocaleString("en")} words of reasons`, + `${arcs.length} constraints across ${linked}, ${totalWords.toLocaleString("en")} words of reasons, ` + + `${shelf.length} published pages (${loose.length} with no copy anywhere)`, ); From 2d0e8a0d86b71bc5e65af8a2d7ae5d47b3efb4d5 Mon Sep 17 00:00:00 2001 From: Omar Eid Date: Sun, 16 Aug 2026 22:27:06 -0500 Subject: [PATCH 44/95] Quoting a receipt is not a purchase MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The sweep counted a publish wherever the confirmation sentence appeared in a session log, which is not the same thing and was never going to be. Testing it put two invented confirmations into the log of the session doing the testing, and the next run reported them as two real pages nobody had registered — eleven published where nine had been. A listing of the account's pages would have done the same, twenty-four at a time. It now parses the logs instead of grepping them: collect the ids of the calls that were actually the publish tool, and read the confirmation only out of the results those calls returned. A shell command that prints the sentence, or a listing that pastes two dozen addresses, cannot forge that pairing. Back to nine published against nine registered. The reminder itself is unchanged — it reads the payload it is handed and never went near the logs. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt --- scripts/artifact-sweep.mjs | 53 +++++++++++++++++++++++++++++++------- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/scripts/artifact-sweep.mjs b/scripts/artifact-sweep.mjs index cae6d17..70066d3 100644 --- a/scripts/artifact-sweep.mjs +++ b/scripts/artifact-sweep.mjs @@ -81,24 +81,57 @@ function decisionLinks() { return links; } -/* Every publish this machine has a record of, newest occurrence winning for the - * source path — a page republished from the tree after being drafted in scratch - * has stopped being an orphan, and the last publish is the one that says so. */ +/* Every publish this machine has a record of. + * + * NOT by grepping the log. A log is full of that sentence for reasons that are + * not publishes: a listing of the account's pages pastes two dozen addresses + * into whichever session ran it, and testing this very script wrote two fake + * confirmations into the log of the session that tested it — which the first + * version of this function then reported as two unregistered pages. Quoting a + * receipt is not a purchase. + * + * So: parse the log, take the ids of the calls that were actually the publish + * tool, and read the confirmation only out of the results those calls returned. + * Nothing else in a transcript can forge that pairing. + * + * Newest occurrence wins for the source path — a page drafted in a scratch + * directory and later republished from the tree has stopped being an orphan, + * and the last publish is the one that says so. */ function publishes(dir) { const found = new Map(); if (!existsSync(dir)) return found; for (const f of readdirSync(dir).filter((n) => n.endsWith(".jsonl"))) { - let text; + let lines; try { - text = readFileSync(join(dir, f), "utf8"); + lines = readFileSync(join(dir, f), "utf8").split("\n"); } catch { continue; // a log being written while we read it is not an error here } - for (const [, path, url] of text.matchAll(PUBLISHED)) { - const prev = found.get(url) ?? { url, times: 0, paths: [] }; - prev.times += 1; - if (!prev.paths.includes(path)) prev.paths.push(path); - found.set(url, prev); + const fromPublish = new Set(); + for (const line of lines) { + if (!line.trim()) continue; + let ev; + try { + ev = JSON.parse(line); + } catch { + continue; + } + const content = ev?.message?.content; + if (!Array.isArray(content)) continue; + for (const part of content) { + if (part.type === "tool_use" && part.name === "Artifact") fromPublish.add(part.id); + if (part.type !== "tool_result" || !fromPublish.has(part.tool_use_id)) continue; + const text = + typeof part.content === "string" + ? part.content + : (part.content ?? []).map((c) => c.text ?? "").join("\n"); + for (const [, path, url] of String(text).matchAll(PUBLISHED)) { + const prev = found.get(url) ?? { url, times: 0, paths: [] }; + prev.times += 1; + if (!prev.paths.includes(path)) prev.paths.push(path); + found.set(url, prev); + } + } } } return found; From 2ec9f4d7822f9aa06f06f7d3ebbf93c060865d66 Mon Sep 17 00:00:00 2001 From: Omar Eid Date: Mon, 17 Aug 2026 06:17:14 -0500 Subject: [PATCH 45/95] The panel that got rebuilt had never been on the map MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deleting the typed table and redrawing the comparison from the page artwork changed where that feature lives, and the map did not know where it lived before either — the only mention was a note on the popover saying it showed "the diff against the current ayah", pointing at nothing. So the two files somebody would actually need to open now have rows: the arithmetic on word indices, and the panel that crops both pages through it. The note on the first one carries the reason the table went, because that is the part a later reader will otherwise undo: it is easy to look at a panel built from word boxes, find it fiddly, and think a small text table would be simpler. It would be, and it would reintroduce running scripture to a code repository, show the reader a plainer spelling than the page underneath, and cover twelve ayahs out of six thousand. A skill heading still asserted the unscoped claim — no Quran text enters this repo — three lines above quoting the scoped rule that replaced it. A heading that contradicts its own paragraph is the version people remember. Two checks the fix had promised and nobody had run: none of the deleted table's 36 distinct token strings is byte-present in the 337 KB of built JS, and adding an undeclared tree under the shipped assets folder does now fail the licence map by name, which is the behaviour that gate was extended for. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt --- .claude/skills/mushaf-reference/SKILL.md | 4 +++- docs/map.json | 10 ++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/.claude/skills/mushaf-reference/SKILL.md b/.claude/skills/mushaf-reference/SKILL.md index 143abd0..14bd4db 100644 --- a/.claude/skills/mushaf-reference/SKILL.md +++ b/.claude/skills/mushaf-reference/SKILL.md @@ -107,7 +107,9 @@ These are not style preferences. Each one is a rule this repo already enforces s the geometry comes from the vendored KFGQPC SVGs, and a differently-printed page table would be "a source of disagreement rather than of truth." Check against it; do not import it. -2. **No Quran text enters this repo.** The standing rule is in +2. **Nothing we vendor and nothing we ship is Quran text.** Not the unscoped form + this heading used to carry — it contradicted the rule quoted directly beneath it, + and the tree has never satisfied it. The standing rule is in `packages/etl/scripts/morphology.mjs` — *"nothing this project vendors, and nothing it ships, is Quran text"* — and `gate:scripture` is what enforces it, failing on any run of three consecutive fully-vowelled words in any source file. It is not `gate:notext`, which diff --git a/docs/map.json b/docs/map.json index a67dffa..7b70aeb 100644 --- a/docs/map.json +++ b/docs/map.json @@ -110,6 +110,16 @@ "symbol": "HopPopover", "note": "The chip's expansion: the destinations, and the diff against the current ayah." }, + { + "file": "packages/core/src/verse-diff.ts", + "symbol": "wordDiff", + "note": "Which words two look-alike ayahs share and which they do not, as arithmetic on word indices — no text anywhere in it. Read the header before changing anything here: this replaced a hand-typed table of twelve ayahs' text that shipped in the bundle, and every one of the three reasons it went matters. It was running scripture in a code repository, which is the shape takedown notices get scoped to; it was typed in a plainer spelling than the mus'haf on screen, so the panel showed the reader different letters from the page underneath it; and twelve ayahs out of six thousand is not a feature. None of it was needed, because an edge already carries the matching run on BOTH sides in the print's own word numbering — the same numbering the word boxes use — so the shared stretch is a lookup and what diverges is the leftover at either end. 2,544 of 2,996 look-alike edges carry a span; `null` for the other 452 is not a failure but the data declining to guess, since build-adjacency only emits a span where the shared run occurs in exactly ONE place on both sides. `divergentRuns` beside it returns ranges rather than indices because the caller paints them with `WordIndex.bandsFor`, which wants a run whole so it can collapse it to one rectangle per line of print." + }, + { + "file": "apps/web/src/components/DiffView.tsx", + "symbol": "DiffView", + "note": "The panel that answers 'why do these two get confused' by drawing the mus'haf's own ink, cropped word by word out of both pages, with the divergent tails washed. The reader compares the printing rather than a transcription of it — which is the point, since a transcription is exactly what was wrong before. Costs a fetch of the TARGET's page, so it is lazy on expand: the panel lives inside a popover opened on demand, and both call sites (`HighlightMenu`, `HopPopover`) pass the page in only once it is open. Falls back to the plain note when there is no span, no shard, or no page — the same fallback the old table left for every pair it did not cover." + }, { "file": "apps/web/src/components/PageStage.tsx", "symbol": "async navigateTo", From 507b02d33dca27605ce2f7a4dcc0d1901c96125b Mon Sep 17 00:00:00 2001 From: Omar Eid Date: Mon, 17 Aug 2026 08:29:29 -0500 Subject: [PATCH 46/95] The plan file was describing work that had already been done MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both plans this file has held are finished, and it was still written as though neither had started. A plan that claims outstanding work is worse than no plan: somebody reads it and does the work twice, or reads one stale claim and stops trusting the rest of the file. The displaced sitting plan was recovered and audited item by item against the tree rather than against its own account of itself. All eight fixes landed, the guard against the trap the page is built in landed, the registers landed, and the item it called owed and still unrecorded is recorded — saying considerably more than the plan expected, because reading that sitting turned up two further defects in the instrument. So it is not restored as a plan. What was worth keeping from it is the table of what each fix was for. What is left in both cases is reader work. 1,717 of 1,877 marks have not been sat, and two checks need a device and a person. The instrument being trustworthy was the point of that plan; the instrument existing never was. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt --- .claude/plans/shimmering-wandering-lobster.md | 305 ++++++------------ 1 file changed, 95 insertions(+), 210 deletions(-) diff --git a/.claude/plans/shimmering-wandering-lobster.md b/.claude/plans/shimmering-wandering-lobster.md index d10db81..30f55c3 100644 --- a/.claude/plans/shimmering-wandering-lobster.md +++ b/.claude/plans/shimmering-wandering-lobster.md @@ -1,218 +1,103 @@ -# The repo's most-repeated claim is false, and nothing was checking it +# Both plans this file has held are finished -> **This file previously held the sitting-instrument plan**, which is unfinished and is *not* -> superseded. It is recoverable in full at `git show 83041f3:.claude/plans/shimmering-wandering-lobster.md` -> and should be restored to its own file before that work resumes. +This file has carried two plans. **Neither is outstanding.** It is kept as the record of +what they were and how each was checked, because the second one was audited on +2026-08-17 against the tree rather than against its own account of itself, and that +audit is the useful thing here. -## Context - -Twenty-two times across twenty files, in three shipped `NOTICE.txt` files and in the sentence -`docs/decisions/loop-5.md:119` leans on, this repo asserts some version of *"there is no Quran -text here."* Measured against the tree, the unscoped form of that claim is false. - -Two sites hold running scripture. Measurement is the **longest run of consecutive fully-vowelled -words** — one vowelled word is a specimen, a phrase is a passage somebody can recover: - -| file | longest phrase | vowelled words | ships to the site? | -| --- | --- | --- | --- | -| `packages/core/src/verse-text.ts` | **11 words** | 160 | **yes** — all 48 token strings are byte-present in `apps/web/dist/assets/index-*.js` | -| `packages/etl/scripts/lib/tajweed-fold.test.mjs` | **5 words** | 29 | no — pipeline test, repository only | - -Every other Arabic-bearing file in the tree scores **1 or 2**: our own interface Arabic -(`ar.gen.ts`, `format.ts`), domain terms (*madd*, *tamm*), and single-word pipeline specimens -(tajweed edge cases, join fixtures, encoding probes). **Nothing sits at 3 or 4.** That gap is -what makes a gate possible rather than a matter of taste. - -Nothing caught either site. `gate:notext` sounds like it would and does not — it forbids `` -elements in page SVGs because of a Safari paint bug, which is a rendering requirement wearing a -name that reads like a scripture check. - -**Why it matters more than tidiness.** The Crossway/ESV takedown campaign of July–August 2025 -ran eleven notices, roughly half of them **path-scoped into code repositories** — a data -directory in one, a **test-fixtures directory** in another. Blast radius tracks the directory -holding the text, not the project's purpose. Both of our sites are that shape. Unlike everything -else in `docs/design/what-we-distribute.md`, this needs no licensing opinion from anybody. - -**And the shipped panel is also wrong on its own terms.** `verse-text.ts` is typed in a plainer -spelling than the mus'haf on screen. A reader comparing 2:48 and 2:123 sees one spelling in the -panel and different letters on the page underneath it. - -**Outcome:** no running scripture in the tree, a gate that refuses the next one, the licensing -map covering everything that actually ships, and the panel showing the reader the same ink the -page does. +What is left in both cases is **reader work, not code** — sittings that have to be sat, a +verdict that has to be given on a device. Those live in `docs/validation/ledger.json` +and `docs/issues.json`, which are where somebody should look, not here. --- -## The work - -Ordered so the gate lands last and passes on arrival. - -### ① Draw the diff panel from the page artwork — deletes `verse-text.ts` - -**The classification is already shipped, for every edge.** This was the discovery that made the -option cheap. Adjacency edges already carry the target's page and the matching word range on -both sides: - -``` -2:48 → 2:123 page 19 span {from:[1,13]} toSpan {from:[1,13]} -``` - -and the word shards already carry per-ayah boxes in the same index space: - -``` -2:48 page 7 from 1 23 boxes → indices 1..23 -2:123 page 19 from 1 22 boxes → indices 1..22 -``` - -Words 1–13 are the shared opening; 14–23 against 14–22 is where they diverge. That is exactly -what the hand-typed `cls 0 / 1 / 2` encoded — **already present, for all 44,431 links rather -than twelve ayah pairs.** - -So the panel becomes: read the edge's `span`/`toSpan`, load each side's page SVG and word shard, -crop each word's box out of the page, and wash the words that fall outside the shared range. - -- **Reuse, do not build:** `loadPageSvg` and `loadWordShard` (`apps/web/src/assets.ts:90,197`) - and `WordIndex` (`packages/core/src/words.ts:120`), which already keys bare `"2:48"` to rects - in reading order and holds the print's start index. -- **Rewrite** `apps/web/src/components/DiffView.tsx` — `TokenRow` becomes a row of cropped word - SVGs. Keep `lang="ar" dir="rtl"` and the existing `styles.dA`/`dB` washes. -- **Replace** `diffPair` in `packages/core/src/verse-text.ts` with a pure function over an edge — - no text, no table. Rename the module to say what it now is; drop `VERSE_TEXT`, `verseTokens`, - `DiffToken`, `DiffClass` from `packages/core/src/index.ts:105-111`. -- **First step before any of it:** confirm on 2:48/2:123 that `span.from` and the shard's `from` - share an index space across a page boundary. The whole design rests on that one assumption and - it is ten minutes to check. -- **The target's page must be fetched** when the panel expands — it is inside a popover opened on - demand, so lazy on expand. Both call sites: `HighlightMenu.tsx:213`, `HopPopover.tsx:166`. -- **Falls back exactly as today:** no edge, no shard, or no page → render nothing, and the hop row - keeps its plain note. That path already exists for the 6,224 pairs the table never covered. -- **While in there:** edge `note` fields carry bare Arabic (`"شفاعة ↔ عدل order swapped"`) and - ship in the adjacency assets. Two unvowelled words is a specimen, not a phrase — but check - where the note text originates before leaving it. - -### ② The second site — `packages/etl/scripts/lib/tajweed-fold.test.mjs` - -Five running words. Replace the phrase fixtures with per-word cases drawn from the same -edge-case set the rest of the file already uses, or with escaped code points where a word is -genuinely the thing under test. **Do not weaken what the test asserts** — it is checking fold -behaviour on real orthography and that coverage has to survive. - -### ③ A gate that refuses the next one — `scripts/gate-scripture.mjs` - -Model it on `scripts/gate-text-sources.mjs` exactly: same -`git ls-files --cached --others --exclude-standard` enumeration, same `SOURCE_RE`/`EXCLUDE_RE` -split with the reasoning stated in the file, same untracked-files rationale (its comments at -lines 42–56 explain why tracked-only is a real blind spot; that argument applies here unchanged). - -**The rule:** fail on any run of **three or more consecutive fully-vowelled Arabic words**, where -a word counts as vowelled at half or more of its letters carrying a mark. Measured margin — 11 -and 5 on the two offenders, 1 or 2 everywhere else, nothing between. - -**The allowlist:** a named list of single-specimen files with a one-line reason each, so the -~90 lone vowelled words in `probe-tajweed-words.mjs`, `tajweed-fold.mjs`, `mark-join*.mjs`, -`probe-diacritics.mjs` and `segmentation.mjs` read as reviewed rather than unnoticed. **The -phrase rule is not allowlistable** — a run of three fails regardless of what the list says. - -Wire into `package.json` beside the other gates, into `make ci`, and into `gate:gates`. - -State in the header what `gate:notext` is actually about, so the next reader does not assume that -one covers this. - -### ④ Make the licensing map see the whole assets folder — `scripts/gate-notices.mjs` - -Three defects, all in one file: - -- **Blind by construction.** `BUCKETS` declares three paths (`roots`, `skins`, `adj`) and the - cross-check at line 187 compares that declaration against `LICENSES.md` in both directions — - but it never enumerates `apps/web/public/assets/` itself. Add that enumeration: every entry - must be either a declared bucket or a named entry in an "ours, no inherited terms" list. -- **Two shipped trees are in no bucket.** `words/hafs-kfqc/` — 604 files, 2.8 MB — and - `manifest.json` at 24,471 bytes. The string `words` appears in neither `SOURCES.md` nor - `LICENSES.md` nor this gate. (`pages` is deliberate: it is covered by `LICENSES.md`'s "What we - do not license" — keep it out of the buckets and put it in the named list with that reason.) -- **A false declaration.** Line 103 asserts `"ayah-pages.json": "ours"` for the roots bucket. - Confirmed false. Line 129 already declares the honest form for the same file in the `adj` - bucket — mirror that. - -### ⑤ Say where the transliteration table came from — `packages/etl/scripts/build-roots.mjs:86` - -The comment reads *"Buckwalter → Arabic (corpus.quran.com/java/buckwalter.jsp, verbatim)"* — a -table copied from the GPL corpus's own page. Buckwalter is a published transliteration **scheme** -and a forty-odd entry character mapping is thin ground for anyone to claim, but the comment as -written says we copied it from them. Cite the scheme's original publication rather than the -corpus's rendering of it, and add a row to `SOURCES.md` if the citation does not resolve to -something already listed there. - -### ⑥ The prose that is wrong - -Four sites. **The distinction that matters:** *scoped* claims stay — "our shards carry no Quran -text", "no Quran text crosses the wire", "this file contains no Quran text" are all still true -and all still worth saying. Only the **unscoped repo-wide** form is false. - -- **Narrow the standing rule** where it is stated without scope: `morphology.mjs:14`, - `probe-mark-labels.mjs:76`, `probe-mark-ink.mjs:65`, `probe-encodings.mjs:39`, - `build-tajweed.mjs:35`, `scripts/probe-reference.mjs:34`, `docs/design/etl-pipeline.md:241`, - `docs/design/word-indexing.md:467`, `.claude/skills/mushaf-reference/SKILL.md:110`. It becomes - a rule about what is *vendored and shipped* — which is the thing it was always defending — and - it names the gate that now enforces it. - **Leave `docs/decisions/loop-5.md:119` alone.** It says *"our shards emit no Quran text"*, and - that is scoped and true; the CC BY-ND argument standing on it is not damaged. An earlier note - of mine overstated this. -- **`LICENSES.md:56-58`** — *"The app code is not a derivative of the corpus data — it reads the - shards at runtime — so nothing here reaches our source by way of the data."* The premise is - true (every asset read in `apps/web/src/assets.ts` is a `fetch` by URL; no `readFileSync` or - `import` targets `packages/etl/data/`). The conclusion does not follow, because content - arrived by **typing**, not across that boundary. Rewrite so it claims what it can prove. -- **`LICENSES.md:131`** — the page artwork is *"reproduced here unmodified"*. False: svgo runs at - `floatPrecision 1`. Say what is applied. -- **`SOURCES.md:86`** — *"4b applies exactly two declared transforms"*, against - `vendor-pages.mjs:25-26` which says three. Measured: `POLYGON_REPAIRS` is **23 entries across - 19 distinct pages**, `ID_REPAIRS` is **2**. The code is right and the register is stale; also - correct the two-`id`-repairs description, which omits the polygon repairs entirely. +## Plan one — the repo's most-repeated claim was false + +**Landed** across `87b4241`, `5676128`, `a97fa98`, `a053bbd`, with two gaps closed later +in `2ec9f4d`. + +Twenty-two times across twenty files this project asserted some version of *there is no +Quran text here*, and two files held running scripture — one of which shipped in the +bundle. Both are gone. The claim is now scoped to what is vendored and shipped, and a +gate refuses the next one. + +| item | state | +| --- | --- | +| ① draw the comparison from the page artwork, delete the typed table | done — `verse-diff.ts` replaces `verse-text.ts`, panel redrawn | +| ② the second site, a pipeline test holding a phrase | done | +| ③ a gate that refuses the next one | done — wired into the composite, the Makefile and CI | +| ④ make the licence map see the whole assets folder | done | +| ⑤ say where the transliteration table came from | done | +| ⑥ the nine prose sites stating the rule unscoped | done | +| registers — issues, map, the licensing design doc | done; the second site is tracked on its own row | + +**Two things the plan promised and nobody had run, since checked and passing:** none of +the deleted table's 36 distinct token strings is byte-present in the built bundle +(strings recovered from git, never typed); and adding an undeclared tree under the +shipped assets folder now fails the licence map by name — tested by creating one, +confirming the failure, removing it. + +**Two gaps the plan did not know it had.** The comparison panel had no row on the code +map — not after the rewrite and not before it, the only trace being a note on the popover +saying it showed "the diff against the current ayah" and pointing at nothing. The map +gate missed it because it only checks pointers in *staged* files, so deleting a mapped +file and adding its replacement passes in silence. Both files have rows now, and the note +on the arithmetic carries *why* the table went, because that is the part a later reader +would otherwise undo: a small text table genuinely does look simpler, and it would bring +back all three defects at once. Separately, a skill heading still asserted the unscoped +claim three lines above quoting the scoped rule that replaced it. + +**One measurement worth not re-taking.** Six of fourteen shipped edge notes carry Arabic, +up to five words in one — more than the plan assumed — but the longest fully-vowelled run +across all of them is **0**. Specimens, not passages, so within the rule as the gate +measures it. --- -## Registers - -- **`docs/issues.json`** — four rows are already written for this work - (`twelve-verses-of-scripture-ship-in-the-bundle`, - `the-transliteration-table-is-copied-into-our-code`, - `the-artwork-pipeline-declares-two-transforms-and-applies-three`, - `the-asset-manifest-is-in-no-licence-bucket`). Move each to `fixed` with `closedBy` as it - lands, and **add one row for the second scripture site**, which no row covers. Hand-edited. - Then `pnpm issues:doc && pnpm gate:issues && make tasks-doc`. -- **`docs/map.json`** — `gate-scripture.mjs` needs a row, and the `gate-notices.mjs` row needs - the assets-folder enumeration recorded. Hand-edited. -- **`docs/design/what-we-distribute.md`** — item ④ carries the finding and predicts this fix; - update it to what was actually done, and record the second site, which the document does not - yet know about. -- **`LICENSES.md` / `SOURCES.md`** — the two new rows from ④ and any row from ⑤. -- **No new decision record.** Nothing here is a choice between defensible options; it is a set of - claims that do not match the tree. The open licensing decisions are untouched. - -## Verification - -1. `node scripts/gate-scripture.mjs` on the tree **before** ① and ② — it must name exactly the - two files, and nothing else. A gate that cannot reproduce the finding that motivated it is - not measuring what it claims. -2. After ① and ②, the same run passes, and - `grep -c $'َ' packages/core/src/verse-text.ts` finds no file. -3. `pnpm --filter @hifth/core test && pnpm --filter @hifth/web test` — `verse-text.test.ts` is - deleted with its subject; `DiffView` gets a test that a pair with a known span washes the - right word indices on both sides. -4. Rebuild the web bundle and confirm **zero** of the 48 token strings survive in - `apps/web/dist/assets/index-*.js`. That byte-presence check is how the finding was made and - it is how the fix is proved. -5. Open a hop from 2:48 to 2:123 in the browser and confirm by eye: the panel draws real printed - words, the two divergent endings are washed, and the letters match the page underneath. -6. `pnpm gate:notices` — it must now fail if a tree is added to `apps/web/public/assets/` with - no row. Test that by adding an empty directory and confirming the failure, then removing it. -7. `pnpm issues:doc && pnpm gate:issues`, `make tasks-doc`, `git add -A && make ci`. -8. **Commit code and docs separately.** - -## Operational - -- `cd /Users/omareid/Workspace/git/hifth && ./scripts/with-lock.sh