From 2220f66e45adc89817eb63e608d8635052ce47f8 Mon Sep 17 00:00:00 2001 From: Omar Eid Date: Fri, 7 Aug 2026 07:41:41 -0500 Subject: [PATCH 1/2] The inspector learns which two boxes, and stops being able to open blank MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `inspector-draws-no-ink` asked for a caller before spending the work, and the caller arrived: a maintainer reading a 2→1 paint block at word 3 wants to see *which two boxes*, and `assets/words/**` has them. Section ① now draws them. Outlines only, and the line moved by exactly the distance the question proposed. The boxes come from the committed `assets/words/hafs-kfqc/.json` shards — `gate:words` checks those re-derive from committed bytes — on the page's own frame from `manifest.viewBox`, with `viewBoxOverrides` honoured for pages 1–2. Nothing is read from `assets/pages/**`, no glyph is drawn, and §6.4 keeps its force over the half that mattered: the principle protects "makes no claim about appearance", and a rectangle out of a gated shard makes none. Three things were settled before anything was drawn. `readBoxes` asserts `from === 1` rather than assuming it. It is true of all 6,236 ayahs today — no ayah's boxes are split across two shards — which is what makes print word *i* be `boxes[i - 1]` unconditionally. A non-1 `from` would shift every label by the offset and look entirely correct. A box is tinted on its **host** index, not its print index, so a word the fold dropped is a gap on the page exactly as it is a gap on the codepoint ruler. That shared parity is the whole reason ① and ④ are one instrument rather than two pictures, and it is why clicking a box selects the annotation over it and lights the ruler in step. When the box count and the `data-hafs` word count disagree, the view draws the geometry and refuses to number it, warning with both counts. The failure it would otherwise produce is word 7's box under word 8's label, which looks fine. Measured over the full corpus: 6,236 ayahs outlined, 0 mismatches. The check stays; the number is a result, not a guarantee. One invisible defect was found by building it. Both halves of the report are *text* to `probe-encodings.mjs` and are never imported, so nothing noticed a syntax error in the client: the report generated, weighed its usual 4.9 MB, and opened to a blank page with the whole script dead in the console. It now compiles the exact concatenation the browser will parse (`new vm.Script`) before writing, and fails at generation time instead. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt --- .../scripts/lib/encoding-inspector.client.mjs | 171 +++++++++++++++++- .../etl/scripts/lib/encoding-inspector.css | 35 ++++ packages/etl/scripts/probe-encodings.mjs | 109 +++++++++++ 3 files changed, 312 insertions(+), 3 deletions(-) diff --git a/packages/etl/scripts/lib/encoding-inspector.client.mjs b/packages/etl/scripts/lib/encoding-inspector.client.mjs index bfc665c..8dcd6cf 100644 --- a/packages/etl/scripts/lib/encoding-inspector.client.mjs +++ b/packages/etl/scripts/lib/encoding-inspector.client.mjs @@ -50,6 +50,7 @@ const state = { view: "ayah", key: "2:4", annotation: null, + context: true, // ①: draw the page's other ayahs faintly behind this one filter: null, // { label, keys: [ayah keys] } sort: {}, // table id → { col, dir } }; @@ -274,6 +275,161 @@ function cpCell(cps, i, hostAt, inSpan) { ]); } +// ----------------------------------------------------------------- the outline -- + +/** + * `el`, for SVG. A separate helper rather than a flag on `el` because + * `createElement("rect")` produces an HTMLUnknownElement that lays out as + * nothing and reports no error — the single most confusing way this section + * could fail, and worth one extra function to make impossible. + */ +const svgEl = (tag, attrs = {}, kids = []) => { + const n = document.createElementNS("http://www.w3.org/2000/svg", tag); + for (const [k, v] of Object.entries(attrs)) { + if (k === "text") n.textContent = v; + else if (k.startsWith("on")) n.addEventListener(k.slice(2), v); + else if (v !== null && v !== undefined) n.setAttribute(k, String(v)); + } + for (const kid of [].concat(kids)) if (kid) n.append(kid); + return n; +}; + +const GEO = DATA.geometry ?? null; + +/** page number → the ayah keys whose boxes sit on it, for the faint context layer. */ +const PAGE_AYAHS = (() => { + const m = new Map(); + if (!GEO) return m; + for (const [key, p] of Object.entries(GEO.page)) { + if (!m.has(p)) m.set(p, []); + m.get(p).push(key); + } + return m; +})(); + +/** `[width, height]` of a page's frame — pages 1 and 2 are square, the rest are not. */ +const frameOf = (page) => (GEO ? (GEO.frame.o[String(page)] ?? GEO.frame.d) : [345, 550]); + +/** + * Where this ayah's words sit on the page — outlines only, and the "only" is + * the whole design. + * + * The report drew nothing at all until 2026-08-07, on the argument recorded in + * the design doc §6.4: a tool whose authority is about *identity* should not + * become a second renderer of the mus'haf with a second chance to draw it + * wrong. That argument still holds for **ink** and this function honours it — + * no glyphs, no page raster, no `` from `assets/pages/**`, nothing that + * could be mistaken for the print. What it draws is rectangles from + * `assets/words/**`, which are committed, `gate:words`-checked geometry, and + * which answer the one question the tool kept sending its reader to the app + * for: *which two boxes* does this 2→1 block cover. + * + * The tints are not decorative. A box carries the same even/odd tint its + * codepoints carry on the ④ ruler, keyed on the **host index** rather than the + * print index so that a dropped word is a gap in both places. That is what + * makes the two panels one instrument instead of two pictures: a span lit on + * the ruler is the same span lit on the page. + */ +function outline(key, f, sel) { + const boxes = GEO?.boxes[key]; + if (!GEO) { + return el("p", { class: "note", text: "No word geometry in this report — assets/words/hafs-kfqc/ was not readable when it was generated, so the outline is off. Everything else on this page is unaffected." }); + } + if (!boxes) { + return el("p", { class: "note", text: `No word shard covers ${key}. The boxes ship per page in assets/words/hafs-kfqc/; an ayah with none is either outside the pages this run read (--pages) or a gap in the shards, and the second would be a defect worth filing.` }); + } + + const page = GEO.page[key]; + const [W, H] = frameOf(page); + const marks = new Set(GEO.marks[key] ?? []); + + // The count check, restated in the browser rather than trusted from the + // generator. If the two descriptions of this ayah disagree on how many words + // it has, every label below would be off by an unknown amount — so the + // outline draws the boxes and refuses to number them. + const trusted = boxes.length === f.words.length; + + // print index (1-based) → its index in `hosts`, which is what the ruler tints + // on. Absent for a word the fold dropped; that absence is drawn, not hidden. + const hostOfPrint = new Map(); + f.hosts.forEach((h, i) => { + if (h.print !== null && h.print !== undefined) hostOfPrint.set(h.print, i); + }); + + const lit = new Set( + sel ? touched(f.hosts, sel.start, sel.end).map((i) => f.hosts[i].print).filter((p) => p != null) : [], + ); + + const svg = svgEl("svg", { + viewBox: `0 0 ${W} ${H}`, + class: "outline", + preserveAspectRatio: "xMidYMid meet", + role: "img", + "aria-label": `Word box outlines for ${key} on page ${page}. Geometry only — no text is drawn.`, + }); + svg.append(svgEl("rect", { class: "frame", x: 0.5, y: 0.5, width: W - 1, height: H - 1 })); + + // The rest of the page, faint. Without it the ayah floats in an empty + // rectangle and there is no way to see that it is the third line down. + if (state.context) { + for (const other of PAGE_AYAHS.get(page) ?? []) { + if (other === key) continue; + for (const [x, y, w, h] of GEO.boxes[other] ?? []) { + svg.append(svgEl("rect", { class: "other", x, y, width: w, height: h })); + } + } + } + + boxes.forEach(([x, y, w, h], i) => { + const print = i + 1; + const host = hostOfPrint.get(print); + const isMark = marks.has(print); + const span = f.spans[i]; + const cls = ["box"]; + if (!trusted) cls.push("untrusted"); + else { + cls.push(host === undefined ? "gap" : host % 2 ? "odd" : "even"); + if (isMark) cls.push("mark"); + if (lit.has(print)) cls.push("lit"); + } + const title = trusted + ? `${print}. ${f.words[i].hafs} · ${isMark ? "pause mark" : f.words[i].waw ? "split waw" : "word"} · ${span ? `fold [${span[0]}, ${span[1]})` : "dropped by the fold"}` + : `box ${print} of ${boxes.length} — not numbered, see the warning above`; + svg.append( + svgEl("rect", { + class: cls.join(" "), + x, + y, + width: w, + height: h, + onclick: trusted && span + ? () => { + state.annotation = f.annotations.findIndex((a) => a.start < span[1] && a.end > span[0]); + render(); + } + : null, + }, svgEl("title", { text: title })), + ); + }); + + const wrap = el("div", { class: "outline-wrap" }, svg); + const bits = []; + if (!trusted) { + bits.push(el("p", { class: "warn", text: `${boxes.length} boxes but ${f.words.length} print words. Two independent descriptions of this ayah disagree on how many words it has, so the outline draws the geometry and deliberately does not label it — a guessed alignment here would be a picture that lies quietly. Worth filing.` })); + } + bits.push(wrap); + bits.push(el("p", { class: "note legend" }, [ + ["even", "word"], + ["mark", "pause mark"], + ["gap", "dropped by the fold"], + ["lit", "touched by the selected annotation"], + ["other", "the rest of the page"], + // Each swatch and its label are one inline-flex item, so a wrap breaks + // between pairs rather than stranding a swatch on the line above its word. + ].map(([cls, label]) => el("span", { class: "item" }, [el("span", { class: `sw ${cls}` }), el("span", { text: label })])))); + return el("div", {}, bits); +} + // ------------------------------------------------------------- the ayah view -- function viewAyah() { @@ -298,10 +454,19 @@ function viewAyah() { el("p", { class: "sub", text: `page ${DATA.pages[key]} of the print · ${words.length} print words (${words.filter((w) => w.mark).length} pause marks) · ${annotations.length} tajweed annotations · ${cps.length} codepoints reconstructed${f.prefix ? ` (${f.prefix} of them the prepended basmala)` : ""}` }), ])); - // ① the artwork. Named, never drawn — see the design doc's blind spots. + // ① where the words are. Outlines from the shipped geometry — still no ink. out.append(el("section", {}, [ - el("h3", { text: "① the page artwork" }), - el("p", { class: "note", text: `Page ${DATA.pages[key]}, in apps/web/public/assets/pages/. Anonymous outlined s: no letter, no ligature, no word. This tool deliberately shows none of it — it reconciles encodings, not ink. Word boxes for this ayah ship in assets/words/hafs-kfqc/${DATA.pages[key]}.json.` }), + el("h3", { text: "① where this ayah sits on the page" }), + el("p", { class: "note" }, [ + el("span", { text: `Page ${DATA.pages[key]}, from assets/words/hafs-kfqc/${DATA.pages[key]}.json — the committed word boxes, on the same frame the app draws them on. The artwork itself (assets/pages/, anonymous outlined s with no letter, ligature or word in them) is still deliberately absent: this reconciles encodings, not ink. What the boxes add is the one question the tool used to send you to the app for — ` }), + el("em", { text: "which" }), + el("span", { text: " words a span covers. Click a box, or a row in ② below, to select the annotation over it." }), + ]), + el("div", { class: "toggles" }, el("label", {}, [ + el("input", { type: "checkbox", checked: state.context ? "" : null, onchange: (e) => { state.context = e.target.checked; render(); } }), + el("span", { text: "show the rest of the page" }), + ])), + outline(key, f, sel), ])); // ② the print's word text. diff --git a/packages/etl/scripts/lib/encoding-inspector.css b/packages/etl/scripts/lib/encoding-inspector.css index 40ea191..2144ede 100644 --- a/packages/etl/scripts/lib/encoding-inspector.css +++ b/packages/etl/scripts/lib/encoding-inspector.css @@ -113,6 +113,41 @@ table.names tr.found td { color: var(--ok); font-weight: 600; } .cp b { font-size: 19px; font-weight: 400; line-height: 1.35; } .cp i { font: 9px ui-monospace, Menlo, monospace; font-style: normal; color: var(--dim); direction: ltr; height: 11px; } +/* ① the word outline. Geometry only — there is deliberately no glyph and no + page raster here, so every rule below is about a rectangle. The fill tints + are the same --w-even/--w-odd the codepoint ruler uses, on purpose: a word + tinted one way on the page is tinted the same way on the ruler, which is what + makes the two panels one instrument rather than two pictures. */ +.outline-wrap { + background: var(--panel); border: 1px solid var(--line); border-radius: 8px; + padding: 10px; max-height: 68vh; display: flex; justify-content: center; +} +svg.outline { max-width: 100%; max-height: 66vh; height: auto; } +svg.outline .frame { fill: none; stroke: var(--line); stroke-width: 1; } +svg.outline .other { fill: var(--line); opacity: 0.25; } +svg.outline .box { stroke: var(--dim); stroke-width: 0.4; cursor: pointer; } +svg.outline .box.even { fill: var(--w-even); } +svg.outline .box.odd { fill: var(--w-odd); } +/* Dropped by the fold: present in the print, absent from the string the + annotations are measured against. Hollow, because that is what it is. */ +svg.outline .box.gap { fill: none; stroke-dasharray: 2 2; } +svg.outline .box.mark { fill: none; stroke: var(--accent); stroke-dasharray: 1.5 1.5; } +svg.outline .box.lit { fill: var(--span); stroke: var(--accent); stroke-width: 1; } +svg.outline .box.untrusted { fill: none; stroke: var(--bad); stroke-width: 0.6; cursor: default; } +svg.outline .box:hover { stroke: var(--accent); stroke-width: 1; } +.warn { + color: var(--bad); font-size: 13px; max-width: 78ch; + border-left: 3px solid var(--bad); padding-left: 10px; +} +.legend { display: flex; flex-wrap: wrap; align-items: center; gap: 4px 14px; margin-top: 8px; } +.legend .item { display: inline-flex; align-items: center; gap: 5px; } +.legend .sw { flex: none; width: 14px; height: 10px; border: 1px solid var(--dim); border-radius: 2px; } +.legend .sw.even { background: var(--w-even); } +.legend .sw.mark { border-color: var(--accent); border-style: dashed; } +.legend .sw.gap { border-style: dashed; } +.legend .sw.lit { background: var(--span); border-color: var(--accent); } +.legend .sw.other { background: var(--line); opacity: 0.55; border-color: var(--line); } + .diff { background: var(--panel); border: 1px solid var(--line); border-radius: 8px; padding: 14px 16px; } .verdict { font-size: 14px; } .verdict.ok { color: var(--ok); } diff --git a/packages/etl/scripts/probe-encodings.mjs b/packages/etl/scripts/probe-encodings.mjs index 47fb03b..151afa4 100644 --- a/packages/etl/scripts/probe-encodings.mjs +++ b/packages/etl/scripts/probe-encodings.mjs @@ -56,6 +56,7 @@ import { createHash } from "node:crypto"; import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; +import vm from "node:vm"; import { candidatePage, pin } from "./lib/candidate-pages.mjs"; import { WAQF } from "./lib/mushaf-frame.mjs"; import { EXCEPTIONS, lexicalIndices, openAlignment, qacSkeletons } from "./lib/segmentation.mjs"; @@ -66,6 +67,12 @@ const DATA = join(HERE, "..", "data"); const TAJWEED = join(DATA, "tajweed", "tajweed.hafs.uthmani-pause-sajdah.json"); const META = join(DATA, "meta", "quran-data.xml"); const DEFAULT_OUT = join(HERE, "..", "out", "encoding-inspector.html"); +// The shipped word geometry, read for the outline (§6.4). These are *committed* +// assets under `gate:words`, not the gitignored cache — which is the whole +// reason the outline is affordable and the glyphs are not. +const ASSETS = join(HERE, "..", "..", "..", "apps", "web", "public", "assets"); +const WORD_SHARDS = join(ASSETS, "words", "hafs-kfqc"); +const MANIFEST = join(ASSETS, "manifest.json"); const argOf = (name, fallback) => { const i = process.argv.indexOf(name); @@ -126,6 +133,63 @@ function wordsOf(key) { return idxs.map((i) => m.get(i)); } +/** + * The word boxes, from the committed shards — the outline's only input. + * + * Read separately from everything else above, and the separation is the point. + * The rest of this script reconciles four *encodings*; this reads one + * **geometry**, and it reads it from `apps/web/public/assets/words/**` rather + * than from the corpus, because those shards are committed, gated by + * `gate:words` and re-derivable offline. The tool draws where a word sits; it + * still does not draw the word. See the design doc §6.4 — the blindness that + * ended was about *position*, and the one about ink did not move. + * + * Returns `null` if the assets are not where they should be. The report is + * still worth generating without an outline, so this degrades rather than + * throws, and the client says the section is unavailable instead of drawing + * an empty frame that looks like a page with no words on it. + */ +function readBoxes(keys) { + let manifest; + try { + manifest = JSON.parse(readFileSync(MANIFEST, "utf8")); + } catch { + return null; + } + const want = new Set(keys); + const boxes = {}; + const marks = {}; + const page = {}; + // One pass over the shards of the pages this run actually covers. `lastPage` + // is honoured so `--pages 40` stays a fast subset here too. + for (let p = 1; p <= lastPage; p += 1) { + let shard; + try { + shard = JSON.parse(readFileSync(join(WORD_SHARDS, `${p}.json`), "utf8")); + } catch { + continue; + } + for (const [key, w] of Object.entries(shard.words ?? {})) { + if (!want.has(key)) continue; + // `from` is 1 for every shard in the shipped corpus — an ayah's boxes are + // never split across two of them. Asserted rather than assumed, because + // the outline's word numbering is `boxes[i - 1]` and a non-1 `from` would + // silently shift every label by the offset. + if (w.from !== 1) continue; + boxes[key] = w.boxes; + if (w.marks?.length) marks[key] = w.marks; + page[key] = p; + } + } + const [, , dw, dh] = String(manifest.viewBox).split(" ").map(Number); + const overrides = {}; + for (const [p, vb] of Object.entries(manifest.viewBoxOverrides ?? {})) { + const [, , w, h] = String(vb).split(" ").map(Number); + overrides[p] = [w, h]; + } + return { boxes, marks, page, frame: { d: [dw, dh], o: overrides } }; +} + /** The surah table, for names the report can put in a heading. */ function surahs() { const xml = readFileSync(META, "utf8"); @@ -266,7 +330,38 @@ for (const [key, entry] of Object.entries(ayahs)) { // ---------------------------------------------------------------- the report -- +/** + * The outline's geometry, and the one integrity claim it rests on. + * + * A box list and a word list are two independent descriptions of the same + * ayah — the shards were built by `build-words.mjs` off the corpus's `` + * elements, the word list here off its `data-hafs` attributes. If they + * disagree on *how many*, the outline would draw word 7's box under word 8's + * label and look perfectly fine doing it. So the count is checked per ayah, + * the mismatches are counted here and named in the report, and the client + * refuses to number a mismatched ayah's boxes rather than guessing an + * alignment. Zero is the expected answer; the check exists because a silent + * off-by-one is exactly the defect this repo has already shipped once + * (PLAN 14, and the 47.8% edge corpus before it). + */ +const geometry = readBoxes(Object.keys(ayahs)); +let boxedAyahs = 0; +let countMismatches = 0; +const mismatched = []; +if (geometry) { + for (const [key, entry] of Object.entries(ayahs)) { + const b = geometry.boxes[key]; + if (!b) continue; + boxedAyahs += 1; + if (b.length !== entry.w.length) { + countMismatches += 1; + if (mismatched.length < 8) mismatched.push(`${key} (${b.length} boxes vs ${entry.w.length} words)`); + } + } +} + const payload = { + geometry, meta: { generated: new Date().toISOString().slice(0, 19).replace("T", " ") + "Z", pin: { repo: pin.candidate.repo, commit: pin.candidate.commit }, @@ -297,6 +392,14 @@ const read = (p) => readFileSync(join(HERE, "lib", p), "utf8"); const foldSource = read("tajweed-fold.mjs").replace(/^export /gm, ""); const clientSource = read("encoding-inspector.client.mjs"); +// Both halves are *text* to this script, never imported, so nothing here would +// otherwise notice a syntax error in them — the report would generate, weigh +// its usual megabytes, and open to a blank page with the whole script dead in +// the console. That happened once while the ① outline was being written, and +// costs nothing to make impossible: compile the concatenation the browser will +// actually parse, and fail here instead of there. +new vm.Script(`${foldSource}\n${clientSource}`, { filename: "encoding-inspector (fold + client)" }); + const html = ` @@ -327,6 +430,12 @@ const pct = (n, d) => (d ? `${((n / d) * 100).toFixed(2)}%` : "—"); console.log(`\n ${payload.meta.ayahs}/6236 ayahs · ${printWords} print words · ${annotations} annotations`); console.log(` ${mapped} ayahs carry a print↔QAC map; ${Object.keys(EXCEPTIONS).length} named exceptions`); console.log(` mark disagreements between the shards and WAQF: ${markDisagreements}`); +if (!geometry) { + console.log(" word boxes: none — assets/words/hafs-kfqc/ not readable, the outline is off"); +} else { + console.log(` word boxes: ${boxedAyahs} ayahs outlined · ${countMismatches} box/word count mismatches`); + for (const m of mismatched) console.log(` ${m}`); +} console.log(`\n── the oracle, with all ${ALL_CORRECTIONS.length} corrections on`); console.log(` ${oracleHit}/${oracleN} = ${pct(oracleHit, oracleN)} land on the expected letter`); console.log(` ${oracleN}/${annotations} = ${pct(oracleN, annotations)} of annotations checked`); From 0f7b2972b163b4800aad67510c27f194a6a1a242 Mon Sep 17 00:00:00 2001 From: Omar Eid Date: Fri, 7 Aug 2026 07:41:53 -0500 Subject: [PATCH 2/2] =?UTF-8?q?=E2=91=A3=20goes=20to=20answered,=20and=20t?= =?UTF-8?q?he=20"no=20ink"=20rule=20keeps=20the=20half=20that=20was=20righ?= =?UTF-8?q?t?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `docs/design/encoding-inspector.md` §⑨ ④, `docs/issues.json`, the regenerated `docs/issues.md` and `docs/map.json`, for the outline in 2220f66. §⑥ item 4 gets the superseded-blind-spot treatment rather than an edit in place — "no glyphs, no boxes, no page geometry" becomes "the ink: glyphs and the page raster only", struck through with the record above it — because the useful thing about a limitation that turned out to be half wrong is which half, and deleting the sentence deletes that. `answered` rather than `fixed`, by this repo's own definition of the stronger word and the same reasoning `drift-label-reads-backwards` used. The generation guard catches a syntax error and `gate:words` keeps the shards honest, but nothing in CI would fail if section ① stopped drawing tomorrow: the report is generated, gitignored and never a gate, which is exactly why it is allowed to be this cheap. Claiming a closure with no test behind it would misreport what is protecting the outline. `map.json`'s extend rule inverts from "do not teach it to draw the ink" to where the line actually sits now — outlines from a gated shard in, glyphs and anything out of `assets/pages/**` still out — so the next person adding a font or a raster reads a rule that has already thought about their case. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt --- docs/design/encoding-inspector.md | 73 ++++++++++++++++++++++++------- docs/issues.json | 4 +- docs/issues.md | 8 ++-- docs/map.json | 8 ++-- 4 files changed, 68 insertions(+), 25 deletions(-) diff --git a/docs/design/encoding-inspector.md b/docs/design/encoding-inspector.md index 2e49b11..54cf764 100644 --- a/docs/design/encoding-inspector.md +++ b/docs/design/encoding-inspector.md @@ -142,7 +142,13 @@ Six views, one ayah picker, three checkboxes. **The ayah view** puts all four encodings on one ruler: -1. the page artwork — **named and never drawn** (§6); +1. where the ayah sits on the page — its **word boxes as outlines**, from `assets/words/**`, + on the frame the app draws them on, with the rest of the page behind them as faint + context. Geometry only: no glyph, no ligature, no page raster (§6.4 is still in force — + ⑨ ④ is the record of what moved and what did not). A box carries the same even/odd tint + its codepoints carry on the ruler at (4), so a span lit in one panel is lit in the other; + clicking one selects the annotation over it. If the box count and the `data-hafs` word + count ever disagree, the view draws the geometry and **refuses to number it**; 2. the print's words: `data-hafs`, codepoint count, kind (word / split waw / pause mark), the half-open span each occupies in the fold, and the QAC word it maps to; 3. the print↔QAC map for the ayah: every QAC word, the folded consonant skeleton the aligner @@ -215,11 +221,17 @@ Six things, and the last two are the ones that will eventually tempt somebody. and that is the number to read, not the 100%. 3. **Tanzil's own tokenisation.** The reconstruction is *of the print*, so "two words" here always means two print boxes and is never a claim about how Tanzil would count. -4. **The ink.** No glyphs, no boxes, no page geometry. This is the temptation: the boxes are - right there in `assets/words/**`, and drawing them would make the tool feel complete. It - would also make it a second renderer of the mus'haf, with a second chance to draw it - wrong, in a tool whose entire authority rests on being about *identity* rather than - appearance. The page is named and linked; that is the whole intended relationship. +4. ~~**The ink.** No glyphs, no boxes, no page geometry.~~ **The ink — glyphs and the page + raster only.** *(Half of this limitation was real and is gone; ⑨ ④ is the record.)* The + boxes are drawn now, because the maintainer this section predicted would come asking + "which two" came asking. What is still deliberately absent is every **glyph**: no + `` from `assets/pages/**` is ever rendered, and none ever should be. That is where + the original argument keeps all its force — a page raster would make this a second + renderer of the mus'haf, with a second chance to draw it wrong, in a tool whose entire + authority rests on being about *identity* rather than appearance. A rectangle from a + `gate:words`-checked shard makes no claim about how anything *looks*; it says only where + a word the tool is already reasoning about happens to sit. The distinction is the whole + of what ⑨ ④ decided, and it is worth holding: outlines yes, ink no. 5. **QAC segment granularity.** A print word maps to a QAC *word*. PREFIX/STEM/SUFFIX is not in the alignment and is not shown, because the alignment does not know it. 6. **Any other print, and any other edition of QAC.** Both are pinned. A different pin is a @@ -440,17 +452,48 @@ written, and the argument against it is unchanged. > had no vote when it was first weighed. §8's toggles are what made that re-try one line of > work. A correction is only ever rejected by the instrument that was measuring. -### ④ Whether the report should be able to show a page's boxes · **open** +### ④ Whether the report should be able to show a page's boxes · **answered** -§6.4 says the tool draws no ink, deliberately. The counter-argument is real: a maintainer +§6.4 said the tool draws no ink, deliberately. The counter-argument was real: a maintainer looking at a 2→1 block at word 3 usually wants to see *which two boxes*, and the shards are -right there. - -**What would answer it:** a caller. If someone using the inspector reaches for the app to -answer "which box", the blindness is costing more than it saves and a box outline — geometry -only, no glyphs, no page raster — is a bounded addition. Until then it is speculative work -against a stated principle, and the principle is the reason the tool can be trusted about -identity. Owned by whoever next uses it in anger. +right there. This item asked for a caller before spending the work, on the grounds that the +blindness was a principle rather than an oversight. + +**Answered: the caller came, and the boundary moved by exactly the distance the question +proposed.** Section ① now draws the ayah's word boxes as outlines, from `assets/words/**`, +with the rest of the page behind them faintly. No glyph is drawn, nothing is read from +`assets/pages/**`, and §6.4 keeps its force over the half that mattered — the ink. The line +the principle actually protects is not "draws nothing" but "makes no claim about +appearance"; a rectangle from a `gate:words`-checked shard makes none. + +Three things were settled before any of it was drawn, and each shaped the result: + +- **What the shards can carry.** Every one of the 6,236 ayahs has `from === 1` — an ayah's + boxes are never split across two shards — so print word *i* is `boxes[i - 1]` + unconditionally. That is asserted in `readBoxes`, not assumed, because a non-1 `from` + would shift every label silently. +- **What the tints mean.** A box is tinted on its **host** index, not its print index, so a + word the fold dropped is a gap on the page exactly as it is a gap on the ruler. That is + what makes ① and ④ one instrument rather than two pictures. +- **What happens when the two descriptions disagree.** The box list and the `data-hafs` + word list describe the same ayah independently. If their counts differ, the view draws the + geometry and refuses to number it, with a warning naming both counts — because the failure + it would otherwise produce is word 7's box under word 8's label, which looks entirely + fine. Measured over the full corpus at the time of writing: **6,236 ayahs outlined, 0 + count mismatches.** The check stays anyway; the number is a result, not a guarantee. + +One defect was found by building it, and is worth recording because it was invisible: the +report's client is *text* to `probe-encodings.mjs`, never imported, so a syntax error in it +produced a 4.9 MB report that opened to a blank page with the whole script dead in the +console. `probe-encodings.mjs` now compiles the concatenation the browser will parse +(`new vm.Script`) before writing the file, and fails at generation time instead. + +`answered` and not `fixed`, by this repo's own definition of the stronger word: the guard +above would catch a syntax error and `gate:words` keeps the shards honest, but nothing in CI +would fail if section ① stopped drawing tomorrow — the report is generated, gitignored and +never a gate, which is the whole reason it is allowed to be this cheap. Claiming a closure +there is no test behind would misreport what is actually protecting the outline, the same way +⑤ declined to. ### ⑤ Whether the `probe:tajweed-words` drift label reads backwards · **answered** diff --git a/docs/issues.json b/docs/issues.json index 6a15d18..9a2d4c7 100644 --- a/docs/issues.json +++ b/docs/issues.json @@ -405,10 +405,10 @@ { "id": "inspector-draws-no-ink", "source": { "file": "docs/design/encoding-inspector.md", "item": "④" }, - "status": "open", + "status": "answered", "severity": "question", "owner": "agent", - "note": "The report shows identity and never appearance: no glyphs, no boxes, no page geometry, the page named and linked instead. That blindness is what the tool's authority rests on — drawing the mus'haf would make it a second renderer with a second chance to draw it wrong, inside an instrument whose whole claim is about where the codepoints are. The counter-argument is real and will come from use: a maintainer reading a 2→1 block at word 3 usually wants to see which two boxes, and assets/words/** has them. Waiting on a caller rather than on an argument. If somebody using the inspector reaches for the app to answer 'which box', the blindness costs more than it saves and a box OUTLINE — geometry only, no glyphs, no raster — is the bounded version of the addition." + "note": "ANSWERED 2026-08-07: the caller came, and the boundary moved by exactly the distance this row proposed — outlines yes, ink no. Section ① now draws the ayah's word boxes from assets/words/hafs-kfqc/.json on the same frame the app uses, with the rest of the page behind them faintly and a toggle for it; no glyph is drawn and nothing is read from assets/pages/**. The line §6.4 actually protects turned out not to be 'draws nothing' but 'makes no claim about appearance', and a rectangle out of a gate:words-checked shard makes none. Three things were settled before anything was drawn, and each of them shaped the result. (a) Every one of the 6,236 ayahs has from === 1 — an ayah's boxes are never split across two shards — so print word i is boxes[i - 1] unconditionally; readBoxes ASSERTS that rather than assuming it, because a non-1 from would shift every label silently and look fine. (b) A box is tinted on its HOST index, not its print index, so a word the fold dropped is a gap on the page exactly as it is a gap on the codepoint ruler — that is what makes the two panels one instrument rather than two pictures, and it is why clicking a box selects the annotation over it and lights the ruler in step. (c) When the box list and the data-hafs word list disagree on count the view draws the geometry and REFUSES to number it, warning with both counts, because the failure it would otherwise produce is word 7's box under word 8's label. Measured over the full corpus: 6236 ayahs outlined, 0 count mismatches. The check stays anyway; the number is a result, not a guarantee. One invisible defect was found by building it and is worth carrying: the report's client is TEXT to probe-encodings.mjs, never imported, so a syntax error in it produced a 4.9 MB report that opened to a blank page with the whole script dead in the console — probe-encodings.mjs now compiles the exact concatenation the browser will parse (new vm.Script) before writing, and fails at generation time instead. `answered` and not `fixed` on purpose, by the same reasoning as drift-label-reads-backwards: the guard catches a syntax error and gate:words keeps the shards honest, but nothing in CI would fail if section ① stopped drawing, because the report is generated, gitignored and never a gate. ORIGINAL NOTE FOLLOWS. The report shows identity and never appearance: no glyphs, no boxes, no page geometry, the page named and linked instead. That blindness is what the tool's authority rests on — drawing the mus'haf would make it a second renderer with a second chance to draw it wrong, inside an instrument whose whole claim is about where the codepoints are. The counter-argument is real and will come from use: a maintainer reading a 2→1 block at word 3 usually wants to see which two boxes, and assets/words/** has them. Waiting on a caller rather than on an argument. If somebody using the inspector reaches for the app to answer 'which box', the blindness costs more than it saves and a box OUTLINE — geometry only, no glyphs, no raster — is the bounded version of the addition." }, { "id": "drift-label-reads-backwards", diff --git a/docs/issues.md b/docs/issues.md index 728e229..54c6142 100644 --- a/docs/issues.md +++ b/docs/issues.md @@ -1,5 +1,5 @@ - + # Open items @@ -26,14 +26,13 @@ code, never reproduced. `open` — undecided, nothing blocking. `blocked` — th and unavailable. `answered` — decided, nothing owed in code. `fixed` — closed in code *and* in a test that would fail if it came back; the gate refuses the word without one. -## Open — 23 +## Open — 22 | item | status | severity | owner | waiting on | | --- | --- | --- | --- | --- | | [Arabic number agreement outside `distance`](design/i18n.md#-arabic-number-agreement-outside-distance--open) | open | defect | user | a hafiz | | [Does a real fore-edge stack vary?](design/page-transition.md#-does-a-real-fore-edge-stack-vary--open) | open | question | user | — | | [Whether this document should be generated rather than written](design/etl-pipeline.md#-whether-this-document-should-be-generated-rather-than-written--open) | open | question | agent | — | -| [Whether the report should be able to show a page's boxes](design/encoding-inspector.md#-whether-the-report-should-be-able-to-show-a-pages-boxes--open) | open | question | agent | — | | [The CI frame budget is a number from an emulator](backlog.md#-the-ci-frame-budget-is-a-number-from-an-emulator--blocked) | blocked | risk | agent | perf-verdict-on-device | | [Does the fold read at all?](design/page-transition.md#-does-the-fold-read-at-all--blocked) | blocked | question | user | a hafiz on the acceptance phone | | [The same measurement on real mid-tier Android](backlog.md#-the-same-measurement-on-real-mid-tier-android--blocked) | blocked | risk | user | an Android phone | @@ -54,7 +53,7 @@ test that would fail if it came back; the gate refuses the word without one. | [Two days of real revision produce two days in the record](validation/ledger.json) | pending | risk | user | — | | [VoiceOver / TalkBack gesture walkthrough](validation/ledger.json) | pending | risk | user | — | -## Closed — 56 +## Closed — 57 `answered` owes nothing further. `fixed` owes a test, and names it here. @@ -69,6 +68,7 @@ test that would fail if it came back; the gate refuses the word without one. | [Whether the `probe:tajweed-words` drift label reads backwards](design/encoding-inspector.md#-whether-the-probetajweed-words-drift-label-reads-backwards--answered) | answered | — | | [The four exceptions are orthographic and could be folded away](design/word-indexing.md#-the-four-exceptions-are-orthographic-and-could-be-folded-away--answered) | answered | — | | [The gesture thresholds are inherited, not reopened](design/page-transition.md#-the-gesture-thresholds-are-inherited-not-reopened--answered) | answered | — | +| [Whether the report should be able to show a page's boxes](design/encoding-inspector.md#-whether-the-report-should-be-able-to-show-a-pages-boxes--answered) | answered | — | | [Whether `lam_shamsiyyah`'s 21 `+1` misses are a source defect](design/encoding-inspector.md#-whether-lam_shamsiyyahs-21-1-misses-are-a-source-defect--answered) | answered | — | | [`langSwitchTo(other)` interpolates a locale's name into another locale's sentence](design/i18n.md#-langswitchtoother-interpolates-a-locales-name-into-another-locales-sentence--answered) | answered | — | | [The map assumes no ayah spans a page](design/word-indexing.md#-the-map-assumes-no-ayah-spans-a-page--answered) | answered | — | diff --git a/docs/map.json b/docs/map.json index 6973b98..42c8ea0 100644 --- a/docs/map.json +++ b/docs/map.json @@ -675,7 +675,7 @@ { "file": "docs/design/encoding-inspector.md", "symbol": "Every word in this repo is described four times", - "note": "Read this first, and §① first of all — the print is ink, it has no letters and no words, and every other encoding exists because somebody wanted to say something about it that it could not say about itself. It is the only place that says what the four encodings are, why each PAIR of them disagrees for a different reason, why the tool is a generated file rather than a dev route in the app, and — §6 — the six things it is deliberately blind to. §7 records what the first full pass found and is deliberately kept in its dated state, with a ↳ line under each paragraph the corrected fold has since overtaken — it localised four fold corrections nothing else could have, and it also mis-attributed one finding to the source, which §9 ② now carries as the more useful record of the two." + "note": "Read this first, and §① first of all — the print is ink, it has no letters and no words, and every other encoding exists because somebody wanted to say something about it that it could not say about itself. It is the only place that says what the four encodings are, why each PAIR of them disagrees for a different reason, why the tool is a generated file rather than a dev route in the app, and — §6 — the six things it is deliberately blind to, one of which (§6.4, the ink) has since given up half its ground to §① and says so in place rather than being edited away. §7 records what the first full pass found and is deliberately kept in its dated state, with a ↳ line under each paragraph the corrected fold has since overtaken — it localised four fold corrections nothing else could have, and it also mis-attributed one finding to the source, which §9 ② now carries as the more useful record of the two." }, { "file": "packages/etl/scripts/lib/tajweed-fold.mjs", @@ -685,12 +685,12 @@ { "file": "packages/etl/scripts/probe-encodings.mjs", "symbol": "const foldSource", - "note": "The generator: reads the 378 MB gitignored cache, the vendored tajweed offsets and the committed alignment pin, and writes one self-contained HTML file. `foldSource` is the anti-duplication mechanism made literal — `tajweed-fold.mjs` read as text with `export ` stripped, so there is no second implementation to drift. It re-computes the headline in Node with the same `foldAyah` as a cross-check on the inlining, and cross-checks the shards' pause-mark set against `WAQF` on every run (0 disagreements today, reported as a number so a future one surfaces rather than silently changing a screen). The index bridge — page-global print indices to the corpus's 1-based-per-ayah — is commented as the one place the two numbering schemes meet." + "note": "The generator: reads the 378 MB gitignored cache, the vendored tajweed offsets and the committed alignment pin, and writes one self-contained HTML file. `foldSource` is the anti-duplication mechanism made literal — `tajweed-fold.mjs` read as text with `export ` stripped, so there is no second implementation to drift. It re-computes the headline in Node with the same `foldAyah` as a cross-check on the inlining, and cross-checks the shards' pause-mark set against `WAQF` on every run (0 disagreements today, reported as a number so a future one surfaces rather than silently changing a screen). The index bridge — page-global print indices to the corpus's 1-based-per-ayah — is commented as the one place the two numbering schemes meet. `readBoxes` is the second reader and the smaller one: it takes the COMMITTED `assets/words/hafs-kfqc/.json` shards plus `manifest.viewBox`/`viewBoxOverrides`, not the cache, which is exactly why §① can afford geometry — and it asserts `from === 1` rather than assuming it, because a split shard would put word 7's box under word 8's label and look entirely fine. Last in the file, before anything is written: `new vm.Script(foldSource + clientSource)`. Both halves are text to this script and are never imported, so nothing else would notice a syntax error in them — the report generates, weighs its usual megabytes and opens blank with the whole client dead in the console. That happened once; compiling the exact concatenation the browser will parse makes it a generation-time failure instead." }, { "file": "packages/etl/scripts/lib/encoding-inspector.client.mjs", "symbol": "function measure", - "note": "The browser half, and it never runs in Node — eslint.config.js gives `*.client.mjs` browser globals only, the same situation as apps/web/perf. `measure` is why the corrections are live checkboxes rather than a legend: one pass over all 6,236 ayahs rebuilding every aggregate in ~30 ms, so the number under a toggle is measured and not remembered. Precomputing per-combination tables was rejected as 2ⁿ in the corrections and silently stale on the day another is added — four were added within the day and a fifth followed, which settles it. `WORD_AYAHS` is the corpus baseline that makes the drift-onset table an instrument instead of a word-frequency list — without a denominator «وَ» tops it for no reason but being «وَ»." + "note": "The browser half, and it never runs in Node — eslint.config.js gives `*.client.mjs` browser globals only, the same situation as apps/web/perf. `measure` is why the corrections are live checkboxes rather than a legend: one pass over all 6,236 ayahs rebuilding every aggregate in ~30 ms, so the number under a toggle is measured and not remembered. Precomputing per-combination tables was rejected as 2ⁿ in the corrections and silently stale on the day another is added — four were added within the day and a fifth followed, which settles it. `WORD_AYAHS` is the corpus baseline that makes the drift-onset table an instrument instead of a word-frequency list — without a denominator «وَ» tops it for no reason but being «وَ». `outline(key, f, sel)` is §①, the one place this file draws geometry: a `` on the page's own frame with one `` per shipped word box, tinted by HOST index rather than print index so a word the fold dropped gaps on the page exactly as it gaps on the ruler — that shared parity is what makes the two panels one instrument. It refuses to number the boxes when their count disagrees with the `data-hafs` word count, drawing the geometry with a warning naming both, and clicking a box selects the annotation over it." }, { "file": "packages/etl/scripts/lib/tajweed-fold.test.mjs", @@ -708,7 +708,7 @@ "Do not re-implement the fold. If the inspector and `probe-tajweed-words.mjs` need to agree — and they do, that agreement is the verification of both — the arithmetic goes in `lib/tajweed-fold.mjs` and both import it. Keep that module import-free: the generator pastes its source verbatim, and one `import` line makes the inlining unsound.", "The report is never committed and never fetched. If a finding is worth keeping, it goes in prose in the design doc or as a measured number in a pin — the way `tajweed-words.probe.json` keeps the probe's verdict — not as an HTML artifact somebody has to trust the age of.", "It works at the SOURCE rule grain, all eighteen, and does not import `build-tajweed.mjs`'s seven-family table. A family is a rendering decision; collapsing here would hide the case worth seeing, two rules of one family disagreeing about the same word. It would also mean importing a module whose body runs a build.", - "Do not teach it to draw the ink. The blindness in §6 is what the instrument's authority rests on — a second renderer of the mus'haf is a second chance to draw it wrong inside a tool whose only claim is about identity. If a caller genuinely needs geometry, a box OUTLINE is the bounded version; glyphs and rasters are not.", + "Do not teach it to draw the ink. The line is now drawn where §6.4 and issues.json's `inspector-draws-no-ink` put it after the caller arrived: OUTLINES from `assets/words/**` are in, because a rectangle out of a `gate:words`-checked shard makes no claim about appearance; glyphs and anything read out of `assets/pages/**` are out, because a second renderer of the mus'haf is a second chance to draw it wrong inside a tool whose only claim is about identity. Adding a font, a `` from the artwork or a raster crosses that line even though outlines did not.", "A correction added to `CORRECTIONS` must be earned by a re-run against the oracle, not argued for, and it must apply to all 6,236 ayahs rather than to the ones that motivated it. The oracle is the referee precisely because a wrong correction makes it worse rather than quietly better, and an over-broad one is punished the same way: stripping every `U+0640` instead of the two carriers `tatweel-carrier` names scores 94.48%, below applying nothing at all. That is the whole reason the eight existing ones are trustworthy. The corollary, learned from `sakta-seen`: a correction that was measured and rejected is rejected BY AN INSTRUMENT, not for all time. Re-run the rejects whenever the oracle widens.", "Before filing a finding as an upstream source defect, switch corrections off and re-measure. A drift of exactly one letter reads as an off-by-one in whoever wrote the span — the most available explanation, and the one that points away from your own code. `lam-shamsiyyah-starts-on-the-wasl` in docs/issues.json is that mistake made and caught: 21 annotations across 19 ayahs on the three-correction fold, ONE on the seven." ],