Skip to content
Merged
20 changes: 18 additions & 2 deletions docs/design/encoding-inspector.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,11 @@ Six views, one ayah picker, three checkboxes.
⑨ ④ is the record of what moved and what did not). A box carries the same even/odd tint
its codepoints carry on the ruler at (4), so a span lit in one panel is lit in the other;
clicking one selects the annotation over it. If the box count and the `data-hafs` word
count ever disagree, the view draws the geometry and **refuses to number it**;
count ever disagree, the view draws the geometry and **refuses to number it**. With
`--marks`, each word's **named marks** are drawn as hairline rectangles inside it — the
name and the codepoint index on hover, and the selected annotation's own mark lit inside
its lit word. That is the level below a word, and
[`sub-word-marks.md`](sub-word-marks.md) §⑧ ① is where the arithmetic behind it lives;
2. the print's words: `data-hafs`, codepoint count, kind (word / split waw / pause mark),
the half-open span each occupies in the fold, and the QAC word it maps to;
3. the print↔QAC map for the ayah: every QAC word, the folded consonant skeleton the aligner
Expand Down Expand Up @@ -231,7 +235,11 @@ Six things, and the last two are the ones that will eventually tempt somebody.
authority rests on being about *identity* rather than appearance. A rectangle from a
`gate:words`-checked shard makes no claim about how anything *looks*; it says only where
a word the tool is already reasoning about happens to sit. The distinction is the whole
of what ⑨ ④ decided, and it is worth holding: outlines yes, ink no.
of what ⑨ ④ decided, and it is worth holding: outlines yes, ink no. `--marks` goes a
level finer and stays on the same side of that line — a mark's rectangle is geometry from
`lib/diacritics.mjs` with a name from a measured dictionary, and it still draws no stroke
of the print. The temptation the rule anticipates gets stronger here, because a mark's box
is small enough that filling it in would *look* like the mark. It must not.
5. **QAC segment granularity.** A print word maps to a QAC *word*. PREFIX/STEM/SUFFIX is not
in the alignment and is not shown, because the alignment does not know it.
6. **Any other print, and any other edition of QAC.** Both are pinned. A different pin is a
Expand Down Expand Up @@ -327,10 +335,18 @@ count so a future disagreement surfaces as a number rather than as a wrong scree
pnpm probe:encodings # from the cache, all 604 pages, ~90s
pnpm probe:encodings --fetch # fill the cache first
pnpm probe:encodings --pages 30 # a fast subset while changing the client
pnpm probe:encodings --marks # + the level below a word
pnpm probe:encodings --out /tmp/x.html
open packages/etl/out/encoding-inspector.html
```

`--marks` is opt-in rather than the default for one reason, and it is a size: it puts 326,515
more rectangles in the payload and takes the report from 5.0 MB to **13.8 MB**. The extraction
itself is cheap — `readDiacritics` bboxes only the mark paths, so the whole corpus costs
about a second on top of a run that already reads all 604 pages. What is expensive is the
page a browser then has to hold. Most questions this tool is opened for are about the four
encodings and do not need it.

`packages/etl/out/` is gitignored, and that is load-bearing rather than tidy: **there is no
Quran text in this repo and there will not be.** The report is full of Arabic; every
codepoint of it is derived at runtime from the gitignored cache, and committing one would
Expand Down
231 changes: 218 additions & 13 deletions docs/design/sub-word-marks.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/issues.json
Original file line number Diff line number Diff line change
Expand Up @@ -464,8 +464,8 @@
"status": "open",
"severity": "question",
"owner": "agent",
"blockedBy": ["the encoding inspector drawing the marks"],
"note": "Opened 2026-08-07 with mark-A. Two measurements now exist over the same words and nothing has checked whether they meet. word-indexing.md ⑪ ⑤ lands 59,975 of 60,057 tajweed annotations (99.86%) on the letter their rule names, 83.31% of them inside a single print word; sub-word-marks.md ⑤ puts 326,515 named mark boxes inside those same words, 0 of them outside. The open question is whether a span's [start,end) coincides with a mark a reader can be SHOWN — whether madd_246 opening at a codepoint means there is a maddah box there — and, if so, whether highlighting the mark is a truer rendering of the rule than washing the whole word. What would answer it: the encoding inspector (mark-B), which already reconciles the print, the ligature corpus, QAC and the tajweed offsets on one screen and would gain the mark boxes over the same frame. Nothing offline can: the correspondence is between a codepoint in a text word-indexing.md ⑪ ⑤ RECONSTRUCTS and an outline on a page, and only an eye closes that gap. What must not happen instead is deriving the correspondence from the fact that both numbers exist — reading a mapping off where the offsets happen to land and then declaring that they land there is the circularity ⑪ ⑤ names about its own oracle, and it passes on a broken answer. Note this is a rendering question and not a recitation one: DIACRITICS records what the corpus wrote in an attribute and this repo asserts nothing about how any of the twenty-six are pronounced."
"blockedBy": ["a hafiz looking at the drawn marks"],
"note": "Narrowed again 2026-08-07 by mark-B, and the offline half is now ANSWERED — the remaining half needs an eye and nothing else. `pnpm probe:encodings --marks` walks all 60,057 annotations from a Tanzil offset down to a drawn path through lib/mark-join.mjs (the same join probe:diacritics ④/⑤ measured, extracted so the inspector and the probe cannot drift), and the answer is a SPLIT rather than a rate: 47.51% open on a codepoint the print draws a named path for — the rectangle to light — and 51.52% open on a base letter with no path above it, which is the CORRECT answer and not a shortfall, because ten of the eighteen rules name a consonant (qalqalah → ق, lam_shamsiyyah → ل, ghunnah → ن/م) and the print draws a consonant as a letter outline. The eight rules that name a mark reach one almost always: hamzat_wasl → wasla 98.11%, madd_2 98.58%, iqlab 99.82%. Nothing is unreachable for a structural reason — no-host, no-word and unjoined are all 0 — and the remainder is explained rather than dropped: 497 respelt words, whose offsets do not address data-hafs by construction, and 82 oracle misses. So a mark-granular highlight is available for the rules about marks and not for the rules about letters, and any UI built on this has to SAY WHICH IT IS DOING; that is a constraint mark-C inherits and it was not visible before the walk. What is left is what arithmetic cannot reach: whether the box the dictionary names sits where a reader LOOKS for that mark, and whether lighting it reads as a truer rendering of the rule than washing the word. The inspector now draws the boxes with the selected annotation's own mark lit inside its word, so the screen those two questions need exists — sub-word-marks.md §⑧ ① carries the table. Opened 2026-08-07 with mark-A. Two measurements now exist over the same words and nothing has checked whether they meet. word-indexing.md ⑪ ⑤ lands 59,975 of 60,057 tajweed annotations (99.86%) on the letter their rule names, 83.31% of them inside a single print word; sub-word-marks.md ⑤ puts 326,515 named mark boxes inside those same words, 0 of them outside. The open question is whether a span's [start,end) coincides with a mark a reader can be SHOWN — whether madd_246 opening at a codepoint means there is a maddah box there — and, if so, whether highlighting the mark is a truer rendering of the rule than washing the whole word. Narrowed 2026-08-07 by sub-word-marks.md ⑤: with the codepoint→name dictionary pinned (34 of 34 tokens, by elimination, validated on 62,931 held-out runs), the first half — does madd_246 open at a codepoint the print draws a maddah for — is now an offline measurement, and should be made before anyone looks at a screen because it is cheap and it bounds what the looking is for. What ⑤ did NOT close, and what still needs the encoding inspector (mark-B), which already reconciles the print, the ligature corpus, QAC and the tajweed offsets on one screen and would gain the mark boxes over the same frame: whether the box the dictionary NAMES sits where a reader looks for that mark, and whether lighting it reads as a truer rendering of the rule than washing the word. The first is a claim about the picture — every step of ⑤ is a correspondence between a reconstructed text and the corpus's own attributes, and none of it looks at where the outline sits relative to the letter that wrote it — and the second was never a measurement at all. What must not happen instead is deriving the correspondence from the fact that both numbers exist — reading a mapping off where the offsets happen to land and then declaring that they land there is the circularity ⑪ ⑤ names about its own oracle, and it passes on a broken answer. Note this is a rendering question and not a recitation one: DIACRITICS records what the corpus wrote in an attribute and this repo asserts nothing about how any of the twenty-six are pronounced."
},

{
Expand Down
4 changes: 2 additions & 2 deletions docs/issues.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<!-- generated by scripts/build-issues-doc.mjs — do not edit; run `make issues-doc` -->
<!-- issues-hash: 2fd80b6fe3b4 -->
<!-- issues-hash: 993130398fa5 -->

# Open items

Expand Down Expand Up @@ -38,7 +38,7 @@ test that would fail if it came back; the gate refuses the word without one.
| [Does a real fore-edge stack vary?](design/page-transition.md#-does-a-real-fore-edge-stack-vary--open) | open | question | user | — |
| [Whether this document should be generated rather than written](design/etl-pipeline.md#-whether-this-document-should-be-generated-rather-than-written--open) | open | question | agent | — |
| [Whether the GPL/App-Store reading in ①–③ is right](design/track-b-native.md#-whether-the-gplapp-store-reading-in--is-right--open) | open | risk | user | a licensing opinion |
| [Does a tajweed span land on a mark a reader can be shown](design/sub-word-marks.md#-does-a-tajweed-span-land-on-a-mark-a-reader-can-be-shown--open) | open | question | agent | the encoding inspector drawing the marks |
| [Does a tajweed span land on a mark a reader can be shown](design/sub-word-marks.md#-does-a-tajweed-span-land-on-a-mark-a-reader-can-be-shown--open) | open | question | agent | a hafiz looking at the drawn marks |
| [`PLAN.md` states an unachievable order and one wrong citation](design/track-b-native.md#-planmd-states-an-unachievable-order-and-one-wrong-citation--open) | open | defect | agent | gpl-and-the-app-store |
| [Whether Track B should exist at all after ④ and ⑤](design/track-b-native.md#-whether-track-b-should-exist-at-all-after--and---open) | open | question | user | web v1.0 and somebody using it |
| [The CI frame budget is a number from an emulator](backlog.md#-the-ci-frame-budget-is-a-number-from-an-emulator--blocked) | blocked | risk | agent | perf-verdict-on-device |
Expand Down
22 changes: 21 additions & 1 deletion docs/map.json
Original file line number Diff line number Diff line change
Expand Up @@ -624,7 +624,22 @@
{
"file": "packages/etl/scripts/probe-diacritics.mjs",
"symbol": "const SLACK",
"note": "The measurement that had to come before anything shipped, over all 604 cached pages: the vocabulary is complete (26 of 26 names drawn), every one of 326,515 marks lands inside the word box the app already ships (0 escapes, 0 unmatched, slack 0.2 — rounding, not registration), and the shard tree would weigh 7.35 MB raw / 2.28 MB gz. Containment is checked against the *committed* shards rather than boxes computed in the same pass, so the two sides cannot share a mistake and agree about it. It ships nothing on purpose: 2.28 MB of assets no caller fetches is the same waste `gate:assets` names for a non-vendored edition. What it cannot answer is whether a mark is on the right *letter* — that needs an eye, and belongs in the encoding inspector."
"note": "The measurement that had to come before anything shipped, over all 604 cached pages: the vocabulary is complete (26 of 26 names drawn), every one of 326,515 marks lands inside the word box the app already ships (0 escapes, 0 unmatched, slack 0.2 — rounding, not registration), and the shard tree would weigh 7.35 MB raw / 2.28 MB gz. Containment is checked against the *committed* shards rather than boxes computed in the same pass, so the two sides cannot share a mistake and agree about it. It ships nothing on purpose: 2.28 MB of assets no caller fetches is the same waste `gate:assets` names for a non-vendored edition. Five questions, and the order between them is the argument: ④ establishes that the counts work out per ligature, and only then does ⑤ establish which path is which. What none of them can answer is whether the ink lands where a reader's eye goes — that needs an eye, and belongs in the encoding inspector."
},
{
"file": "packages/etl/scripts/probe-diacritics.mjs",
"symbol": "function supported(want, got)",
"note": "⑤'s engine, and the reason the codepoint→name dictionary is evidence rather than an assumption. Arc consistency over bipartite matchings: a run contributes a *bag* of codepoint tokens beside a *bag* of drawn names, and a pairing is deleted only when no perfect one-to-one assignment of that run can use it. Position is never consulted — pairing them off left to right would assume the print draws marks in written order, which is the thing in question, and 1.36% of multi-mark runs say otherwise. Plain set intersection is the wrong operator and was tried first: it presumes the relation is already a function, and drove `U+0653` to an empty candidate set, which is how the `أ`-carrier madda convention was found. Every single-mark run is held out, because a one-mark run forces its own pairing and scoring against it would report 100% by construction. Result: 34 of 34 tokens pinned in two passes, 62,931 of 62,931 held-out runs predicted correctly."
},
{
"file": "packages/etl/scripts/lib/mark-join.mjs",
"symbol": "export function markPaths",
"note": "④ and ⑤'s arithmetic extracted so there is exactly one of it, and the only entry point a caller should want: a word as `readDiacritics` returns it → `[{ at, len, token, name, mark }]`, where `at` is a **codepoint** index into that word's own `data-hafs`. That index is the whole point — tajweed offsets count codepoints, so this is the bridge from an offset to a rectangle. Three steps behind it: `letters` folds `\\p{Mn}` *and* `\\p{Lm}` onto the letter before (the tatweel and the small waw/yeh are text letters the print draws as marks, which is what made ④'s first draft off by one for every seated hamza), `align` matches ligatures to letter runs by CONTENT with backtracking (order is not reading order — «ٱلرَّحِيمِ» draws `[لر|حيم|ٱ]`, and a left-to-right walk agreed on every length while misassigning every mark), and `pairMarks` pairs a wanted token to a drawn name through the frozen 34-entry `DRAWN_NAME`, breaking same-name ties right to left. Its refusals are the load-bearing part: it returns `null` — never a partial answer — for a mark count that disagrees, a name the tokens did not ask for, or letters no assignment of ligatures can cover, because a partial answer would look exactly like an answer. Extracted rather than left in the probe for the reason `tajweed-fold.mjs` gives about itself: `probe-encodings.mjs --marks` draws these rectangles, and if its join were a second implementation, a clean screen would stop being evidence about the probe."
},
{
"file": "packages/etl/scripts/lib/mark-join.test.mjs",
"symbol": "describe(\"markPaths\"",
"note": "28 tests on words small enough to count by hand, and most of them assert that `markPaths` returns `null`. The probe is the measurement — 86,962 words joined, 34 tokens pinned — and it needs 380 MB of gitignored cache to say anything, which is why it is a probe; what a test can hold is the arithmetic and the refusals. Marks are written as `[diacriticId(name), x, y, w, h]` rather than pasted ids, so reordering `DIACRITICS` fails here instead of silently re-labelling what this file claims. One test earned its comment the hard way: `مَـَٔ` yields three tokens, not two — the tatweel and the sajda overline get no path, but the hamza-above on the tatweel does."
}
],
"extend": [
Expand Down Expand Up @@ -717,6 +732,11 @@
"symbol": "describe(\"driftOnset\"",
"note": "54 tests, and they are what makes the shared module safe to share. They pin the sign convention both probes now read, the eighteen-entry ORACLE (its size, that every entry carries the tajweed reason for its letter set, that any letter of a set counts, and that `near` is a one-position window rather than a slop allowance — `iqlab`'s meem is written high over a fatha and LOW over a kasra, and missing the low form scored 85.05%), `oracleDensity`'s pricing of a hit, each of the eight corrections' effect in isolation — including a `respellerFor` block that checks each `respell` fires on its own shape and on nothing else, so deleting one fails a test rather than quietly moving an aggregate — and `driftOnset`'s refusals — it does not narrow with a hit AFTER the miss, and it reports `bounded: false` rather than pretending to a left edge it does not have. Vitest globals are imported explicitly here: the eslint test-globals block covers `**/*.test.{ts,tsx}` and this is `.mjs`."
},
{
"file": "packages/etl/scripts/probe-encodings.mjs",
"symbol": "function markLevel(key, entry, cps, hosts)",
"note": "`--marks`, the level below a word, opt-in because it is a size: 326,515 more rectangles takes the report from 5.0 MB to 13.8 MB. It walks every one of the 60,057 tajweed annotations from its codepoint offset to the path the print drew — oracle hit, host word, respell check, then `markPaths` — and counts the outcome into one of eight named classes rather than a rate, because `letter` is an ANSWER and not a shortfall: ten of the eighteen rules name a consonant, and the print draws a consonant as a letter outline, not a mark. The measured split is 47.51% reaching a named path against 51.52% landing on a base letter, with 0 unreachable for any structural reason (`no-host`, `no-word`, `unjoined` all zero) and the remainder explained rather than dropped — 497 respelt words, whose offsets do not address `data-hafs` by construction, and 82 oracle misses. `docs/design/sub-word-marks.md` §⑧ ① carries the table and the argument that this is a split rather than a rate, which is the constraint any mark-granular UI inherits. The predicted-vs-observed name tally it prints is a smoke alarm, not evidence: `pairMarks` pairs BY name, so the two can only disagree where a composite spans more than one codepoint. On the client side the rectangles go in `outline()` beside the word boxes, hollow and hairline — the ink rule holds a level finer, and it is under more pressure there, because a mark's box is small enough that filling it in would read as the mark itself."
},
{
"file": ".gitignore",
"symbol": "packages/etl/out/",
Expand Down
Loading
Loading