The ligature is the join, the marks say their names by elimination, and an offset walks down to one - #86
Merged
Merged
Conversation
`readDiacritics` returns the ligature structure beside the flat mark list, and
`probe:diacritics` gains ④ to measure whether it holds.
Containment (②) files a mark under the right word. It says nothing about which
letter, and a tajweed rule is a [start, end) over codepoints — so without a
letter-level join a rule can be highlighted no finer than the whole word. The
corpus offers exactly one join: <g id="md-ligature-…"> names the letters it
draws in data-text and nests the marks drawn on them.
④ partitions data-hafs into the letters the print outlines, walks the ligatures
across that partition, and compares mark counts. Every failure is bucketed by
cause, because the check is layered and quoting the per-ligature agreement alone
would silently condition it on a filter the reader cannot see.
All 604 pages: of 91,451 entries the print calls words, 4,486 draw no letters at
all (the pause marks, ۩, ۞). Of the remaining 86,965, 86,880 join cleanly —
99.90%.
Getting there needed four print conventions read off the markup, not assumed;
each one is why an earlier draft of this read 97.75%:
- a bare hamza ء is an outline like any letter, not a named mark
- \p{Lm} folds: the tatweel is a tooth drawn into its neighbour, while the
small waw ۥ and small yeh ۦ are named marks despite Unicode calling them
letters
- a vowel then an iqlab meem ۭ or ۢ is one composite glyph, `kasra iqlab`
- a seated hamza and ٱ are a base outline plus their own named path
The remaining 85 entries are left alone on purpose. Each rule above exists
because the markup showed the print doing something; adding rules until the
number reads 100% would fit the rule to the data and make ④ agree with the
corpus by construction, which is the one property that would stop it being
evidence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
§⑤ records ④'s corpus measurement: the rejected positional attempt (88.79% and a visibly wrong pairing tail), the markup that replaced it, the four print conventions that had to be learned, the 99.90% result, and the four families the 85-entry residual falls into — each with the word that demonstrates it. §⑦ said the letter question was unanswerable because the corpus expresses letter order "as ligature ids this does not read". It reads them now, so that reason is gone and the section would have been quietly wrong. The conclusion survives on a better reason: counts are necessary and not sufficient. A ligature agreeing on three marks does not establish the second is over the second letter, and a word whose marks were internally permuted would pass ④ exactly as a correct one does. Only an eye closes that, which is still mark-B. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
…loses ④ was at 99.90%. The user asked for 100%, and the route was the one that got it from 88.79% to 97.75% to 99.90%: dump the markup for each residual family and encode what the print actually does. No rule here was added to move the number — every one of them names the word that demonstrated it. Three families were left, and reading them turned up two errors of my own: - `align` replaces the left-to-right walk. Ligature document order is not reading order — «ٱلرَّحِيمِ» is drawn `[لر|حيم|ٱ]` — and a letter can be drawn twice, «فَلَا» as `[فلا|ا]`. The old length check let both pass and then misassigned every mark while the totals balanced. Matching on content is strictly stronger, and some words that used to pass now fail. - `FOLD`/`FOLDS` are one class used on both sides. A ligature's `data-text` carries the tatweel that `letters` folds away, so «مَـَٔابٗا» → `[مـا|با]` looked unassignable when it is simply spelt with its tooth. - The seated hamza always gets its path. A first reading of «أَيۡدِيهِمۡ» suggested the ligature's own spelling decided it; «أَنَّ» is drawn `[أ|ن]` and still carries `hamza` then `fatha`. That conditional cost 151 words. 86,962 of 86,965 lettered words now join cleanly. The three left are the corpus disagreeing with itself, and the docblock names them rather than absorbing them, because a rule for either would be a rule for one word: «أَيۡدِيهِمۡ» loses its hamza path in 2 of its 26 occurrences, and 17:7's «لِيَسُـُٔواْ» draws a `small waw` and a `maddah` its own `data-hafs` does not write. Neither costs anything downstream: ② decides whether the geometry ships, and ② is exact — 0 marks outside their word, unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
§⑤'s table and its four-family residual list were written against 99.90% and are now wrong in both directions — the number is 100.00% of lettered words and the residual is three entries, not eighty-five. The conventions table gains the two rows that closed it (the sajda overline, the split and out-of-order ligatures) and corrects the seated-hamza row to say **always**. §⑦ said counts are necessary and not sufficient and left it abstract. It now carries the probe's own evidence against itself: the `codepoint → name` tally is built only from ligatures whose counts agree and still contains 611 pairings of a sukun with a `hamza` path. «بِٱلۡأٓخِرَةِ» is one — the run `لأ` is written `ۡ أ ٓ` and drawn `hamza, sukun, fatha`. Three wanted, three drawn, ④ passes, and all three pairings are wrong. Which is the argument for mark-B in one example: the tally's head is trustworthy, its tail is not, and no arithmetic tells them apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
probe-diacritics grows a fifth question. ④ established that a ligature drawing three letters carries three marks; it never said which drawn path was the tanween and which the sukun, and a tajweed rule that wants to light the tanween needs exactly that. The obvious answer — pair them off left to right — is not an answer. It assumes the print draws marks in the order the text writes them, which is the thing in question, and 1.36% of multi-mark runs say otherwise. The old `pairs` tally did exactly that, and its output was cited in the design doc as proof the gap could not be closed. It was measuring its own assumption. Deleted. ⑤ never looks at position. Each agreeing ligature contributes a bag of codepoint tokens beside a bag of drawn names, and the correspondence falls out of elimination across the corpus: arc consistency over bipartite matchings, where a pairing dies only when no perfect assignment of that run can use it. Set intersection was tried first and is the wrong operator — it presumes the relation is already a function, and drove U+0653 to an empty candidate set. That empty set turned out to be the finding: the print draws a combining madda as `maddah` on nineteen carriers and as `fatha` on a hamza-carrying alef, 277 times of 277, so the codepoint alone is not a function and needs the carrier. `U+0653@hamza` exists for that reason only. Every single-mark run is held out. A one-mark run forces its own pairing, so scoring against it would report 100% by construction — an earlier version of this did, and it was circular. 152,101 runs, 62,931 held out, 2,869 shapes to learn from 34 of 34 tokens pinned in two passes, 0 shapes unsatisfiable 62,931 of 62,931 held-out runs predicted correctly (100.00%) Order is measured only afterwards, once pairing is settled without it: 98.64% drawn as written, 99.56% with R1 (a seated hamza's sign drawn last), stated before it was scored. A second candidate — shadda after its vowel — was put up the same way and refuted by its own score at −40 runs. Dropped. Also fixes the usage line, which named a `--filter @hifth/etl` script that does not exist; and excludes markless ligatures from the run set, where they agreed vacuously and diluted the held-out percentage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
sub-word-marks.md §⑦ argued no arithmetic could say which drawn path belongs to which codepoint, and offered as proof that the probe's own tally carried 611 pairings of a sukun with a path named `hamza`. Those 611 were an artifact of how that tally was built — it zipped a run's codepoints against its paths by position, so every word where the print reorders was miscounted by construction. The section is rewritten rather than deleted, because the refutation is worth keeping: a check that shares a mistake with the thing it checks will agree with it. What survives §⑦ is smaller and real. ⑤ closes identity — this path is the sukun and not the hamza — and every step of it is a correspondence between a reconstructed text and the corpus's own attributes. None of it looks at the picture. A print that named its paths correctly and placed one a letter to the left would satisfy ①–⑤ exactly as a correct one does. That is mark-B's check, and it is why mark-B is a separate step rather than a review of mark-C. §⑤ gains the propagation: 34 of 34 tokens pinned, 62,931 held-out runs predicted at 100.00%, and order measured only afterwards — 98.64% as written, 99.56% with R1, with the refuted shadda rule recorded as refuted. The conventions table goes from six to eight. The two new rows are one convention stated as two: a combining madda draws as `maddah` on nineteen carriers and as `fatha` on a hamza-carrying alef, 277 of 277 — the only place in the corpus where a codepoint's drawn name depends on the letter under it. The geometry leans toward it being the madda drawn short (0.89× a same-line fatha, against the maddah's one constant width), and nothing downstream needs the answer. issues.json ① is narrowed, not closed: half of "does a tajweed span land on a mark" is now an offline measurement and should be made before anyone looks at a screen. The half that needs an eye is stated as two questions instead of one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
… to land
`lib/mark-join.mjs` extracts ④ and ⑤'s arithmetic out of `probe-diacritics.mjs`
so there is exactly one of it: `letters` folds `\p{Mn}` and `\p{Lm}` onto the
letter before, `align` matches ligatures to letter runs by content with
backtracking, and `pairMarks` pairs a wanted token to a drawn name through the
frozen 34-entry `DRAWN_NAME`. `markPaths` is the one entry point a caller
should want — a word in, `[{ at, len, token, name, mark }]` out, with `at` a
codepoint index into that word's own `data-hafs`, which is the bridge a tajweed
offset needs.
It refuses rather than approximates. `null` for a mark count that disagrees, a
name the tokens did not ask for, or letters no assignment of ligatures can
cover — because a partial answer here would look exactly like an answer, which
is the defect ④'s left-to-right walk already shipped once while every length
balanced. `mark-join.test.mjs` is 28 tests on words small enough to count by
hand and most of them assert one of those refusals.
`probe-encodings.mjs --marks` is the caller. It walks all 60,057 tajweed
annotations from a Tanzil offset down to a drawn path and counts the outcome
into eight named classes rather than a rate:
drawn 28535 47.51% a named path — the rectangle to light
letter 30943 51.52% a base letter, with nothing above it
respelt 497 0.83% the offset does not address `data-hafs`
oracle-miss 82 0.14% no position to resolve in the first place
no-host / no-word / unjoined / basmala 0 0.00%
The 51.52% is an answer. Ten of the eighteen rules name a consonant — qalqalah
on ق, lam_shamsiyyah on ل, ghunnah on ن or م — and the print draws a consonant
as a letter outline. The eight that name a mark reach one almost always:
hamzat_wasl → wasla 98.11%, madd_2 98.58%, iqlab 99.82%. So the finding is a
split rather than a rate, and any mark-granular UI has to say which of the two
it is doing.
The inspector draws the boxes with the selected annotation's own mark lit
inside its word. Hollow and hairline: the outlines-yes-ink-no rule holds a
level finer, and it is under more pressure there, because a mark's box is small
enough that filling it in would read as the mark itself.
`--marks` stays opt-in for one reason and it is a size: 326,515 more rectangles
takes the report from 5.0 MB to 13.8 MB.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
…er half needs an eye `sub-word-marks.md` §⑧ ① keeps its **open** status and gains the measurement that halves it. The offline half is answered: all 60,057 tajweed annotations walked from a Tanzil offset to a drawn path, 47.51% reaching a named rectangle and 51.52% landing on a base letter, with nothing unreachable for a structural reason and the remainder — 497 respelt words, 82 oracle misses — named rather than dropped. The section argues why the 51.52% is an answer instead of a shortfall, because that is the part a reader would otherwise take for a coverage number and try to raise: ten of the eighteen rules name a consonant, and the print draws a consonant as a letter outline. So the finding is a split, and a mark-granular UI has to say which of the two it is doing. That is the constraint mark-C inherits and it was not visible before the walk. What stays open is what arithmetic cannot reach — whether the box the dictionary names sits where a reader *looks* for that mark, and whether lighting it reads truer than washing the word. The inspector now draws that screen, so the blocker moves from "the inspector drawing the marks" to a hafiz looking at them. `encoding-inspector.md` gains `--marks` in §④ ①, §⑥ 4 and §⑧: the level, the size it costs, and the restated ink line. §⑥ 4 is the one that needed rewriting rather than appending — the temptation the rule anticipates gets *stronger* a level down, because a mark's box is small enough that filling it in would look like the mark. `map.json` gains three pointers, hand-edited: `lib/mark-join.mjs` and its test under word-geometry, `markLevel` under encoding-inspector. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
packages/core/src/skins.tshas said since Loop 6a that sub-word highlighting isimpossible because the print's glyphs are "anonymous outlined
<path>s". That istrue of
assets/pages/**. It is false of the ligature corpus, and this branchestablishes how false with two numbers instead of a grep: which word a mark
belongs to (④), and now which mark it is (⑤).
④ — the join
A tajweed rule is a
[start, end)over codepoints. Containment files a mark underthe right word; it says nothing about which letter. The corpus offers exactly
one join —
<g id="md-ligature-…">names the letters it draws indata-textandnests the marks drawn on them — and ④ asks whether that join holds across all 604
pages.
88.79% → 97.75% → 99.90% → 100.00%, and the number moved each time a markup dump
explained a family — never because a rule was added to move it. That order is the
whole point: a rule fitted to the data would make ④ agree with the corpus by
construction, which is exactly the property that would stop it being evidence.
Two of my own errors surfaced on the way:
reading order (
«ٱلرَّحِيمِ»→[لر|حيم|ٱ]) and a letter can be drawn in twoligatures (
«فَلَا»→[فلا|ا]). A length check passed both and then misassignedevery mark while the totals balanced.
alignnow matches on content, which isstrictly stronger — some words that used to pass now fail.
ligature's spelling.
«أَنَّ»is drawn[أ|ن]and still carrieshamzathenfatha; the conditional cost 151 words.The three that remain
Named in the docblock rather than absorbed, because a rule for either would be a
rule for one word:
«أَيۡدِيهِمۡ»at 21:28 and 22:76 — the word occurs 26 times; 24 drawhamza, fatha, sukun, kasra, kasra, sukunand these 2 draw the same list withoutthe
hamza.«لِيَسُـُٔواْ»at 17:7 — the print draws asmall wawand amaddahits owndata-hafswrites no codepoint for. The only word in the corpus where asmall wawpath appears without aU+06E5.Neither costs anything downstream. ② is what decides whether the geometry ships,
and ② is exact: 326,515 marks, 0 outside their own word, unchanged.
⑤ — which mark is which
④ counts. Counting says a ligature drawing three letters carries three marks; it
does not say which drawn path is the tanween and which is the sukun, and a tajweed
rule that wants to light the tanween needs exactly that.
The obvious answer — pair them off left to right — is not an answer at all: it
assumes the print draws marks in the order the text writes them, which is the
thing in question. So ⑤ never looks at position. Each agreeing ligature contributes
a bag of codepoint tokens beside a bag of drawn names, and the
correspondence is recovered by elimination across the whole corpus: if a run wants
{sukun, أ, fatha}and the print draws{hamza, sukun, fatha}, then once two arepinned elsewhere the third follows.
The mechanism is arc consistency over bipartite matchings (
supported), notset intersection. Intersection presumes the relation is already a function and
drove
U+0653to an empty candidate set on the first run — which is how theأ-carrier convention below was found. Arc consistency deletes a pairing only whenno perfect one-to-one assignment of a run's bag can use it, so it cannot
over-eliminate.
It is checked on data it was not shown. Every run carrying exactly one mark is
withheld from the propagation, because a one-mark run forces its own pairing and
scoring against it would report 100% by construction.
And only now, order
Measured afterwards, once pairing was settled without it — of 89,170 runs carrying
two marks or more:
R1 was stated before it was scored. A second candidate — shadda drawn after its
vowel — was refuted at −40 runs and deliberately not implemented; the docblock
records it. The residual 390 runs (0.44%) are enumerated by family, led by 50
instances of
«شَيۡـٔٗا».The one contradiction, resolved into a convention
U+0653(combining madda) onأis drawn as a shortened fatha-shaped stroke thecorpus names
fatha— 277 of 277, 0.89× the width of a same-line fatha — whileall nineteen other carriers get a path named
maddah(4,682 of them, one constantwidth corpus-wide). It gets its own token
U+0653@hamzasolely so the relationstays a function. This is the only place in the dictionary where a codepoint's
drawn name depends on the letter under it.
§⑦ said this was impossible, and was wrong
The section previously argued — with the probe's own output — that the
codepoint → namegap could not be closed offline, citing 611 pairings of asukun with a
hamzapath. Those 611 were an artifact of the tally positionallyzipping
wantagainstgot, which arc consistency never does.«بِٱلۡأٓخِرَةِ»was the worked example and ⑤ pairs it correctly.
The refutation is retired in prose rather than deleted, because it is a clean
example of the failure mode this repo keeps finding: a check that shares a
mistake with the thing it checks will agree with it.
What survives §⑦ unchanged: identity is not placement. A print that named its paths
correctly and placed one of them a letter to the left would satisfy ①–⑤ exactly
as a correct one does. That half stays eye-only, and
sub-word-marks.md§⑧ ① nowsplits the question into the offline-measurable half and the two that are not.
mark-B — the offset walks down to a rectangle
④ and ⑤ leave the join sitting inside a probe.
lib/mark-join.mjsextracts thearithmetic so there is exactly one of it, and
markPaths(word)is the entry pointa caller wants:
[{ at, len, token, name, mark }], whereatis a codepointindex into that word's own
data-hafs. That index is the bridge — tajweed offsetscount codepoints, so an annotation can now be walked all the way down to the
rectangle the print drew.
Extracted rather than left in place for the reason
tajweed-fold.mjsgives aboutitself:
probe-encodings.mjs --marksdraws these rectangles, and if theinspector's join were a second implementation, a clean screen would stop being
evidence about the probe — the one failure a diagnostic tool must not have.
It refuses rather than approximates.
nullfor a mark count that disagrees, aname the tokens did not ask for, or letters no assignment of ligatures can cover.
A partial answer here would look exactly like an answer, which is the defect the
left-to-right walk already shipped once while every length balanced.
mark-join.test.mjsis 28 tests on words small enough to count by hand, and mostof them assert one of those refusals.
The walk, over all 60,057 annotations
data-hafsThe 51.52% is an answer, not a shortfall — and this is the part worth reading
twice, because it looks like a coverage number somebody should try to raise. Ten
of the eighteen rules name a consonant:
qalqalahopens on ق,lam_shamsiyyahon ل,
ghunnahon ن or م. The print draws a consonant as a letter outline, not asa named mark. There is nothing above them to light, and there never was.
The eight rules that do name a mark reach one almost always —
hamzat_wasl→wasla98.11%,madd_2→superscript alef/small waw/small yeh98.58%,iqlab→small meemor a… iqlabcomposite 99.82%.So the finding is a split, not a rate: a mark-granular highlight is available
for the rules about marks and not for the rules about letters, and any UI built on
this has to say which it is doing. That is a design constraint mark-C inherits and
it was not visible before the walk.
Nothing is unreachable for a structural reason —
no-host,no-wordandunjoinedare all 0 — and the remainder is counted rather than guessed: 497respelt words, whose offsets do not address
data-hafsby construction, and 82oracle misses.
On screen
The inspector draws each mark's own rectangle inside its word box, with the
selected annotation's mark lit. Verified in a browser rather than asserted:
selecting a
hamzat_waslannotation on 2:4 lights exactly onerect.mkinsideexactly one
rect.box, titledwasla · ٱلَّذِينَ codepoint 0 · fold 2.Hollow and hairline, because the ink rule holds a level finer and is under more
pressure there: a mark's box is small enough that filling it in would look
like the mark. §⑥ 4 of
encoding-inspector.mdsays so in place.--marksis opt-in for one reason and it is a size: 326,515 more rectangles takesthe report from 5.0 MB to 13.8 MB.
What this leaves open
sub-word-marks.md§⑧ ① keeps its open status. Its offline half is answeredabove; what stays is what arithmetic cannot reach — whether the box the dictionary
names sits where a reader looks for that mark, and whether lighting it reads as
a truer rendering of the rule than washing the word. The blocker moves from "the
encoding inspector drawing the marks" to a hafiz looking at them.
Ships nothing
No bytes reach
apps/web/public/assets.make cigreen;gate:budget117.9 KB gzof 150 KB, +0.0 against the baseline.
gate:issues87 indexed across 13 registers,30 open;
gate:map22 features / 158 pointers.🤖 Generated with Claude Code
https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt