Skip to content

The ligature is the join, the marks say their names by elimination, and an offset walks down to one - #86

Merged
omars-lab merged 8 commits into
mainfrom
mark-b-ligature-join
Aug 7, 2026
Merged

The ligature is the join, the marks say their names by elimination, and an offset walks down to one#86
omars-lab merged 8 commits into
mainfrom
mark-b-ligature-join

Conversation

@omars-lab

@omars-lab omars-lab commented Aug 7, 2026

Copy link
Copy Markdown
Owner

packages/core/src/skins.ts has said since Loop 6a that sub-word highlighting is
impossible because the print's glyphs are "anonymous outlined <path>s". That is
true of assets/pages/**. It is false of the ligature corpus, and this branch
establishes how false with two numbers instead of a grep: which word a mark
belongs to
(④), and now which mark it is (⑤).

④ — the join

A tajweed rule is a [start, end) over codepoints. Containment files a mark under
the right word; it says nothing about which letter. The corpus offers exactly
one join — <g id="md-ligature-…"> names the letters it draws in data-text and
nests the marks drawn on them — and ④ asks whether that join holds across all 604
pages.

of 91,451 entries the print calls words
draw no letters at all — pause marks, ۩, ۞ 4,486 4.91%
of the remaining 86,965
join cleanly 86,962 100.00%
no assignment of ligatures to letters exists 0 0.00%
a ligature's mark count disagrees 3 0.00%

88.79% → 97.75% → 99.90% → 100.00%, and the number moved each time a markup dump
explained a family — never because a rule was added to move it. That order is the
whole point: a rule fitted to the data would make ④ agree with the corpus by
construction, which is exactly the property that would stop it being evidence.

Two of my own errors surfaced on the way:

  • The left-to-right walk was wrong twice over. Ligature document order is not
    reading order («ٱلرَّحِيمِ»[لر|حيم|ٱ]) and a letter can be drawn in two
    ligatures («فَلَا»[فلا|ا]). A length check passed both and then misassigned
    every mark while the totals balanced. align now matches on content, which is
    strictly stronger — some words that used to pass now fail.
  • The seated hamza always gets its path. I briefly made it conditional on the
    ligature's spelling. «أَنَّ» is drawn [أ|ن] and still carries hamza then
    fatha; the conditional cost 151 words.

The three that remain

Named in the docblock rather than absorbed, because a rule for either would be a
rule for one word:

  • «أَيۡدِيهِمۡ» at 21:28 and 22:76 — the word occurs 26 times; 24 draw
    hamza, fatha, sukun, kasra, kasra, sukun and these 2 draw the same list without
    the hamza.
  • «لِيَسُـُٔواْ» at 17:7 — the print draws a small waw and a maddah its own
    data-hafs writes no codepoint for. The only word in the corpus where a small waw path appears without a U+06E5.

Neither costs anything downstream. ② is what decides whether the geometry ships,
and ② is exact: 326,515 marks, 0 outside their own word, unchanged.

⑤ — which mark is which

④ counts. Counting says a ligature drawing three letters carries three marks; it
does not say which drawn path is the tanween and which is the sukun, and a tajweed
rule that wants to light the tanween needs exactly that.

The obvious answer — pair them off left to right — is not an answer at all: it
assumes the print draws marks in the order the text writes them, which is the
thing in question. So ⑤ never looks at position. Each agreeing ligature contributes
a bag of codepoint tokens beside a bag of drawn names, and the
correspondence is recovered by elimination across the whole corpus: if a run wants
{sukun, أ, fatha} and the print draws {hamza, sukun, fatha}, then once two are
pinned elsewhere the third follows.

The mechanism is arc consistency over bipartite matchings (supported), not
set intersection. Intersection presumes the relation is already a function and
drove U+0653 to an empty candidate set on the first run — which is how the
أ-carrier convention below was found. Arc consistency deletes a pairing only when
no perfect one-to-one assignment of a run's bag can use it, so it cannot
over-eliminate.

⑤ which mark is which — 152101 runs whose counts agree, 62931 held out
    for the test below, leaving 2869 distinct token-bag/name-bag shapes
    to propagate over; a fixpoint in 2 pass(es)
    34 of 34 tokens pinned to exactly one name; 0 still open
    [no dead shapes]

It is checked on data it was not shown. Every run carrying exactly one mark is
withheld from the propagation, because a one-mark run forces its own pairing and
scoring against it would report 100% by construction.

the held-out test — 62,931 single-mark runs, none of them propagated over
the dictionary predicts the drawn name 62,931 100.00%
predicts a different name 0 0.00%
a token that never appears beside another 0 0.00%

And only now, order

Measured afterwards, once pairing was settled without it — of 89,170 runs carrying
two marks or more:

drawn in the order the text writes them 87,957 98.64%
the same marks, drawn in another order 1,213 1.36%
with R1 — the seated hamza drawn last 88,780 99.56%

R1 was stated before it was scored. A second candidate — shadda drawn after its
vowel — was refuted at −40 runs and deliberately not implemented; the docblock
records it. The residual 390 runs (0.44%) are enumerated by family, led by 50
instances of «شَيۡـٔٗا».

The one contradiction, resolved into a convention

U+0653 (combining madda) on أ is drawn as a shortened fatha-shaped stroke the
corpus names fatha — 277 of 277, 0.89× the width of a same-line fatha — while
all nineteen other carriers get a path named maddah (4,682 of them, one constant
width corpus-wide). It gets its own token U+0653@hamza solely so the relation
stays a function. This is the only place in the dictionary where a codepoint's
drawn name depends on the letter under it.

§⑦ said this was impossible, and was wrong

The section previously argued — with the probe's own output — that the
codepoint → name gap could not be closed offline, citing 611 pairings of a
sukun with a hamza path
. Those 611 were an artifact of the tally positionally
zipping
want against got, which arc consistency never does. «بِٱلۡأٓخِرَةِ»
was the worked example and ⑤ pairs it correctly.

The refutation is retired in prose rather than deleted, because it is a clean
example of the failure mode this repo keeps finding: a check that shares a
mistake with the thing it checks will agree with it.

What survives §⑦ unchanged: identity is not placement. A print that named its paths
correctly and placed one of them a letter to the left would satisfy ①–⑤ exactly
as a correct one does. That half stays eye-only, and sub-word-marks.md §⑧ ① now
splits the question into the offline-measurable half and the two that are not.

mark-B — the offset walks down to a rectangle

④ and ⑤ leave the join sitting inside a probe. lib/mark-join.mjs extracts the
arithmetic so there is exactly one of it, and markPaths(word) is the entry point
a caller wants: [{ at, len, token, name, mark }], where at is a codepoint
index into that word's own data-hafs. That index is the bridge — tajweed offsets
count codepoints, so an annotation can now be walked all the way down to the
rectangle the print drew.

Extracted rather than left in place for the reason tajweed-fold.mjs gives about
itself: probe-encodings.mjs --marks draws these rectangles, and if the
inspector's join were a second implementation, a clean screen would stop being
evidence about the probe — the one failure a diagnostic tool must not have.

It refuses rather than approximates. null for a mark count that disagrees, a
name the tokens did not ask for, or letters no assignment of ligatures can cover.
A partial answer here would look exactly like an answer, which is the defect the
left-to-right walk already shipped once while every length balanced.
mark-join.test.mjs is 28 tests on words small enough to count by hand, and most
of them assert one of those refusals.

The walk, over all 60,057 annotations

where the rule's own letter is drawn
a named path — this is the rectangle to light 28,535 47.51%
a base letter with no named path 30,943 51.52%
the word was respelt, so the offset does not address data-hafs 497 0.83%
the oracle itself misses, so there is no position to resolve 82 0.14%
no host, no word, or a word the join refused 0 0.00%

The 51.52% is an answer, not a shortfall — and this is the part worth reading
twice, because it looks like a coverage number somebody should try to raise. Ten
of the eighteen rules name a consonant: qalqalah opens on ق, lam_shamsiyyah
on ل, ghunnah on ن or م. The print draws a consonant as a letter outline, not as
a named mark. There is nothing above them to light, and there never was.

The eight rules that do name a mark reach one almost always — hamzat_wasl
wasla 98.11%, madd_2superscript alef/small waw/small yeh 98.58%,
iqlabsmall meem or a … iqlab composite 99.82%.

So the finding is a split, not a rate: a mark-granular highlight is available
for the rules about marks and not for the rules about letters, and any UI built on
this has to say which it is doing. That is a design constraint mark-C inherits and
it was not visible before the walk.

Nothing is unreachable for a structural reason — no-host, no-word and
unjoined are all 0 — and the remainder is counted rather than guessed: 497
respelt words, whose offsets do not address data-hafs by construction, and 82
oracle misses.

On screen

The inspector draws each mark's own rectangle inside its word box, with the
selected annotation's mark lit. Verified in a browser rather than asserted:
selecting a hamzat_wasl annotation on 2:4 lights exactly one rect.mk inside
exactly one rect.box, titled wasla · ٱلَّذِينَ codepoint 0 · fold 2.

Hollow and hairline, because the ink rule holds a level finer and is under more
pressure there
: a mark's box is small enough that filling it in would look
like the mark. §⑥ 4 of encoding-inspector.md says so in place.

--marks is opt-in for one reason and it is a size: 326,515 more rectangles takes
the report from 5.0 MB to 13.8 MB.

What this leaves open

sub-word-marks.md §⑧ ① keeps its open status. Its offline half is answered
above; what stays is what arithmetic cannot reach — whether the box the dictionary
names sits where a reader looks for that mark, and whether lighting it reads as
a truer rendering of the rule than washing the word. The blocker moves from "the
encoding inspector drawing the marks" to a hafiz looking at them.

Ships nothing

No bytes reach apps/web/public/assets. make ci green; gate:budget 117.9 KB gz
of 150 KB, +0.0 against the baseline. gate:issues 87 indexed across 13 registers,
30 open; gate:map 22 features / 158 pointers.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt

omars-lab and others added 6 commits August 7, 2026 08:42
`readDiacritics` returns the ligature structure beside the flat mark list, and
`probe:diacritics` gains ④ to measure whether it holds.

Containment (②) files a mark under the right word. It says nothing about which
letter, and a tajweed rule is a [start, end) over codepoints — so without a
letter-level join a rule can be highlighted no finer than the whole word. The
corpus offers exactly one join: <g id="md-ligature-…"> names the letters it
draws in data-text and nests the marks drawn on them.

④ partitions data-hafs into the letters the print outlines, walks the ligatures
across that partition, and compares mark counts. Every failure is bucketed by
cause, because the check is layered and quoting the per-ligature agreement alone
would silently condition it on a filter the reader cannot see.

All 604 pages: of 91,451 entries the print calls words, 4,486 draw no letters at
all (the pause marks, ۩, ۞). Of the remaining 86,965, 86,880 join cleanly —
99.90%.

Getting there needed four print conventions read off the markup, not assumed;
each one is why an earlier draft of this read 97.75%:

  - a bare hamza ء is an outline like any letter, not a named mark
  - \p{Lm} folds: the tatweel is a tooth drawn into its neighbour, while the
    small waw ۥ and small yeh ۦ are named marks despite Unicode calling them
    letters
  - a vowel then an iqlab meem ۭ or ۢ is one composite glyph, `kasra iqlab`
  - a seated hamza and ٱ are a base outline plus their own named path

The remaining 85 entries are left alone on purpose. Each rule above exists
because the markup showed the print doing something; adding rules until the
number reads 100% would fit the rule to the data and make ④ agree with the
corpus by construction, which is the one property that would stop it being
evidence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
§⑤ records ④'s corpus measurement: the rejected positional attempt (88.79% and
a visibly wrong pairing tail), the markup that replaced it, the four print
conventions that had to be learned, the 99.90% result, and the four families the
85-entry residual falls into — each with the word that demonstrates it.

§⑦ said the letter question was unanswerable because the corpus expresses letter
order "as ligature ids this does not read". It reads them now, so that reason is
gone and the section would have been quietly wrong. The conclusion survives on a
better reason: counts are necessary and not sufficient. A ligature agreeing on
three marks does not establish the second is over the second letter, and a word
whose marks were internally permuted would pass ④ exactly as a correct one does.
Only an eye closes that, which is still mark-B.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
…loses

④ was at 99.90%. The user asked for 100%, and the route was the one that got
it from 88.79% to 97.75% to 99.90%: dump the markup for each residual family
and encode what the print actually does. No rule here was added to move the
number — every one of them names the word that demonstrated it.

Three families were left, and reading them turned up two errors of my own:

- `align` replaces the left-to-right walk. Ligature document order is not
  reading order — «ٱلرَّحِيمِ» is drawn `[لر|حيم|ٱ]` — and a letter can be
  drawn twice, «فَلَا» as `[فلا|ا]`. The old length check let both pass and
  then misassigned every mark while the totals balanced. Matching on content
  is strictly stronger, and some words that used to pass now fail.
- `FOLD`/`FOLDS` are one class used on both sides. A ligature's `data-text`
  carries the tatweel that `letters` folds away, so «مَـَٔابٗا» → `[مـا|با]`
  looked unassignable when it is simply spelt with its tooth.
- The seated hamza always gets its path. A first reading of «أَيۡدِيهِمۡ»
  suggested the ligature's own spelling decided it; «أَنَّ» is drawn `[أ|ن]`
  and still carries `hamza` then `fatha`. That conditional cost 151 words.

86,962 of 86,965 lettered words now join cleanly. The three left are the
corpus disagreeing with itself, and the docblock names them rather than
absorbing them, because a rule for either would be a rule for one word:
«أَيۡدِيهِمۡ» loses its hamza path in 2 of its 26 occurrences, and 17:7's
«لِيَسُـُٔواْ» draws a `small waw` and a `maddah` its own `data-hafs` does
not write.

Neither costs anything downstream: ② decides whether the geometry ships, and
② is exact — 0 marks outside their word, unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
§⑤'s table and its four-family residual list were written against 99.90% and
are now wrong in both directions — the number is 100.00% of lettered words and
the residual is three entries, not eighty-five. The conventions table gains the
two rows that closed it (the sajda overline, the split and out-of-order
ligatures) and corrects the seated-hamza row to say **always**.

§⑦ said counts are necessary and not sufficient and left it abstract. It now
carries the probe's own evidence against itself: the `codepoint → name` tally
is built only from ligatures whose counts agree and still contains 611 pairings
of a sukun with a `hamza` path. «بِٱلۡأٓخِرَةِ» is one — the run `لأ` is
written `ۡ أ ٓ` and drawn `hamza, sukun, fatha`. Three wanted, three drawn, ④
passes, and all three pairings are wrong.

Which is the argument for mark-B in one example: the tally's head is
trustworthy, its tail is not, and no arithmetic tells them apart.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
probe-diacritics grows a fifth question. ④ established that a ligature
drawing three letters carries three marks; it never said which drawn path
was the tanween and which the sukun, and a tajweed rule that wants to light
the tanween needs exactly that.

The obvious answer — pair them off left to right — is not an answer. It
assumes the print draws marks in the order the text writes them, which is
the thing in question, and 1.36% of multi-mark runs say otherwise. The old
`pairs` tally did exactly that, and its output was cited in the design doc
as proof the gap could not be closed. It was measuring its own assumption.
Deleted.

⑤ never looks at position. Each agreeing ligature contributes a bag of
codepoint tokens beside a bag of drawn names, and the correspondence falls
out of elimination across the corpus: arc consistency over bipartite
matchings, where a pairing dies only when no perfect assignment of that run
can use it. Set intersection was tried first and is the wrong operator — it
presumes the relation is already a function, and drove U+0653 to an empty
candidate set. That empty set turned out to be the finding: the print draws
a combining madda as `maddah` on nineteen carriers and as `fatha` on a
hamza-carrying alef, 277 times of 277, so the codepoint alone is not a
function and needs the carrier. `U+0653@hamza` exists for that reason only.

Every single-mark run is held out. A one-mark run forces its own pairing, so
scoring against it would report 100% by construction — an earlier version of
this did, and it was circular.

  152,101 runs, 62,931 held out, 2,869 shapes to learn from
  34 of 34 tokens pinned in two passes, 0 shapes unsatisfiable
  62,931 of 62,931 held-out runs predicted correctly (100.00%)

Order is measured only afterwards, once pairing is settled without it:
98.64% drawn as written, 99.56% with R1 (a seated hamza's sign drawn last),
stated before it was scored. A second candidate — shadda after its vowel —
was put up the same way and refuted by its own score at −40 runs. Dropped.

Also fixes the usage line, which named a `--filter @hifth/etl` script that
does not exist; and excludes markless ligatures from the run set, where they
agreed vacuously and diluted the held-out percentage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
sub-word-marks.md §⑦ argued no arithmetic could say which drawn path belongs
to which codepoint, and offered as proof that the probe's own tally carried
611 pairings of a sukun with a path named `hamza`. Those 611 were an artifact
of how that tally was built — it zipped a run's codepoints against its paths
by position, so every word where the print reorders was miscounted by
construction. The section is rewritten rather than deleted, because the
refutation is worth keeping: a check that shares a mistake with the thing it
checks will agree with it.

What survives §⑦ is smaller and real. ⑤ closes identity — this path is the
sukun and not the hamza — and every step of it is a correspondence between a
reconstructed text and the corpus's own attributes. None of it looks at the
picture. A print that named its paths correctly and placed one a letter to
the left would satisfy ①–⑤ exactly as a correct one does. That is mark-B's
check, and it is why mark-B is a separate step rather than a review of
mark-C.

§⑤ gains the propagation: 34 of 34 tokens pinned, 62,931 held-out runs
predicted at 100.00%, and order measured only afterwards — 98.64% as
written, 99.56% with R1, with the refuted shadda rule recorded as refuted.

The conventions table goes from six to eight. The two new rows are one
convention stated as two: a combining madda draws as `maddah` on nineteen
carriers and as `fatha` on a hamza-carrying alef, 277 of 277 — the only place
in the corpus where a codepoint's drawn name depends on the letter under it.
The geometry leans toward it being the madda drawn short (0.89× a same-line
fatha, against the maddah's one constant width), and nothing downstream needs
the answer.

issues.json ① is narrowed, not closed: half of "does a tajweed span land on a
mark" is now an offline measurement and should be made before anyone looks at
a screen. The half that needs an eye is stated as two questions instead of
one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
@omars-lab omars-lab changed the title The ligature is the join, and it reaches every word but three The ligature is the join, and the marks say their names by elimination Aug 7, 2026
omars-lab and others added 2 commits August 7, 2026 14:07
… to land

`lib/mark-join.mjs` extracts ④ and ⑤'s arithmetic out of `probe-diacritics.mjs`
so there is exactly one of it: `letters` folds `\p{Mn}` and `\p{Lm}` onto the
letter before, `align` matches ligatures to letter runs by content with
backtracking, and `pairMarks` pairs a wanted token to a drawn name through the
frozen 34-entry `DRAWN_NAME`. `markPaths` is the one entry point a caller
should want — a word in, `[{ at, len, token, name, mark }]` out, with `at` a
codepoint index into that word's own `data-hafs`, which is the bridge a tajweed
offset needs.

It refuses rather than approximates. `null` for a mark count that disagrees, a
name the tokens did not ask for, or letters no assignment of ligatures can
cover — because a partial answer here would look exactly like an answer, which
is the defect ④'s left-to-right walk already shipped once while every length
balanced. `mark-join.test.mjs` is 28 tests on words small enough to count by
hand and most of them assert one of those refusals.

`probe-encodings.mjs --marks` is the caller. It walks all 60,057 tajweed
annotations from a Tanzil offset down to a drawn path and counts the outcome
into eight named classes rather than a rate:

  drawn        28535  47.51%   a named path — the rectangle to light
  letter       30943  51.52%   a base letter, with nothing above it
  respelt        497   0.83%   the offset does not address `data-hafs`
  oracle-miss     82   0.14%   no position to resolve in the first place
  no-host / no-word / unjoined / basmala   0   0.00%

The 51.52% is an answer. Ten of the eighteen rules name a consonant — qalqalah
on ق, lam_shamsiyyah on ل, ghunnah on ن or م — and the print draws a consonant
as a letter outline. The eight that name a mark reach one almost always:
hamzat_wasl → wasla 98.11%, madd_2 98.58%, iqlab 99.82%. So the finding is a
split rather than a rate, and any mark-granular UI has to say which of the two
it is doing.

The inspector draws the boxes with the selected annotation's own mark lit
inside its word. Hollow and hairline: the outlines-yes-ink-no rule holds a
level finer, and it is under more pressure there, because a mark's box is small
enough that filling it in would read as the mark itself.

`--marks` stays opt-in for one reason and it is a size: 326,515 more rectangles
takes the report from 5.0 MB to 13.8 MB.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
…er half needs an eye

`sub-word-marks.md` §⑧ ① keeps its **open** status and gains the measurement
that halves it. The offline half is answered: all 60,057 tajweed annotations
walked from a Tanzil offset to a drawn path, 47.51% reaching a named rectangle
and 51.52% landing on a base letter, with nothing unreachable for a structural
reason and the remainder — 497 respelt words, 82 oracle misses — named rather
than dropped.

The section argues why the 51.52% is an answer instead of a shortfall, because
that is the part a reader would otherwise take for a coverage number and try to
raise: ten of the eighteen rules name a consonant, and the print draws a
consonant as a letter outline. So the finding is a split, and a mark-granular
UI has to say which of the two it is doing. That is the constraint mark-C
inherits and it was not visible before the walk.

What stays open is what arithmetic cannot reach — whether the box the
dictionary names sits where a reader *looks* for that mark, and whether
lighting it reads truer than washing the word. The inspector now draws that
screen, so the blocker moves from "the inspector drawing the marks" to a hafiz
looking at them.

`encoding-inspector.md` gains `--marks` in §④ ①, §⑥ 4 and §⑧: the level, the
size it costs, and the restated ink line. §⑥ 4 is the one that needed rewriting
rather than appending — the temptation the rule anticipates gets *stronger* a
level down, because a mark's box is small enough that filling it in would look
like the mark.

`map.json` gains three pointers, hand-edited: `lib/mark-join.mjs` and its test
under word-geometry, `markLevel` under encoding-inspector.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuhvbUKjGesE3uMhjCzBGt
@omars-lab omars-lab changed the title The ligature is the join, and the marks say their names by elimination The ligature is the join, the marks say their names by elimination, and an offset walks down to one Aug 7, 2026
@omars-lab
omars-lab merged commit a37ed95 into main Aug 7, 2026
6 checks passed
@omars-lab
omars-lab deleted the mark-b-ligature-join branch August 7, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant