Add soft_hyphenate(); prune non-modern compound suffixes; adopt pyright#23
Merged
Conversation
Insert U+00AD soft hyphens at internal compound-component boundaries of Icelandic word forms, so a typesetter can break long compounds at morphologically valid points. - Wordbase.insert_soft_hyphens(): pure-DAWG primitive. Derives boundaries from the existing compound splitter via a head-recursive decomposition: take the preferred split (longest last part, fewest parts), keep the modifier parts, and recurse only into the head (always a legal standalone suffix). Modifiers are never re-sliced, which keeps spurious seams (byggingar->bygg|ingar, ...|arf|lokkur) out of the result. - Bin.soft_hyphenate(): DB-backed wrapper that additionally leaves closed-class function words untouched (gagnvart, asamt) by consulting BIN. - mode="natural" (default) / "primary"; min_left/min_right/min_word knobs; case-insensitive with original casing preserved (ALL-CAPS, Capitalized, proper nouns); real hyphens and spaces are hard boundaries; idempotent. Documented limitation: a compound *modifier* is left whole (morgunverdarhladbord -> morgunverdar|hlad|bord). Tests cover the natural/ primary modes, casing, the function-word guard, regression cases for the spurious-seam fixes, and a battery of real-world examples. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- CI: replace the mypy type-check step with pyright, honoring pyrightconfig.json (strict mode, target Python 3.9). - Convert AGENTS.md commands to the uv workflow (uv sync --extra dev, uv run pytest, uv run pyright) and add a CLAUDE.md quick-reference so the uv/pyright workflow is picked up at session start. - Commit uv.lock (pyright/pytest pinned in the dev extra). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A handful of suffix forms are inflections of obsolete/poetic/rare words and make poor inflection-template seeds — e.g. 'ingar' (NFFT of the poetic noun 'ingi'), which let 'byggingar' be sliced as 'bygg|ingar'. Remove them at the data source rather than working around them in the splitter. - tools/find_bad_suffixes.py: scans suffixes.txt and flags a form when it is in BIN and *all* of its direct readings carry a non-modern register mark (malsnid in SKALD/GAM/URE/FORN/SJALD/VILLA/STAD/BARN/OTOK). Forms with any modern reading are kept, so non-Core-but-modern words (gunnur, flanni) stay. - src/islenska/resources/suffix-removals.txt: the resulting 127,423 forms. - dawgbuilder.py: feed that list as a removals file to the ordalisti-suffixes build. Removing from the suffix DAWG alone suffices — the splitter rejects any split whose last part is not a legal suffix; ordalisti-all/-prefixes are untouched, and the words remain in BIN for direct lookups. Verified: 'ingar' no longer a legal suffix and 'byggingar' no longer splits, while byggingarkostnadur/skolabokasafn and direct lookups are unchanged; full test suite passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The head-only natural decomposition left a compound *modifier* whole (morgunverðarhlaðborð -> morgunverðar|hlað|borð, missing morgun|verðar). Add an optional recurse_modifier predicate to Wordbase.insert_soft_hyphens: in "natural" mode a modifier is also decomposed when the predicate accepts it. Bin.soft_hyphenate injects a predicate that recurses possessive prefixes — a form that is in BIN with at least one genitive (EF) noun reading. In modifier position a form that can be genitive almost certainly is the linking genitive, so presence (not genitive-only) is enough. Gating on contains() keeps the lookup from falling back to compound resolution. Combined with the suffix-list cleanup, this stays clean: byggingar is possessive but no longer splits (ingar pruned), and morgunverðar -> morgun| verðar, gervigreindar -> gervi|greindar are recovered. The pure-DAWG primitive keeps head-only behaviour unless a predicate is supplied. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a public soft-hyphenation API for Icelandic word forms, plus a
supporting data-quality fix to the compound-suffix list and a tooling switch
to pyright.
soft_hyphenate()— new APIBin.soft_hyphenate(word, ...)inserts U+00AD soft hyphens at internalcompound-component boundaries, so a typesetter can break long compounds at
morphologically valid points:
skólabókasafn→skólabókasafn,EFNAHAGSRÁÐHERRA→EFNAHAGSRÁÐHERRAboundaries; idempotent; function words (
ásamt) left untouched.head, and (in the DB-backed method) also unpacks possessive (genitive)
modifiers:
morgunverðarhlaðborð→morgunverðarhlaðborð.Wordbase.insert_soft_hyphens()(head-only, takes anoptional
recurse_modifierpredicate) and the BÍN-backedBin.soft_hyphenate().Suffix-list cleanup
tools/find_bad_suffixes.pyflags suffix forms whose every BÍN reading is anon-modern register (poetic/old/obsolete/archaic/rare/dialectal/…). The
resulting 127,423 forms (
suffix-removals.txt) are dropped from theordalisti-suffixesbuild, so e.g.ingar(NFFT of poeticingi) is nolonger a legal compound head and
byggingarno longer mis-splits asbygg|ingar. BÍN direct lookups are unchanged.Tooling
CI type-checks with pyright (via
uv) instead of mypy; addsCLAUDE.mdand converts
AGENTS.mdto the uv workflow.Testing
uv run pytest— all green (incl. newtest_soft_hyphenate*cases andreal-world examples);
uv run pyright— 0 errors.🤖 Generated with Claude Code