Skip to content

fix: dictionaries plugin — inflected forms (plurals, tenses) resolve to their lemma definition - #9

Open
symbiodev wants to merge 1 commit into
itsthisjustin:mainfrom
symbiodev:fix/dictionary-form-of-stubs
Open

fix: dictionaries plugin — inflected forms (plurals, tenses) resolve to their lemma definition#9
symbiodev wants to merge 1 commit into
itsthisjustin:mainfrom
symbiodev:fix/dictionary-form-of-stubs

Conversation

@symbiodev

Copy link
Copy Markdown

Bug

Looking up a plural noun (or other inflected form) in a built dictionary only ever shows a stub like "plural of cat" — never the actual definition of cat.

Root cause

The kaikki/Wiktionary source tags inflected-form entries (cats, ran, found, …) with form_of/alt_of and a gloss that is just the stub text ("plural of cat", "simple past of find"). build-dictionaries.mjs wrote that stub as the entry's only definition. Since the reader's dictionary engine resolves a word via a direct .idx hit before it ever falls back to .syn synonyms or stemming (see docs/dictionary.md in crosspoint-reader — "1. Direct match" wins over "3. Stemming"), the stub permanently shadowed the real definition. Stemming, which would have found the lemma, never got a chance to run.

Fix

writeStardict() now resolves form_of/alt_of pointers at build time:

  • A headword whose senses are purely references (no own sense) becomes an alias: its .idx row is folded onto the lemma's .idx row — same (offset, size), zero extra bytes in the .dict.dz.
  • A mixed headword (own sense and a form-of sense, e.g. "found" = "to establish" + "simple past of find") keeps its own text and appends the lemma's definition after it.
  • Alias chains ("runnin'" → "running" → "run") resolve to the material end; a chain that loops keeps its stub rather than resolving to nothing.
  • A form-of entry whose lemma is missing from the source (dead end) is left as-is — the stub is still better than nothing.

Rebuilding english with this fix: 632k of 1.39M headwords (45%) turned from a bare stub into a real, readable definition, at the same total dictionary size (the aliasing adds index rows, not dictionary bytes).

Testing

  • test/dictionaries.test.mjs (new): runs the full build pipeline against a synthetic kaikki source (--source id=file://..., new flag) and asserts on the resulting .idx/.dict.dz — pure aliasing, mixed headwords, dead-end stubs, alias chains, and that aliasing still respects the existing case-fold merge (marched → march/March group).
  • npm test: 20/20 passing (existing suite unaffected).
  • Manually verified against a full rebuild of english (1,385,197 words, 59.3 MB, same as before the fix) loaded onto a device: cats/ran/went/found/children all now show the lemma's definition; mice (whose form_of target isn't in the source under that exact key) still shows its stub as a fallback.

Also touches: --source/--catalog-dir CLI flags (test-only, let the suite build into a scratch dir against a local fixture instead of the network + committed catalog), and a fileURLToPath fix for CATALOG_DIR/the new test's script path so both survive a working directory containing a space.

Related — firmware feature requests

While tracking this down I also confirmed two things worth a firmware-side feature request on crosspoint-reader (not part of this PR, filing separately there):

  1. Select more than one word at once in DictionaryWordSelectActivity — useful for phrasal verbs and set phrases that aren't in the dictionary as a single headword.
  2. Look up a word from inside the definition screen (DictionaryDefinitionActivity) — chaining a lookup when the definition text itself uses an unfamiliar word.

Neither is reachable from the plugin side; both need firmware changes. Filing as issues on crosspoint-reader and will link them here once open.

Inflected forms (plurals, verb tenses, ...) in the kaikki/Wiktionary
source carry a form_of/alt_of pointer plus a stub gloss like "plural of
cat" instead of a real definition. Because a hit on the word's own .idx
row always wins before the reader falls back to .syn or stemming, every
one of these stub entries permanently shadowed the real definition —
looking up a plural noun, for instance, only ever showed "plural of
X", never X's actual meaning.

writeStardict() now folds a pure form-of/alt-of headword's .idx row
onto its lemma's definition bytes (an alias, at zero extra size cost),
appends the lemma definition after a mixed headword's own senses (e.g.
"found": own "to establish" sense plus "simple past of find"), and
follows alias chains to their material end while leaving a dead-end
stub (lemma missing from the source) untouched. Rebuilding english
with this fix turned 632k of 1.39M headwords (45%) from a bare stub
into a real definition, at the same total dictionary size.

Verified end to end in test/dictionaries.test.mjs against a synthetic
kaikki source, and manually against a full rebuild of the english
dictionary loaded onto a reader.
@symbiodev

Copy link
Copy Markdown
Author

Firmware feature requests filed, as mentioned above:

@itsthisjustin

Copy link
Copy Markdown
Owner

I think this is an issue for crosspoint not for the dictionary downloader right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants