fix: dictionaries plugin — inflected forms (plurals, tenses) resolve to their lemma definition - #9
Open
symbiodev wants to merge 1 commit into
Open
Conversation
Inflected forms (plurals, verb tenses, ...) in the kaikki/Wiktionary source carry a form_of/alt_of pointer plus a stub gloss like "plural of cat" instead of a real definition. Because a hit on the word's own .idx row always wins before the reader falls back to .syn or stemming, every one of these stub entries permanently shadowed the real definition — looking up a plural noun, for instance, only ever showed "plural of X", never X's actual meaning. writeStardict() now folds a pure form-of/alt-of headword's .idx row onto its lemma's definition bytes (an alias, at zero extra size cost), appends the lemma definition after a mixed headword's own senses (e.g. "found": own "to establish" sense plus "simple past of find"), and follows alias chains to their material end while leaving a dead-end stub (lemma missing from the source) untouched. Rebuilding english with this fix turned 632k of 1.39M headwords (45%) from a bare stub into a real definition, at the same total dictionary size. Verified end to end in test/dictionaries.test.mjs against a synthetic kaikki source, and manually against a full rebuild of the english dictionary loaded onto a reader.
Author
|
Firmware feature requests filed, as mentioned above:
|
Owner
|
I think this is an issue for crosspoint not for the dictionary downloader right? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
Looking up a plural noun (or other inflected form) in a built dictionary only ever shows a stub like "plural of cat" — never the actual definition of cat.
Root cause
The kaikki/Wiktionary source tags inflected-form entries (
cats,ran,found, …) withform_of/alt_ofand a gloss that is just the stub text ("plural of cat", "simple past of find").build-dictionaries.mjswrote that stub as the entry's only definition. Since the reader's dictionary engine resolves a word via a direct.idxhit before it ever falls back to.synsynonyms or stemming (seedocs/dictionary.mdin crosspoint-reader — "1. Direct match" wins over "3. Stemming"), the stub permanently shadowed the real definition. Stemming, which would have found the lemma, never got a chance to run.Fix
writeStardict()now resolvesform_of/alt_ofpointers at build time:.idxrow is folded onto the lemma's.idxrow — same(offset, size), zero extra bytes in the.dict.dz.Rebuilding
englishwith this fix: 632k of 1.39M headwords (45%) turned from a bare stub into a real, readable definition, at the same total dictionary size (the aliasing adds index rows, not dictionary bytes).Testing
test/dictionaries.test.mjs(new): runs the full build pipeline against a synthetic kaikki source (--source id=file://..., new flag) and asserts on the resulting.idx/.dict.dz— pure aliasing, mixed headwords, dead-end stubs, alias chains, and that aliasing still respects the existing case-fold merge (marched → march/March group).npm test: 20/20 passing (existing suite unaffected).english(1,385,197 words, 59.3 MB, same as before the fix) loaded onto a device:cats/ran/went/found/childrenall now show the lemma's definition;mice(whoseform_oftarget isn't in the source under that exact key) still shows its stub as a fallback.Also touches:
--source/--catalog-dirCLI flags (test-only, let the suite build into a scratch dir against a local fixture instead of the network + committed catalog), and afileURLToPathfix forCATALOG_DIR/the new test's script path so both survive a working directory containing a space.Related — firmware feature requests
While tracking this down I also confirmed two things worth a firmware-side feature request on crosspoint-reader (not part of this PR, filing separately there):
DictionaryWordSelectActivity— useful for phrasal verbs and set phrases that aren't in the dictionary as a single headword.DictionaryDefinitionActivity) — chaining a lookup when the definition text itself uses an unfamiliar word.Neither is reachable from the plugin side; both need firmware changes. Filing as issues on crosspoint-reader and will link them here once open.