fix: retain WordNet adjective satellites and key synsets by offset+POS - #12
Merged
Conversation
The converter discarded every adjective satellite. parse_data_file filtered
on `synset.ss_type == pos` and the adjective pass runs with pos="a", so all
10,717 ss_type="s" synsets in data.adj -- roughly 59% of WordNet's
adjectives -- were dropped while the pointers referencing them were still
emitted, leaving ~10,720 dangling references for any consumer following
adjective relations.
Relaxing that filter alone would not have made those pointers resolve.
WordNet writes "a" in the POS field of every pointer to an adjective,
including the 10,717 similar-to pointers that target satellites; "s" appears
only in ss_type. Offsets cannot carry identity either, being byte offsets
into a per-POS data file: 333 of them name more than one synset. Identity
therefore needs an offset paired with a POS that normalizes "s" to "a",
which is what the new Synset.key and Pointer.target_key provide, and what
the loader, search index, and relation traverser now key on. Previously the
loader silently overwrote colliding synsets and WordNetSearch raised on
them.
Satellite sense enrichment needed the same treatment: it built a head-less
sense key (lemma%5:LL:II::), but real satellite keys carry a
head_word:head_id suffix that data.adj does not record, and the head-less
prefix is ambiguous for 2,966 of 20,384 satellite keys. Lookups now key on
the synset offset from index.sense, which is exact.
Satellites index under both "s" and "a", so an adjective query returns heads
and satellites together while ss_type still reports "s" on the record.
Verified against WordNet 3.1: every raw synset now converts, with no key
collisions and no dangling pointer targets.
BREAKING CHANGE: WordNetLoader.synsets, lemma_index, and the four relation
indices are keyed by SynsetKey ("00001740n") rather than by bare offset.
get_synset still accepts a bare offset and returns None when one is
ambiguous; pass pos or a canonical key to disambiguate.
Closes #11
LEMMA_PATTERN required the first character to be alphanumeric, so WordNet's elided and calibre lemmas failed validation -- and because a failed lemma discards the whole synset, seven synsets vanished from the converted output: .22_caliber, .38_caliber and .45_caliber from data.adj, 'hood and 's_Gravenhage from data.noun, 'tween and 'tween_decks from data.adv. Allow a single leading dot or apostrophe before the first alphanumeric. The rest of the pattern is unchanged, so genuinely malformed lemmas are still rejected. Found while verifying the satellite fix against WordNet 3.1: with this in place, every raw synset in all four data files converts.
Five call sites formatted synset IDs as f"{synset.offset:08d}{ss_type}", but
offset is a str, so the 'd' format code raised ValueError at runtime. In the
CLI the exception was swallowed by a broad except and reported as "Relation
search failed", leaving `glazing search relations --dataset wordnet` broken
outright rather than visibly erroring.
Use synset.key, which both avoids the format bug and gives records the same
identity that pointers resolve to -- the raw ss_type would have produced
"...s" for a satellite record against "...a" for every pointer to it. The
merge dict in load_wordnet_from_jsonl becomes correct across the offsets
that collide between parts of speech for the same reason.
Also route _convert_pos_for_wordnet's WordNet-native codes through
normalize_pos so the "s" -> "a" mapping has one source of truth.
The .initialized marker was created with touch(), carrying no content, and initialize_datasets short-circuited on its mere existence. An upgraded user therefore kept whatever the previous glazing had written forever -- which for this release means a WordNet cache missing all 10,717 satellites, with no signal that anything is wrong. Record a DATA_SCHEMA_VERSION in the marker as JSON and treat a missing, older, or unparseable version as not-current, so `glazing init` re-converts without --force. Conversion runs from the already-downloaded raw files, so this costs no re-download. check_initialization keeps its existing meaning (data present) rather than being overloaded, since a stale cache reported as "not initialized" would be misleading; the new is_data_current answers the staleness question, and importing glazing warns distinctly for each case.
Minor rather than patch release: WordNetLoader.synsets, lemma_index, and the relation indices change key format from bare offset to SynsetKey, which is a breaking change for anyone reading those mappings directly. Pre-1.0, so a minor bump is the semver-appropriate signal. Also backfills the 0.2.3 changelog link reference, which was never added.
aaronstevenwhite
force-pushed
the
fix/wordnet-satellite-synsets
branch
from
July 21, 2026 15:23
2334009 to
2875972
Compare
Recast the two parenthetical asides as comma clauses and parentheses rather than swapping the character, so the sentences still read naturally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11.
The reported bug
The WordNet converter discarded every adjective satellite.
parse_data_filefiltered onsynset.ss_type == pos, and the adjective pass runs withpos="a", so all 10,717ss_type="s"synsets indata.adj, roughly 59% of WordNet's adjectives, were dropped while the pointers referencing them were still emitted. Any consumer following adjective relations hit ~10,720 dangling targets.Why the one-line fix wasn't enough
Two things I verified against raw WordNet 3.1 shaped the design:
Pointer POS is never
"s". Scanning every pointer in all four data files: all 10,717&(similar-to) pointers and all 7,856+pointers that target satellites carrypos == "a"."s"appears only inss_type. So a naive(offset, ss_type)key would leave every pointer-to-satellite dangling. The key must normalizestoa, which is exactly the consistency the issue's "Note for consumers" asks for.Offsets don't identify synsets. They're byte offsets into a per-POS data file, so they collide across files: 117,791 synsets share only 117,458 distinct offsets, giving 333 collisions. glazing keyed synsets by bare offset throughout, so
WordNetLoadersilently overwrote real synsets,WordNetSearch.add_synsetraisedValueErroron them, and every pointer resolved POS-blind.New
Synset.key/Pointer.target_keypair the offset with a normalized POS ("00014377a"), and the loader, search index, and relation traverser key on that. Satellites index under both"s"and"a", so an adjective query returns heads and satellites together whiless_typestill reports"s".Satellite sense enrichment needed the same treatment or the 10,717 new satellites would have arrived with no
sense_number/tag_count: enrichment built a head-less sense key (lemma%5:LL:II::), but real satellite keys carry ahead_word:head_idsuffixdata.adjdoesn't record, and the head-less prefix is ambiguous for 2,966 of 20,384 satellite keys. Lookups now key on the synset offset fromindex.sense, which is exact.Also fixed, found while verifying
LEMMA_PATTERNrequired an alphanumeric first character, so.22_caliber,'hood,'tween,'s_Gravenhageand friends failed lemma validation. A failed lemma discards the entire synset.glazing search relations --dataset wordnetwas broken outright, failing withUnknown format code 'd' for object of type 'str'(synset IDs built viaf"{synset.offset:08d}"against a string offset). The CLI swallowed it into "Relation search failed". Confirmed broken onmain, working here..initializedmarker was contentless, so upgraded users kept outdated data forever, which for this release means a WordNet cache missing all 10,717 satellites. It now records a schema version.Verification against WordNet 3.1
Every raw synset in all four data files now converts.
1424 passed; ruff, mypy (checked against 1.18.2 and CI's latest 2.3.0), and all pre-commit hooks clean. 29 new tests intests/test_wordnet/test_satellites.pycover the key primitives, satellite emission, cross-POS collisions, satellite pointer round-trips, reachability frompos="a", and the enrichment case prefix-matching would get wrong.Breaking change
WordNetLoader.synsets,lemma_index, and the four relation indices are keyed bySynsetKey("00001740n") rather than bare offset. This is unavoidable, since a bare-offset mapping cannot be correct given the collisions.get_synsetstill accepts a bare offset and returnsNonewhen one is ambiguous; passposor a canonical key to disambiguate. Version bumped 0.2.3 to 0.3.0 accordingly.Upgrading: existing converted data is missing all satellites and can't be repaired in place. Run
glazing initto re-convert from already-downloaded sources; no re-download needed.