Skip to content

fix: retain WordNet adjective satellites and key synsets by offset+POS - #12

Merged
aaronstevenwhite merged 6 commits into
mainfrom
fix/wordnet-satellite-synsets
Jul 21, 2026
Merged

fix: retain WordNet adjective satellites and key synsets by offset+POS#12
aaronstevenwhite merged 6 commits into
mainfrom
fix/wordnet-satellite-synsets

Conversation

@aaronstevenwhite

@aaronstevenwhite aaronstevenwhite commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Closes #11.

The reported bug

The WordNet converter discarded every adjective satellite. parse_data_file filtered on synset.ss_type == pos, and the adjective pass runs with pos="a", so all 10,717 ss_type="s" synsets in data.adj, roughly 59% of WordNet's adjectives, were dropped while the pointers referencing them were still emitted. Any consumer following adjective relations hit ~10,720 dangling targets.

Why the one-line fix wasn't enough

Two things I verified against raw WordNet 3.1 shaped the design:

  1. Pointer POS is never "s". Scanning every pointer in all four data files: all 10,717 & (similar-to) pointers and all 7,856 + pointers that target satellites carry pos == "a". "s" appears only in ss_type. So a naive (offset, ss_type) key would leave every pointer-to-satellite dangling. The key must normalize s to a, which is exactly the consistency the issue's "Note for consumers" asks for.

  2. Offsets don't identify synsets. They're byte offsets into a per-POS data file, so they collide across files: 117,791 synsets share only 117,458 distinct offsets, giving 333 collisions. glazing keyed synsets by bare offset throughout, so WordNetLoader silently overwrote real synsets, WordNetSearch.add_synset raised ValueError on them, and every pointer resolved POS-blind.

New Synset.key / Pointer.target_key pair the offset with a normalized POS ("00014377a"), and the loader, search index, and relation traverser key on that. Satellites index under both "s" and "a", so an adjective query returns heads and satellites together while ss_type still reports "s".

Satellite sense enrichment needed the same treatment or the 10,717 new satellites would have arrived with no sense_number/tag_count: enrichment built a head-less sense key (lemma%5:LL:II::), but real satellite keys carry a head_word:head_id suffix data.adj doesn't record, and the head-less prefix is ambiguous for 2,966 of 20,384 satellite keys. Lookups now key on the synset offset from index.sense, which is exact.

Also fixed, found while verifying

  • Seven more silently-dropped synsets. LEMMA_PATTERN required an alphanumeric first character, so .22_caliber, 'hood, 'tween, 's_Gravenhage and friends failed lemma validation. A failed lemma discards the entire synset.
  • glazing search relations --dataset wordnet was broken outright, failing with Unknown format code 'd' for object of type 'str' (synset IDs built via f"{synset.offset:08d}" against a string offset). The CLI swallowed it into "Relation search failed". Confirmed broken on main, working here.
  • Stale caches never re-converted. The .initialized marker was contentless, so upgraded users kept outdated data forever, which for this release means a WordNet cache missing all 10,717 satellites. It now records a schema version.

Verification against WordNet 3.1

before after raw
adjective synsets 7,465 18,185 18,185
satellites 0 10,717 10,717
total synsets n/a 117,791 117,791
distinct keys n/a (333 collisions) 117,791, 0 collisions
dangling pointer targets ~10,720 0
satellite words enriched 0 20,384 / 20,384

Every raw synset in all four data files now converts. 1424 passed; ruff, mypy (checked against 1.18.2 and CI's latest 2.3.0), and all pre-commit hooks clean. 29 new tests in tests/test_wordnet/test_satellites.py cover the key primitives, satellite emission, cross-POS collisions, satellite pointer round-trips, reachability from pos="a", and the enrichment case prefix-matching would get wrong.

Breaking change

WordNetLoader.synsets, lemma_index, and the four relation indices are keyed by SynsetKey ("00001740n") rather than bare offset. This is unavoidable, since a bare-offset mapping cannot be correct given the collisions. get_synset still accepts a bare offset and returns None when one is ambiguous; pass pos or a canonical key to disambiguate. Version bumped 0.2.3 to 0.3.0 accordingly.

Upgrading: existing converted data is missing all satellites and can't be repaired in place. Run glazing init to re-convert from already-downloaded sources; no re-download needed.

The converter discarded every adjective satellite. parse_data_file filtered
on `synset.ss_type == pos` and the adjective pass runs with pos="a", so all
10,717 ss_type="s" synsets in data.adj -- roughly 59% of WordNet's
adjectives -- were dropped while the pointers referencing them were still
emitted, leaving ~10,720 dangling references for any consumer following
adjective relations.

Relaxing that filter alone would not have made those pointers resolve.
WordNet writes "a" in the POS field of every pointer to an adjective,
including the 10,717 similar-to pointers that target satellites; "s" appears
only in ss_type. Offsets cannot carry identity either, being byte offsets
into a per-POS data file: 333 of them name more than one synset. Identity
therefore needs an offset paired with a POS that normalizes "s" to "a",
which is what the new Synset.key and Pointer.target_key provide, and what
the loader, search index, and relation traverser now key on. Previously the
loader silently overwrote colliding synsets and WordNetSearch raised on
them.

Satellite sense enrichment needed the same treatment: it built a head-less
sense key (lemma%5:LL:II::), but real satellite keys carry a
head_word:head_id suffix that data.adj does not record, and the head-less
prefix is ambiguous for 2,966 of 20,384 satellite keys. Lookups now key on
the synset offset from index.sense, which is exact.

Satellites index under both "s" and "a", so an adjective query returns heads
and satellites together while ss_type still reports "s" on the record.

Verified against WordNet 3.1: every raw synset now converts, with no key
collisions and no dangling pointer targets.

BREAKING CHANGE: WordNetLoader.synsets, lemma_index, and the four relation
indices are keyed by SynsetKey ("00001740n") rather than by bare offset.
get_synset still accepts a bare offset and returns None when one is
ambiguous; pass pos or a canonical key to disambiguate.

Closes #11
LEMMA_PATTERN required the first character to be alphanumeric, so WordNet's
elided and calibre lemmas failed validation -- and because a failed lemma
discards the whole synset, seven synsets vanished from the converted output:
.22_caliber, .38_caliber and .45_caliber from data.adj, 'hood and
's_Gravenhage from data.noun, 'tween and 'tween_decks from data.adv.

Allow a single leading dot or apostrophe before the first alphanumeric. The
rest of the pattern is unchanged, so genuinely malformed lemmas are still
rejected.

Found while verifying the satellite fix against WordNet 3.1: with this in
place, every raw synset in all four data files converts.
Five call sites formatted synset IDs as f"{synset.offset:08d}{ss_type}", but
offset is a str, so the 'd' format code raised ValueError at runtime. In the
CLI the exception was swallowed by a broad except and reported as "Relation
search failed", leaving `glazing search relations --dataset wordnet` broken
outright rather than visibly erroring.

Use synset.key, which both avoids the format bug and gives records the same
identity that pointers resolve to -- the raw ss_type would have produced
"...s" for a satellite record against "...a" for every pointer to it. The
merge dict in load_wordnet_from_jsonl becomes correct across the offsets
that collide between parts of speech for the same reason.

Also route _convert_pos_for_wordnet's WordNet-native codes through
normalize_pos so the "s" -> "a" mapping has one source of truth.
The .initialized marker was created with touch(), carrying no content, and
initialize_datasets short-circuited on its mere existence. An upgraded user
therefore kept whatever the previous glazing had written forever -- which
for this release means a WordNet cache missing all 10,717 satellites, with
no signal that anything is wrong.

Record a DATA_SCHEMA_VERSION in the marker as JSON and treat a missing,
older, or unparseable version as not-current, so `glazing init` re-converts
without --force. Conversion runs from the already-downloaded raw files, so
this costs no re-download.

check_initialization keeps its existing meaning (data present) rather than
being overloaded, since a stale cache reported as "not initialized" would be
misleading; the new is_data_current answers the staleness question, and
importing glazing warns distinctly for each case.
Minor rather than patch release: WordNetLoader.synsets, lemma_index, and the
relation indices change key format from bare offset to SynsetKey, which is a
breaking change for anyone reading those mappings directly. Pre-1.0, so a
minor bump is the semver-appropriate signal.

Also backfills the 0.2.3 changelog link reference, which was never added.
@aaronstevenwhite
aaronstevenwhite force-pushed the fix/wordnet-satellite-synsets branch from 2334009 to 2875972 Compare July 21, 2026 15:23
Recast the two parenthetical asides as comma clauses and parentheses rather
than swapping the character, so the sentences still read naturally.
@aaronstevenwhite
aaronstevenwhite merged commit 0de2ab0 into main Jul 21, 2026
9 checks passed
@aaronstevenwhite
aaronstevenwhite deleted the fix/wordnet-satellite-synsets branch July 21, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WordNet converter drops all adjective satellite synsets (ss_type 's')

1 participant