wordhoard

Old English wordhord — one's store of words.

An open, multilingual, frequency-ranked dictionary built entirely from openly-licensed sources. Each entry carries a lemma, part of speech, corpus frequency (rank + count), and — where the language's resources allow — noun gender, inflection, and a CEFR level estimate.

It exists because no such openly-licensed resource combined all of these: the high-quality frequency dictionaries (DeReWo, SUBTLEX, Routledge) are non-commercial or copyrighted, and none pairs frequency + lemma + gender + CEFR under a license you can redistribute. wordhoard builds one from OpenSubtitles frequencies + Wiktionary gender/inflection + (for German) Goethe CEFR calibration.

Coverage

Built and verified for German, French, Spanish, Italian, English (scripts/report.py, NOUN-only — proper nouns excluded as they lack grammatical gender in these lexicons):

lang	words	nouns	gender%	core g% (top-5k)	plural%	verb conj%	notes
de	69,877	27,596	97%	99%	67%	39%	gender der/die/das + full declension; Goethe CEFR anchors
fr	66,814	29,355	97%	100%	47%	26%	gender m/f (kaikki)
es	56,638	22,918	95%	99%	59%	18%	gender m/f (kaikki)
it	68,063	29,028	92%	98%	41%	15%	gender m/f (kaikki)
en	85,142	31,756	—	—	74%	58%	no grammatical gender; plurals only

All five languages also get lemma/POS correction (a kaikki known-inflection lexicon fixes context-free spaCy tagging noise, e.g. German lege→legen/ VERB instead of surviving as its own lemma) and verb conjugation tables (pres.3sg, impf.1sg, cond.3pl, part.perf, ... in word_form, for verbs frequent enough that their infinitive is itself in the build). See DATASET.md for details and known limitations.

Adding a language is mostly a registry entry in src/wordhoard/config.py — the OpenSubtitles frequency backbone and kaikki gender/inflection exist for dozens of languages. See CONTRIBUTING.md.

Get the data

The built dataset (SQLite + per-language CSV) is attached to GitHub Releases — start there if you just want the dictionary. A top-2000-per-language preview is committed under samples/ so you can inspect the shape without a download.

Build it yourself

uv sync
uv run python -m spacy download de_core_news_sm   # download AFTER uv sync (it prunes
                                                  # out-of-band models); fr_/es_/it_core_news_sm, en_core_web_sm
uv run python scripts/fetch.py --lang de          # downloads sources into data/raw/<lang>/
uv run python scripts/build.py --lang de --limit 100000   # --limit 3000 = smoke
uv run python scripts/report.py                   # coverage table
uv run python scripts/export.py --out dist        # release artifacts (CSV + SQLite)
uv run pytest -q

Each build upserts one language into data/build/wordhoard.db, leaving other languages untouched. The kaikki dump (kaikki.jsonl, used for fr/es/it/en gender and for every language's lemma/POS correction + verb conjugation) is a 0.5–3 GB download per language; the test suite needs neither models nor corpora (it stubs spaCy).

Data dictionary

word — one row per (lemma, POS):

column	meaning
lang	ISO code (de, fr, …)
lemma	dictionary form (lowercased; the corpus is lowercased)
pos	Universal POS tag (NOUN, VERB, …)
gender	de: der/die/das; fr/es/it: m/f; en: NULL
frequency_rank	1 = most frequent, per language
frequency_count	aggregated corpus occurrences
cefr_estimate	A1–C2 (anchored for de; frequency-calibrated otherwise)
notes	provenance: cefr:{anchor,est} and, for nouns, ;gen:{cefr,lex,spacy}

word_form — surface forms per word, (word_id, form, feature), UNIQUE(word_id, form). feature is surface for an observed-but-unanalysed form, or a label like pl / nom.pl / gen.sg for an analysed noun inflection, or (for verbs) a conjugation label like pres.3sg / impf.1pl / cond.3sg / part.perf / imp.sg / subj.pres.3sg.

Example queries

-- 20 most frequent French nouns with gender and plural
SELECT w.lemma, w.gender, f.form AS plural
FROM word w LEFT JOIN word_form f ON f.word_id = w.id AND f.feature = 'pl'
WHERE w.lang = 'fr' AND w.pos = 'NOUN'
ORDER BY w.frequency_rank LIMIT 20;

-- German A1 nouns whose gender is known
SELECT lemma, gender FROM word
WHERE lang = 'de' AND pos = 'NOUN' AND cefr_estimate = 'A1' AND gender IS NOT NULL
ORDER BY frequency_rank;

-- present-tense conjugation of the most frequent German verbs
SELECT w.lemma, f.feature, f.form
FROM word w JOIN word_form f ON f.word_id = w.id
WHERE w.lang = 'de' AND w.pos = 'VERB' AND f.feature LIKE 'pres.%'
  AND w.frequency_rank <= (SELECT MIN(frequency_rank) + 20 FROM word
                            WHERE lang = 'de' AND pos = 'VERB')
ORDER BY w.frequency_rank, f.feature;

Sources & licensing

Layer	Source	License
Frequency	OpenSubtitles-2018 via hermitdave/FrequencyWords	MIT
Lemma / POS	spaCy xx_core_news_sm / en_core_web_sm	MIT
Noun gender + declension (de)	German Wiktionary via gambolputty/german-nouns	CC-BY-SA-4.0
Noun gender + plural (fr/es/it/en)	Wiktionary via kaikki.org (wiktextract)	CC-BY-SA-4.0
Lemma/POS correction + verb conjugation (all 5)	Wiktionary via kaikki.org (wiktextract)	CC-BY-SA-4.0
CEFR anchors (de only)	Goethe-Institut A1–B1 wordlists	© Goethe — used only to calibrate the frequency→CEFR mapping; not redistributed

wordhoard is dual-licensed: the code is MIT; the built dataset is CC-BY-SA-4.0 (viral, inherited from the Wiktionary gender/inflection layer). Redistributing the dataset requires attribution — see NOTICE.md. The Goethe wordlists are a calibration input only; no Goethe content ships in the output.

Layout

• schema.sql — word + word_form
• src/wordhoard/ — config (language registry), db, build (pipeline core)
• scripts/ — fetch.py, build.py, report.py (coverage), export.py (artifacts)
• samples/ — committed top-2000-per-language preview
• data/ — sources & built DB (gitignored)