A proof-of-concept comparing language-detection engines — lingua, langdetect, pycld2, fasttext, and langid — on short-text EN / MY / ID (English / Malay / Indonesian) classification, and combining them into a weighted voting ensemble.
Naming:
EN/MY/IDare the language labels used in prose and table headers below;en/ms/idare their BCP-47 ISO codes, used in code font when referring to a literal model output value.
The main evaluation dataset is data/test_case_7_enmyid.txt — 1,273 English/Malay/Indonesian
cases in LANG | text format, filtered from the 5-language test_case_7.txt (2,036 cases,
which also includes Chinese and Tamil). Cases are organized into five word-count buckets so
accuracy and speed can be measured per bucket per language, not just overall:
| Bucket | n | Description |
|---|---|---|
| 1 word | 677 | 50% shared/formal roots (stress-tests EN/MY/ID ambiguity), 50% exclusive/colloquial (tests distinct dialect routing) |
| 2 words | 399 | Same 50/50 shared-root / colloquial split as bucket 1 |
| 3–7 words | 94 | Authentic Bahasa Rojak, code-switching patterns |
| 8–16 words | 73 | Localized slang, multi-word educational phrases |
| 17–50 words | 30 | Full sentences, educational content, paragraphs |
Earlier, smaller test cases (test_case_1.txt through test_case_6.txt) are kept in data/
for history — see the archived reports in reports/archive/ for the runs that used them. Labels
are scored as exact-match BCP 47 codes (en, ms, id).
Evaluated on 1,273 EN/MY/ID short-text cases (test_case_7_enmyid.txt). Current production
baseline (langdetect alone) scores 29.1% overall accuracy and cannot detect Malay at all
(0.0% MY). The best-ALL-accuracy config, S2 Weighted Voting (lingua-high + openlid-v3
pycld2), reaches 70.8% overall accuracy — +41.7 pp over production — while running 8.6–115.5× faster per call. That 70.8% figure comes with a trade-off worth reading before you pick a config: S2 Weighted's own Malay (MY) accuracy is 43.7%, below bothlingua-highrunning alone (55.8%) and the S1 Two-Stage Weighted alternative (56.5%). Choose S2 Weighted if ID accuracy/latency/simplicity matter more than MY protection; choose S1 Two-Stage Weighted if MY protection matters more (only 0.5 pp behind on ALL). Full decision matrix in §9 of the report.
| Strategy | Ensemble | EN | MY | ID | ALL |
|---|---|---|---|---|---|
| langdetect (current production) | — | 42.8% | 0.0% | 44.3% | 29.1% |
| lingua-high (individual) | — | 92.1% | 55.8% | 57.9% | 68.8% |
| S1 two_stage_weighted | ld+li+py (2-stage) | 92.4% | 56.5% | 61.4% | 70.3% |
| S2 weighted (best ALL/ID; MY trade-off, see above) | ol+li+py | 92.1% | 43.7% | 76.0% | 70.8% |
The 70.8%/56.5% figures above also depend on this dataset's bucket composition (53% single-word
cases) — see src/benchmark/reweight_by_real_distribution.py
and §12b of the report
for what these numbers become under more realistic query-length distributions.
pycld2 is the most discriminative single model by AUC (0.97), but its raw accuracy alone is
low (§1 of the report) — it needs the other voters to convert that discriminative power into
correct calls. Full ROC methodology in §1.5 of the report.
Scenario 2 (recommended) wins on steady-state latency but costs more to deploy:
| Dimension | S1 (lingua+langdetect+pycld2) |
S2 (lingua+openlid-v3+pycld2) |
|---|---|---|
| Per-request latency (1-word text) | 5.5216 ms | 0.0478 ms (115.5× faster) |
| Per-request latency (17–50 words) | 2.1027 ms | 0.2435 ms (8.6× faster) |
| Model artifact size | ~2.3 MB | 1.2 GB (openlid-v3.bin) |
| Cold-start load time | ~0.242 s | ~1.155 s (~4.8× slower) |
| Routing logic needed | Two-stage voting required to protect MY accuracy | Single-stage vote, no routing |
Since detection runs synchronously before any downstream NLP stage, per-request latency dominates in production; the 1.2 GB artifact and slower cold start are one-time costs paid at deploy/restart, not per request. Full breakdown in §10 of the report.
All code lives under src/, grouped by purpose:
| Folder | Contents |
|---|---|
src/voting/ |
Core voting-ensemble logic and evaluation scripts (accuracy, kappa, AUC, reproducibility, two-stage voting) for Scenario 1 (lingua-high+langdetect+pycld2) |
src/voting/strategies/ |
Scenario 2 (lingua-high+openlid-v3+pycld2) voting logic, organized by technique: hard.py, soft.py, weighted.py, two_stage.py, plus the run_s2_*.py report-generation scripts. Formerly src/voting/scenario2/ — see the module docstring in src/voting/strategies/__init__.py for the old-name → new-name mapping used by the report's "S1"/"S2" terminology |
src/benchmark/ |
Single-engine benchmark script producing confusion matrices and ROC curves; also reweight_by_real_distribution.py, which recomputes reported accuracy under a different query-length distribution than the test set's own bucket mix |
src/examples/ |
Earlier standalone demo/comparison scripts (lingua vs. langdetect, iterative versions) — kept for history, not part of the main pipeline |
models/ |
Pretrained language-ID model binaries (lid.176.ftz, openlid-v3.bin) |
data/ |
Test-case text files used as evaluation input |
results/ |
Generated artifacts: logs/, logs_scenario2/ (Scenario 2 output; directory name predates the strategies/ rename and is left as-is since it's just generated data), calibration/, confusion_matrix/, roc_curve/ — all reproducible by re-running the scripts above |
reports/ |
Written analysis of results — see reports/language_detection_ensemble_evaluation.md for the current canonical report. Superseded drafts/earlier runs live in reports/archive/. |
tests/ |
Pytest suite: unit tests for the pure, model-free functions in src/voting/core.py, plus one end-to-end test that runs a small fixture dataset through the full S1 and S2 pipelines — see Tests below for exactly what the CI badge does and doesn't cover |
Requires Python 3.12 — pycld2, langid, and statsmodels don't yet have prebuilt
wheels for newer Python versions. requirements.txt installs fasttext-wheel rather than
the official fasttext package, since fasttext has no prebuilt Windows wheel and fails
to compile from source on newer MSVC/pybind11 toolchains — fasttext-wheel is a drop-in,
prebuilt-wheel fork; the import name is still import fasttext.
python3.12 -m venv .venv
.venv\Scripts\activate # Windows
pip install -r requirements.txt
pip install -e . # registers voting/benchmark as importable packagesmodels/openlid-v3.bin (1.2 GB) is excluded from this repo via .gitignore — it's too large
for a normal git push. Download it separately from Hugging Face
and place it at models/openlid-v3.bin before running any openlid-v3 scripts.
Run the main benchmark:
python src/benchmark/benchmark.py [path/to/test_case.txt]Run the voting ensemble evaluation:
python src/voting/voting_main.pyScripts write logs/plots into results/, keyed by test case, and default to
data/test_case_7_enmyid.txt if no input file is given.
python -m pytest tests/What the tests CI badge actually validates (.github/workflows/tests.yml runs this same
command on Ubuntu, Python 3.12, on every push/PR to main):
| Coverage | What it checks | What it does NOT check |
|---|---|---|
tests/test_core.py |
Pure, model-free functions in src/voting/core.py — bucketing, vote-counting arithmetic, path numbering, accuracy aggregation |
Never loads a real model; doesn't touch the actual voting/scoring pipeline |
tests/test_pipeline_e2e.py |
Runs tests/fixtures/e2e_fixture.txt (18 hand-labeled cases) through the real S1 pipeline (lingua-high+langdetect+pycld2) and, when models/openlid-v3.bin is present, the real S2 pipeline (lingua-high+openlid-v3+pycld2); asserts accuracy stays within an expected range |
Not a re-validation of the report's 1,273-case statistical findings — it's a regression guard against pipeline-wiring bugs (broken imports, wrong argument order, a vote function silently returning "unknown"). The S2 half is skipped in CI, because models/openlid-v3.bin (1.2 GB) is excluded via .gitignore and not fetched by the workflow — CI only ever exercises the S1 half of this file |
In short: passing CI means the vote-counting logic and the S1 pipeline wiring are intact. It does
not mean the 70.8%/56.5%/etc. accuracy figures in the report have been re-verified — those
come from the one-off scripts in src/voting/ and src/voting/strategies/ run manually against
the full dataset, not from anything CI runs automatically.
models/lid.176.ftz is Facebook/Meta's pretrained fastText language-identification model,
redistributed here unmodified. It is licensed under
CC BY-SA 3.0, trained on data from
Wikipedia, Tatoeba, and SETimes. See fastText's language-identification
docs
for details. models/openlid-v3.bin is excluded from this repo via .gitignore (see
Setup) and is not redistributed here.
- lingua-py — natural language detection library, tuned for short and mixed-language text
- langdetect — Python port of Google's language-detection library
- pycld2 — Python bindings for Google's Compact Language Detector 2 (CLD2)
- fastText language identification — pretrained fastText models for language ID (176 languages)
- langid.py — stand-alone language identification system
- OpenLID — fastText-based language ID model covering 201 languages, from Burchell et al., "An Open Dataset and Model for Language Identification" (2023); model weights on Hugging Face
- OpenLID-v3 — the version actually used in this repo (
models/openlid-v3.bin); trained by HPLT on OpenLID-v2, glotlid-corpus, and Wikipedia data
- Confusion matrix (scikit-learn)
- ROC curve (scikit-learn) and ROC AUC score
- Cohen's kappa score (scikit-learn) — inter-rater agreement metric
- Voting classifiers / ensembles (scikit-learn) — reference for the weighted-voting ensemble approach used here
- McNemar's test (mlxtend) — paired significance test used to check whether one model/ensemble is genuinely more accurate than another on the same test set
- Wilson score interval — confidence interval for per-language accuracy, more reliable than a normal approximation at small/imbalanced sample sizes
- Expected Calibration Error (ECE) — "On Calibration of Modern Neural Networks" — metric for whether a model's confidence scores match its real-world accuracy
- FLORES-200 — multilingual benchmark dataset whose language/script labels were used to map model outputs to ISO codes
- BCP 47 / IETF language tags (RFC 5646) — standard used for the language codes (e.g.
en,ms,id) that all models are scored against