Himotoki (紐解き, "unraveling") is a Python remake of ichiran, the comprehensive Japanese morphological analyzer. It segments Japanese text into words, provides dictionary definitions, and traces conjugation chains back to their root forms -- all powered by a portable SQLite backend.
- Portable SQLite Backend -- No PostgreSQL setup required. Dictionary data lives in a single file (~3 GB) that is generated on first use.
- Dynamic-Programming Segmentation -- Uses a Viterbi-style algorithm to find the most linguistically plausible word boundaries.
- Deep Dictionary Integration -- Built on JMDict, providing glosses, part-of-speech tags, usage notes, and cross-references.
- Recursive Deconjugation -- Walks the conjugation database to trace inflected forms (passive, causative, te-form, negation, etc.) back to dictionary entries.
- Conjugation Breakdown Tree -- Displays each transformation step in a visual tree with the suffix, grammatical label, and English gloss.
- Compound Word Detection -- Recognizes suffix compounds (te-iru progressive, te-shimau completion, tai desiderative, sou appearance, etc.) and shows their internal structure.
- Scoring Engine -- Implements synergy and penalty heuristics from ichiran to resolve segmentation ambiguities.
pip install himotokiOn first run, Himotoki will offer to download JMDict and build the SQLite database. The process takes approximately 10-20 minutes and requires about 3 GB of free disk space.
himotoki "日本語テキスト"━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Welcome to Himotoki!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
First-time setup required. This will:
- Download JMdict dictionary data (~15MB compressed)
- Generate optimized SQLite database (~3GB)
- Store data in ~/.himotoki/
Proceed with setup? [Y/n]:
Non-interactive setup for CI environments:
himotoki setup --yes# Default: dictionary info with conjugation breakdown
himotoki "学校で勉強しています"* 学校 【がっこう】
1. [n] school
* で
1. [prt] at; in
2. [prt] at; when
3. [prt] by; with
* 勉強しています 【べんきょう しています】
1. [n,vs,vt] study
2. [n,vs,vi] diligence; working hard
└─ する (makes a verb from a noun)
└─ Conjunctive (~te) (て)
└─ いる (indicates continuing action (to be ...ing))
└─ Polite (ます)
# Full output: romanization + dictionary info + conjugation tree
himotoki -f "食べられなかった"taberarenakatta
* taberarenakatta 食べられなかった 【たべられなかった】
1. [v1,vt] to eat
2. [v1,vt] to live on (e.g. a salary); to live off; to subsist on
← 食べる 【たべる】
└─ Potential/Passive (れる): can do / is done (to)
└─ Negative (ない): not
└─ Past (~ta) (かった): did/was
# Simple romanization
himotoki -r "学校で勉強しています"
# Output: gakkou de benkyou shiteimasu
# Kana reading with spaces
himotoki -k "学校で勉強しています"
# Output: がっこう で べんきょう しています
# JSON output for programmatic use
himotoki -j "学校で勉強しています"import himotoki
# Optional: pre-warm caches for faster first request
himotoki.warm_up()
# Analyze Japanese text
results = himotoki.analyze("日本語を勉強しています")
for words, score in results:
for w in words:
print(f"{w.text} 【{w.kana}】 - {w.gloss[:50]}...")Himotoki traces conjugated words through every transformation step, showing the root form and each inflection applied:
$ himotoki -f "書かせられていた"
kakaserareteita
* kakaserareteita 書かせられていた 【かかせられていた】
1. [v5k,vt] to write; to compose; to pen
2. [v5k,vt] to draw; to paint
← 書く 【かく】
└─ Causative (かせ): makes do
└─ Passive (られる): is done (to)
└─ Conjunctive (~te, progressive) (て)
└─ Past (~ta) (た): did/was
A deeply nested chain parsed into its constituent parts:
$ himotoki -f "飲んでしまいたかった"
nondeshimaitakatta
* nondeshimaitakatta 飲んでしまいたかった 【のんでしまいたかった】
1. [v5m,vt] to drink; to swallow; to take (medicine)
← 飲む 【のむ】
└─ Conjunctive (~te) (んで)
└─ しまう (indicates completion / to do something by accident or regret)
└─ Continuative (~i) (い): and (stem)
└─ たい (want to... / would like to...)
└─ Past (~ta) (かった): did/was
Full sentence analysis with per-word dictionary entries and conjugation trees:
$ himotoki "学校で勉強しています"
* 学校 【がっこう】
1. [n] school
* で
1. [prt] at; in
2. [prt] at; when
3. [prt] by; with
* 勉強しています 【べんきょう しています】
1. [n,vs,vt] study
2. [n,vs,vi] diligence; working hard
← 勉強する 【べんきょうする】
└─ Conjunctive (~te, progressive) (て)
└─ Polite (ます)
Himotoki recognizes productive suffix patterns and merges them into compound words with grammatical labels:
| Category | Suffixes | Example |
|---|---|---|
| Progressive / completive | ている, てある, てしまう, ておく | 食べている → "is eating" |
| Giving / receiving | てくれる, てもらう, てあげる, てやる | 読んであげる → "read for someone" |
| Desire / attempt | たい, てほしい, てみる | 食べたい → "want to eat" |
| Appearance / degree | そう, すぎる, っぽい, らしい | 高すぎる → "too expensive" |
| Compound verbs | 出す, 切る, 合う, 込む, 始める, 終わる | 食べ始める → "start eating" |
| Nominalization | さ, み, 方 | 深み → "depth" |
| Contractions | ちゃう, じゃう, とく, なきゃ | 食べちゃった → "ended up eating" |
| Polite / formal | です, でしょう, ください, いたす | 食べてください → "please eat" |
| Na-adjective suffixes | すぎる, っぽい, み, そう | 静かすぎる → "too quiet" |
Himotoki processes Japanese text through three stages:
-
Segmentation -- A dynamic-programming algorithm considers all possible word boundaries and selects the highest-scoring path. Scoring uses dictionary frequency data, part-of-speech synergies, and penalty heuristics ported from ichiran.
-
Suffix Compound Assembly -- Adjacent segments are checked against known suffix patterns (te-iru, te-shimau, tai, sou, etc.). Matching segments are merged into compound WordInfo objects with preserved component structure.
-
Conjugation Chain Resolution -- For each conjugated word, the system queries the conjugation database to walk the
viachain from the surface form back to the dictionary entry. Each step records the conjugation type, suffix text, and English gloss, then formats the result as an indented tree.
himotoki/
segment.py # Viterbi-style segmentation engine
lookup.py # Dictionary lookup, scoring, conjugation data
output.py # WordInfo, conjugation tree, JSON/text formatting
suffixes.py # Suffix compound detection (te-iru, tai, etc.)
synergies.py # Part-of-speech synergy and penalty rules
conjugation_hints.py # Supplementary conjugation patterns
constants.py # Conjugation type IDs, POS tags, glosses
characters.py # Kana/kanji conversion, romanization
counters.py # Japanese counter expression handling
cli.py # Command-line interface
db/ # SQLAlchemy models and connection management
loading/ # JMDict XML parsing and database generation
scripts/
llm_eval.py # LLM-based accuracy evaluation (510 sentences)
check_segments.py # Quick segmentation change checker
llm_report.py # HTML report generator
tests/ # 433 tests (pytest + hypothesis)
data/ # Dictionary data, evaluation datasets
git clone https://github.com/msr2903/himotoki.git
cd himotoki
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest tests/ -x --tb=short# Run all tests
pytest tests/ -x --tb=short
# Run conjugation tree tests only
pytest tests/test_conjugation_tree.py -v
# Run with coverage
pytest tests/ --cov=himotoki --cov-report=term-missingThe project includes an LLM-based evaluation system that scores segmentation accuracy against 510 curated Japanese sentences:
python scripts/llm_eval.py --quick # 50-sentence subset
python scripts/llm_eval.py # Full evaluation
python scripts/llm_eval.py --rescore 5 # Re-evaluate entry #5
python scripts/llm_report.py # Generate HTML reportCurrent accuracy: 510/510 (100%) on the v3 evaluation prompt.
Distributed under the MIT License. See LICENSE for details.