pol-Latn: greedy grapheme matching consumes the base letter of a decomposed diacritic — ⟨rż⟩/⟨rź⟩ mistranscribed via the ⟨rz⟩ digraph

[jc0019]

Summary

In Polish (pol-Latn), words containing the letter sequences ⟨rż⟩ or ⟨rź⟩ (where ⟨ż⟩/⟨ź⟩ are single letters, not part of a digraph) are transcribed incorrectly: the ⟨rz⟩ digraph rule consumes the ⟨r⟩ plus the base letter of the decomposed ⟨ż⟩/⟨ź⟩, deleting /r/ and leaving a stranded combining mark in the output.

Minimal reproduction

import epitran
epi = epitran.Epitran('pol-Latn')

epi.transliterate('skarżyć')     # 'skaʐ̇ɨt͡ɕ'   expected: 'skarʐɨt͡ɕ'
epi.transliterate('drży')        # 'dʐ̇ɨ'        expected: 'drʐɨ'
epi.transliterate('zmarźnięta')  # 'zmaʐ́ɲɛnta'  expected: 'zmarʑɲɛnta'

epi.trans_list('skarżyć')
# ['s', 'k', 'a', 'ʐ', '̇', 'ɨ', 't͡ɕ']   ← /r/ lost; stray U+0307 in output

The input encoding does not matter (NFC and NFD inputs both reproduce it). Environment: epitran 1.35.2, Python 3.11.

Root cause

Epitran.general_trans normalizes the input to NFD before matching (simple.py), so ⟨ż⟩ becomes z + U+0307. The greedy longest-match regex built by _build_greedy_match_regex then matches the digraph key rz against r + z, splitting the decomposed letter: the combining dot is orphaned and later emitted as an unmapped character, while ⟨rż⟩ = /rʐ/ surfaces as ⟨rz⟩ = /ʐ/.

The same failure class can in principle affect any language whose map contains a multigraph XY where Y is also the base letter of a decomposable letter in that orthography.

Proposed fix

Forbid a grapheme match from ending immediately before a combining mark, by appending a negative lookahead in _build_greedy_match_regex:

graphemes = sorted(g2p_keys, key=len, reverse=True)
pattern = f"({r'|'.join(graphemes)})"
# Don't split a base letter from its combining mark(s) — unless this map
# deliberately maps bare combining marks (e.g. tone diacritics).
if not any(unicodedata.combining(g[0]) for g in graphemes if g):
    pattern += r'(?!\p{M})'
return regex.compile(pattern, regex.I)

Tested on pol-Latn: skarżyć → skarʐɨt͡ɕ, drży → drʐɨ, zmarźnięta → zmarʑɲɛnta, while regular ⟨rz⟩/⟨ż⟩ words (rzeka, żaba, może) are unchanged. The unicodedata.combining guard avoids regressing languages whose maps key bare combining marks directly.

Happy to submit a PR with this change plus regression tests if the approach looks right to you.

Context

Found while building a phonemicized Polish child-directed-speech lexicon for phonotactic-learning experiments; the bug affected 32 of ~44,000 word types (the skarżyć, drżeć, marznąć/zmarźnięty families).