StanzaNlpEngine: German multi-word tokens (im/am/zum) replace the doc text and drop every NER entity; indented text ends in 500 "Did not find word"

[svkaenel]

Describe the bug

With StanzaNlpEngine and the German model, any text containing a German multi-word token (a preposition-article contraction such as im, am, zum, zur, beim, vom, ins, ans) loses every NER entity: /analyze returns HTTP 200 with an empty list where the same sentence with the contraction written out returns the PERSON. On indented text (JSON snippets, tables, lists) the same defect turns into HTTP 500 Did not find word '...' in the list of tokens although it is expected to be found from LemmaContextAwareEnhancer.

Cause (read from presidio_analyzer/nlp_engine/stanza_nlp_engine.py, same on main today):

1. StanzaNlpEngine.load hardcodes processors="tokenize,pos,lemma,ner"; Stanza force-adds mwt for de ("Language de package default expects mwt, which has been added") and expands im → in + dem.
2. The inlined spacy-stanza tokenizer flattens token.words in StanzaTokenizer.__get_tokens_with_heads, so the expanded words no longer match the text. __get_words_and_spaces raises, and _convert_doc then replaces the whole doc text by " ".join(tokens) (the "Due to multiword token expansion or an alignment issue, the original text has been replaced by space-separated expanded tokens." warning).
3. The Stanza NER spans carry offsets of the original text. In the replaced doc they hit no token boundary, char_span returns None, and the entity is silently dropped → 200, empty result.
4. In indented text the replaced text is shorter than the original, so the token list ends before the text does. For a pattern recognizer hit past the last token, LemmaContextAwareEnhancer._find_index_of_match_token raises ValueError("Did not find word ...") → 500. (The loop condition tokens_indices[i] == start or start < tokens_indices[i] + len(token) accepts any start before the end of some token, so it only fires once the hit lies beyond the tokenizer's coverage. That is why the offending word in the message varies between runs.)

This is the presidio-side face of explosion/spacy-stanza#70 (open since 2021). Because Presidio inlines the tokenizer, the fix has to land here.

Pattern recognizers (EMAIL, IP, IBAN, ...) are not affected, they run on the original text. The stock spaCy de_core_news_* engine is not affected either (no MWT expansion).

To Reproduce

Image mcr.microsoft.com/presidio-analyzer:2.2.362 (presidio-analyzer 2.2.362, stanza 1.14.0) with a Stanza NLP configuration:

nlp_engine_name: stanza
models:
  - lang_code: de
    model_name: de
ner_model_configuration:
  labels_to_ignore: [O]
  model_to_presidio_entity_mapping:
    PER: PERSON
    LOC: LOCATION
    ORG: ORGANIZATION
    MISC: NRP

Minimal pair (score_threshold 0.4, entities: ["PERSON"], language: "de"):

text	result
Wir treffen uns im Büro mit Thomas Bergmann.	[]
Wir treffen uns in dem Büro mit Thomas Bergmann.	PERSON 32–47
Wir gehen zum Termin mit Thomas Bergmann. / ... zu dem Termin ...	[] / PERSON 28–43
Beim Kunden spricht Thomas Bergmann. / Bei dem Kunden ...	[] / PERSON 23–38
Die Mail vom Kunden kam von Thomas Bergmann. / ... von dem Kunden ...	[] / PERSON 32–47

curl -s http://localhost:5001/analyze -H 'Content-Type: application/json' \
  -d '{"text":"Wir treffen uns im Büro mit Thomas Bergmann.","language":"de","score_threshold":0.4,"entities":["PERSON"]}'
# -> []

The 500 path, same container: one multi-word token plus enough whitespace that the space-joined replacement text ends before the IP starts (a run of 40 spaces here; real-world triggers are indented JSON, tables and lists):

curl -s -i http://localhost:5001/analyze -H 'Content-Type: application/json' \
  -d '{"text":"Wir treffen uns im Büro.\n                                        \nServer 192.168.10.20","language":"de","entities":["IP_ADDRESS"]}'
# -> HTTP 500 {"error":"Did not find word '192.168.10.20' in the list of tokens although it is expected to be found"}

The same text with in dem instead of im returns 200 with the IP at 77–90.

In-process, without HTTP:

from presidio_analyzer.nlp_engine import StanzaNlpEngine
engine = StanzaNlpEngine(models=[{"lang_code": "de", "model_name": "de"}])
engine.load()
doc = engine.nlp["de"]("Wir treffen uns im Büro mit Thomas Bergmann.")
print(repr(doc.text))   # 'Wir treffen uns in dem Büro mit Thomas Bergmann . '  <- text replaced
print(list(doc.ents))   # []
doc = engine.nlp["de"]("Wir treffen uns in dem Büro mit Thomas Bergmann.")
print([(e.text, e.label_, e.start_char, e.end_char) for e in doc.ents])  # [('Thomas Bergmann', 'PER', 32, 47)]

Expected behavior

doc.text == text, NER entities with their original offsets (PERSON 28–43 for the first sentence), no 500 on indented text.

Additional context

A build-time patch we run in production keeps a multi-word token as one surface token in __get_tokens_with_heads (text and offsets of the token, the remaining annotations borrowed from its first word, lemma = surface form). With it the alignment holds, doc.text == text, entities keep their original offsets, and lemmas of ordinary tokens (hence context words) are unchanged. Verified on a 12-slice German document: before 11× 200 with 0 PERSON and 1× 500, after 12× 200 with 22 PERSON; the English leg identical.

for token in sentence.tokens:
    if len(token.words) > 1:
        heads.append(0)
        tokens.append(_SurfaceToken(token))   # text/lemma = token.text, start/end_char, ner of the token,
        continue                              # every other attribute via __getattr__ from token.words[0]
    for word in token.words:
        ...
offset += sum(1 if len(token.words) > 1 else len(token.words) for token in sentence.tokens)

A processors="tokenize,ner" variant is not a way out: Stanza adds mwt for de regardless, and lemmas become empty.

The trade-off is that the morpho-syntactic annotation of the expanded words is lost for that token, which for a PII analyzer is the right side to be on: the silent empty result is the harmful outcome here, a 200 with no PERSON is indistinguishable from "nothing there" for every caller. Happy to turn this into a PR against StanzaTokenizer if that direction is acceptable.