fix(analyzer): keep Stanza multi-word tokens as surface tokens to preserve text and entity offsets by ManoharPaturi · Pull Request #2253 · data-privacy-stack/presidio

ManoharPaturi · 2026-09-13T16:57:45Z

StanzaTokenizer.__get_tokens_with_heads flattened stanza multi-word-token expansions (im -> in + dem), so token texts no longer matched the source text. _convert_doc then rebuilt the doc text from those tokens, which corrupted the document text (see the issue repro), made Stanza NER spans miss every token boundary so entities were silently dropped, and could raise ValueError from the lemma enhancer on indented text.

Multi-word tokens now stay a single surface token (a small wrapper carrying surface text/lemma plus the first expanded word's annotations), with a word-index to token-index map that remaps dependency heads across the collapsed token. Single-word tokens keep the previous code path byte-for-byte.

Verified with the real German model: doc.text is preserved, PER and IP_ADDRESS entities are detected where pristine drops them. 11 new tests (pure-unit head remap + German model tests over all 8 contractions) fail on main and pass here; existing stanza suites pass unchanged (12 + 13 + 16), ruff clean vs main baseline.

…serve text and entity offsets Stanza's mwt processor expands German contractions (im/am/zum/zur/beim/ vom/ins/ans) into multiple words. StanzaTokenizer.__get_tokens_with_heads flattened the expanded words, which broke alignment with the original text: _convert_doc then replaced the doc text with space-separated expanded tokens, silently dropped every NER entity (offsets no longer mapped to tokens), and, on indented text, let tokens_indices end before pattern matches near the end of the text, raising 'Did not find word ...' in LemmaContextAwareEnhancer (HTTP 500). Keep a multi-word token as a single surface token (MultiWordTokenSurface: text/lemma are the surface form, other annotations come from the first expanded word) and remap 1-based word heads to the collapsed token list so dependency heads around the collapsed token stay correct. Fixes data-privacy-stack#2249

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

ArjunPakhan · 2026-09-14T11:08:33Z

Hi @ManoharPaturi 🚀

I pulled branch pr-2253 locally and verified the analyzer changes in stanza_nlp_engine.py.

Linting and formatting pass cleanly via ruff, and unit tests pass. Wrapping multi-word tokens as surface tokens while remapping dependency heads across collapsed tokens cleanly resolves the German MWT offset corruption (im, am, zum, zur, beim, vom, ins, ans) without introducing regressions to single-word token execution paths.

Great fix! +1 for maintainers to approve workflow execution and merge.

ManoharPaturi · 2026-09-14T15:36:06Z

thanks a lot for pulling the branch and testing it yourself, glad it holds up across the contractions. appreciate the detailed writeup.

omri374 · 2026-09-15T12:51:39Z

@@ -0,0 +1,137 @@
+# Fix: StanzaNlpEngine German multi-word tokens replace the doc text and drop entities (#2249)


please remove this file

Copilot

🟡 Changes recommended

Dependency-head remapping still uses an incorrect relative index.

Get a fresh assessment by requesting another Copilot review.

Review details

Files reviewed: 3/3 changed files
Comments generated: 1
Review effort level: Lite

+                    head = (
+                        offset
+                        + word_index_to_token_index.get(token.head - 1, token_index)
+                        - len(tokens)
+                    )


ManoharPaturi · 2026-09-15T15:37:08Z

@omri374 removed, thanks.

on the copilot point about the head remap using an incorrect relative index: i checked it against the real German model rather than the synthetic fixture, and the remap comes out correct on every token. for 'Wir treffen uns im Büro mit Thomas Bergmann.' stanza gives words (in, dem) as ids 4,5 both governed by id 6 (Büro), which collapses to token 3 governed by token 4:

token	stanza gov	expected rel	our output
Wir (w1)	w2	+1	+1
treffen (w2)	root	0	0
uns (w3)	w2	-1	-1
im (w4+w5)	w6	+1	+1
Büro (w6)	w2	-3	-3
mit (w7)	w8	+1	+1
Thomas (w8)	w2	-5	-5
Bergmann (w9)	w8	-1	-1
. (w10)	w2	-7	-7

the offset + word_index_to_token_index term is the governor's absolute doc index and len(tokens) is the current token's absolute index at that point, so the difference is the relative head. happy to add this sentence as a regression test if useful.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

ArjunPakhan · 2026-09-15T16:52:24Z

Hey @ManoharPaturi! 🚀

Glad the German model trace held up and disproved the Copilot bot finding. Since we worked through verifying the MWT offset logic and head remapping edge cases together, would you mind adding a co-author trailer to your final commit before merge?

You can just add this to the end of the commit message:

Co-authored-by: Arjun Sanjay Pakhan arjunpakhan@gmail.com

Appreciate it, and great job getting this fix put together!

omri374

Thanks!

omri374 · 2026-09-16T09:07:45Z

+        self.sentences = sentences
+
+
+def test_get_tokens_with_heads_collapses_mwt_and_remaps_heads():


Please add @pytest.mark.skip_engine("stanza_de"). We don't want to download a german stanza model on every unit test run

omri374 · 2026-09-16T09:08:08Z

+    assert heads == [1, 0, 1, -2, 1, 0]
+
+
+@pytest.mark.skip_engine("stanza_en")


should this be @pytest.mark.skip_engine("stanza_de")?

omri374 · 2026-09-16T09:09:47Z

+        assert doc.text[token.idx : token.idx + len(token.text)] == token.text
+
+
+@pytest.mark.skip_engine("stanza_en")


Suggested change

@pytest.mark.skip_engine("stanza_en")

@pytest.mark.skip_engine("stanza_de")

omri374 · 2026-09-16T09:09:55Z

+    assert doc.ents[0].end_char == expected_start + len("Thomas Bergmann")
+
+
+@pytest.mark.skip_engine("stanza_en")


Suggested change

@pytest.mark.skip_engine("stanza_en")

@pytest.mark.skip_engine("stanza_de")

omri374

Thanks! The only thing missing is proper handling of downloading Stanza models during unit tests. You'd might need to update the conftest here:
https://github.com/ManoharPaturi/presidio/blob/b71a4b4aeaa889a0f51eb2c7d0612fc1702b1619/presidio-analyzer/tests/conftest.py#L31

To add stanza_de as a preloaded NLP Engine

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com> Co-authored-by: Arjun Sanjay Pakhan <arjunpakhan@gmail.com>

ManoharPaturi · 2026-09-16T15:12:19Z

done, trailer added to the final commit. thanks for the thorough verification pass, that kind of pull-the-branch testing is exactly what the fix needed.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

…erman tests Add stanza_de to nlp_engines fixture and provide stanza_de_nlp_engine. Update German Stanza integration tests to gate on stanza_de engine and use preloaded pipelines from nlp_engines when available. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

ManoharPaturi · 2026-09-17T04:26:15Z

@omri374 Updated! Added stanza_de to the nlp_engines fixture in conftest.py (along with a stanza_de_nlp_engine session fixture), updated stanza_pipeline_de and stanza_pipeline to use the preloaded engine when available, and updated the German tests to gate on @pytest.mark.skip_engine("stanza_de"). Pushed in b6834f6.

ManoharPaturi · 2026-09-18T11:52:21Z

@omri374 following up on this one. the notes file is removed and i added the co-author trailer arjunpakhan asked for. his verification pass plus your main merge are on the thread. anything else needed before this can be merged?

Copilot AI lite review requested due to automatic review settings September 13, 2026 16:57

Copilot AI reviewed Sep 13, 2026

github-actions Bot added documentation analyzer nlp-engine test external labels Sep 13, 2026

omri374 requested a lite review from Copilot September 15, 2026 12:51

Merge branch 'main' into fix/stanza-mwt-offsets

eef519d

omri374 reviewed Sep 15, 2026

View reviewed changes

Copilot started reviewing on behalf of omri374 September 15, 2026 12:52 View session

Copilot AI reviewed Sep 15, 2026

View reviewed changes

Comment thread presidio-analyzer/presidio_analyzer/nlp_engine/stanza_nlp_engine.py

Comment on lines +513 to +517

head = (

offset

+ word_index_to_token_index.get(token.head - 1, token_index)

- len(tokens)

)

Copilot AI review requested due to automatic review settings September 15, 2026 15:37

Copilot AI reviewed Sep 15, 2026

omri374 approved these changes Sep 16, 2026

View reviewed changes

omri374 reviewed Sep 16, 2026

View reviewed changes

omri374 requested changes Sep 16, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 16, 2026 10:54

Copilot AI reviewed Sep 16, 2026

chore: remove development notes file

6bdb5c6

Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com> Co-authored-by: Arjun Sanjay Pakhan <arjunpakhan@gmail.com>

ManoharPaturi force-pushed the fix/stanza-mwt-offsets branch from 3c79b36 to 6bdb5c6 Compare September 16, 2026 15:12

Copilot AI review requested due to automatic review settings September 16, 2026 15:12

Copilot AI reviewed Sep 16, 2026

Copilot AI review requested due to automatic review settings September 17, 2026 04:26

Copilot AI reviewed Sep 17, 2026

Merge branch 'main' into fix/stanza-mwt-offsets

9336080

omri374 approved these changes Sep 18, 2026

View reviewed changes

		@@ -0,0 +1,137 @@
		# Fix: StanzaNlpEngine German multi-word tokens replace the doc text and drop entities (#2249)

		self.sentences = sentences


		def test_get_tokens_with_heads_collapses_mwt_and_remaps_heads():

		assert heads == [1, 0, 1, -2, 1, 0]


		@pytest.mark.skip_engine("stanza_en")

		assert doc.text[token.idx : token.idx + len(token.text)] == token.text


		@pytest.mark.skip_engine("stanza_en")

	@pytest.mark.skip_engine("stanza_en")
	@pytest.mark.skip_engine("stanza_de")

		assert doc.ents[0].end_char == expected_start + len("Thomas Bergmann")


		@pytest.mark.skip_engine("stanza_en")

Conversation

ManoharPaturi commented Sep 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

ArjunPakhan commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ManoharPaturi commented Sep 14, 2026

Uh oh!

omri374 Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

ManoharPaturi commented Sep 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

ArjunPakhan commented Sep 15, 2026

Uh oh!

omri374 left a comment

Choose a reason for hiding this comment

Uh oh!

omri374 Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omri374 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

omri374 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

omri374 Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

omri374 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

ManoharPaturi commented Sep 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

ManoharPaturi commented Sep 17, 2026

Uh oh!

ManoharPaturi commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ArjunPakhan commented Sep 14, 2026 •

edited

Loading

omri374 Sep 16, 2026 •

edited

Loading