Skip to content

fix(analyzer): support per-entity context words (#1711) - #2267

Open
developer-rpai wants to merge 1 commit into
data-privacy-stack:mainfrom
developer-rpai:fix/per-entity-context-words-1711
Open

developer-rpai wants to merge 1 commit into
data-privacy-stack:mainfrom
developer-rpai:fix/per-entity-context-words-1711

Conversation

@developer-rpai

Copy link
Copy Markdown

Change Description

Adds per-entity-type context words, resolving the use case in #1711: a recognizer that supports multiple entities (e.g. one TransformersRecognizer for ACCOUNTNUMBER, CREDIT_CARD, IBAN_CODE, US_ITIN) can now scope its context words per entity type, so "tax id" no longer boosts ACCOUNTNUMBER detections and "account" no longer boosts US_ITIN detections.

  • EntityRecognizer.context accepts a dict mapping each entity type to its own list of words, e.g. context={"ACCOUNTNUMBER": ["account"], "US_ITIN": ["tax", "itin"]}. Malformed configs fail fast in the constructor (TypeError/ValueError).
  • LemmaContextAwareEnhancer selects the words applicable to each result's entity type (_get_context_words_for_entity); entity types with no entry get no boost.
  • Recognizer YAML configs (BaseRecognizerConfig, CustomRecognizerConfig, per-language LanguageContextConfig) accept the same dict form for context, widened from List[str].
  • PatternRecognizer context goes through the same constructor validation.
  • Docs: docs/analyzer/index.md documents the new form.

Issue reference

Fixes #1711

Behavior changes

  • New, opt-in capability only. The flat-list form of context is byte-for-byte unchanged in behavior: the enhancer still boosts when a surrounding word matches any word in the list.
  • With a dict context, an entity type with no entry receives no context boost (previously impossible to express).
  • EntityRecognizer(context=...) and PatternRecognizer(context=...) now validate the value at construction and raise TypeError/ValueError on malformed input (non-string words, non-dict/list values) instead of failing silently at analysis time.

Test evidence

New presidio-analyzer/tests/test_per_entity_context_words.py (8 tests):

  • unit tests for per-entity word selection (dict / flat list / None) and constructor validation errors;
  • integration tests through LemmaContextAwareEnhancer + spaCy en_core_web_lg: a two-entity recognizer with {"FAKE_ACCOUNT": ["account", "bank"], "FAKE_ITIN": ["tax", "itin"]} boosts only the entity whose words appear nearby (pre-fix: no boost ever fires with a dict context — the 2 boost tests fail on main);
  • configuration-path test: a YAML context dict reaches the constructed recognizer via RecognizerRegistryProvider.

Regression runs, all green on the branch:

  • tests/test_per_entity_context_words.py — 8 passed
  • tests/test_lemma_context_aware_enhancer.py, tests/test_pattern_recognizer.py, tests/test_recognizer_registry_provider.py, tests/test_context_support.py, tests/test_analyzer_engine.py — all passed
  • uv run ruff check + ruff format — clean

Checklist

  • I have reviewed the contribution guidelines
  • I agree to follow this project's Code of Conduct
  • I confirm that I have the right to submit this contribution and that it does not knowingly contain proprietary or confidential code.
  • My code includes unit tests
  • All unit tests and lint checks pass locally
  • My PR contains documentation updates / additions if required

)

EntityRecognizer.context now also accepts a dict mapping each
supported entity type to its own list of context words, so that
context words for one entity do not boost detections of another
entity from the same recognizer. LemmaContextAwareEnhancer selects
the words applicable to each result's entity type; the flat list
behavior is unchanged. Recognizer YAML configs accept the same
dict form for global and per-language context.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LemmaContextAwareEnhancer- per entity type context words

1 participant