fix(analyzer): support per-entity context words (#1711) - #2267
Open
developer-rpai wants to merge 1 commit into
Open
developer-rpai wants to merge 1 commit into
developer-rpai wants to merge 1 commit into
Conversation
) EntityRecognizer.context now also accepts a dict mapping each supported entity type to its own list of context words, so that context words for one entity do not boost detections of another entity from the same recognizer. LemmaContextAwareEnhancer selects the words applicable to each result's entity type; the flat list behavior is unchanged. Recognizer YAML configs accept the same dict form for global and per-language context.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Description
Adds per-entity-type context words, resolving the use case in #1711: a recognizer that supports multiple entities (e.g. one
TransformersRecognizerforACCOUNTNUMBER,CREDIT_CARD,IBAN_CODE,US_ITIN) can now scope its context words per entity type, so "tax id" no longer boostsACCOUNTNUMBERdetections and "account" no longer boostsUS_ITINdetections.EntityRecognizer.contextaccepts a dict mapping each entity type to its own list of words, e.g.context={"ACCOUNTNUMBER": ["account"], "US_ITIN": ["tax", "itin"]}. Malformed configs fail fast in the constructor (TypeError/ValueError).LemmaContextAwareEnhancerselects the words applicable to each result's entity type (_get_context_words_for_entity); entity types with no entry get no boost.BaseRecognizerConfig,CustomRecognizerConfig, per-languageLanguageContextConfig) accept the same dict form forcontext, widened fromList[str].PatternRecognizercontext goes through the same constructor validation.docs/analyzer/index.mddocuments the new form.Issue reference
Fixes #1711
Behavior changes
contextis byte-for-byte unchanged in behavior: the enhancer still boosts when a surrounding word matches any word in the list.context, an entity type with no entry receives no context boost (previously impossible to express).EntityRecognizer(context=...)andPatternRecognizer(context=...)now validate the value at construction and raiseTypeError/ValueErroron malformed input (non-string words, non-dict/list values) instead of failing silently at analysis time.Test evidence
New
presidio-analyzer/tests/test_per_entity_context_words.py(8 tests):None) and constructor validation errors;LemmaContextAwareEnhancer+ spaCyen_core_web_lg: a two-entity recognizer with{"FAKE_ACCOUNT": ["account", "bank"], "FAKE_ITIN": ["tax", "itin"]}boosts only the entity whose words appear nearby (pre-fix: no boost ever fires with a dict context — the 2 boost tests fail onmain);contextdict reaches the constructed recognizer viaRecognizerRegistryProvider.Regression runs, all green on the branch:
tests/test_per_entity_context_words.py— 8 passedtests/test_lemma_context_aware_enhancer.py,tests/test_pattern_recognizer.py,tests/test_recognizer_registry_provider.py,tests/test_context_support.py,tests/test_analyzer_engine.py— all passeduv run ruff check+ruff format— cleanChecklist