Conversation
… is omitted in a registry config A custom deny-list recognizer loaded through RecognizerRegistryProvider got deny_list_score=0.0 when the YAML/dict entry omitted the field, because CustomRecognizerConfig declared default=0.0 and dumped it. The same entry created in code or via add_pattern_recognizer_from_dict gets the PatternRecognizer default of 1.0, and a 0.0 deny-list match never passes a positive score threshold. Default the field to None and dump the custom config with exclude_none=True, like the other pass-through config models, so an omitted score reaches PatternRecognizer as "not set" and an explicit score is honored unchanged. Fixes data-privacy-stack#2242
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The change is narrowly scoped, aligns provider behavior with existing constructor defaults, and is covered by both model-level and provider-path tests.
Pull request overview
This PR fixes a scoring inconsistency for custom deny-list recognizers loaded via RecognizerRegistryProvider by ensuring that an omitted deny_list_score does not get serialized as 0.0 and passed to PatternRecognizer, allowing the constructor default (1.0) to apply consistently across all construction paths.
Changes:
- Update
CustomRecognizerConfig.deny_list_scoretoOptional[float] = Noneand ensureCustomRecognizerConfig.model_dump()excludesNoneby default so omitted YAML keys don’t override constructor defaults. - Add model-level tests to verify omitted vs explicit
deny_list_scoreserialization behavior. - Add provider-level tests to confirm the registry/provider path yields the same deny-list score behavior as in-code
PatternRecognizerconstruction, and update the provider docs accordingly.
File summaries
| File | Description |
|---|---|
| presidio-analyzer/presidio_analyzer/input_validation/yaml_recognizer_models.py | Makes deny_list_score optional and ensures omitted fields don’t get dumped as explicit kwargs. |
| presidio-analyzer/tests/test_yaml_recognizer_models.py | Adds unit tests asserting omitted deny_list_score is absent from dumps and explicit values are preserved. |
| presidio-analyzer/tests/test_recognizer_registry_provider.py | Adds integration tests verifying provider-loaded custom deny-list recognizers use the PatternRecognizer default score when omitted, and honor explicit scores. |
| docs/analyzer/recognizer_registry_provider.md | Documents that deny_list_score is optional and defaults to PatternRecognizer’s 1.0 when omitted. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Description
A custom deny-list recognizer that omits
deny_list_scorecurrently gets a different score depending on how it is loaded:PatternRecognizer(...)andRecognizerRegistry.add_pattern_recognizer_from_dict(...)give the constructor default1.0, whileRecognizerRegistryProvider(YAML / dict registry config) gives0.0, becauseCustomRecognizerConfig.deny_list_scoredeclareddefault=0.0and that default was dumped and passed to the recognizer. A deny-list match with score0.0is filtered by any positivescore_threshold, so a YAML deny list without an explicit score never fired through the provider path.This PR:
CustomRecognizerConfig.deny_list_scoretoOptional[float] = Field(default=None, ge=0.0, le=1.0);CustomRecognizerConfigthe samemodel_dump(exclude_none=True)override the other pass-through config models already have, so an omitted field is left out of the kwargs that reachPatternRecognizer.from_dictand the constructor default applies. OtherOptionalfields on the custom config (patterns,context,supported_language,country_code, ...) already default toNonein the recognizer constructor, and the loader reads them with.get(...), so leaving them out of the dump does not change their handling;docs/analyzer/recognizer_registry_provider.md.Behavior change (existing users): a registry config (YAML or dict) whose custom recognizer has a
deny_listbut nodeny_list_scorenow yields deny-list matches with score1.0instead of0.0, i.e. those terms are now detected instead of being dropped below the threshold. Configs that setdeny_list_scoreexplicitly are unchanged. Detection patterns and scores of the predefined recognizers are not touched.Tests:
deny_list_scoreisNoneand absent from the dump; an explicit value is dumped unchanged;RecognizerRegistryProvider(new-style entry and legacysupported_languageentry): the constructed recognizer hasdeny_list_score == 1.0, its deny-list pattern has score1.0, the same as an in-codePatternRecognizer, andanalyze("Dear Mr. Smith")returns the match with score1.0; an explicitdeny_list_score: 0.4is honored.Ran
uv run pytest tests/test_yaml_recognizer_models.py tests/test_recognizer_registry_provider.py(104 passed; the two*_chunker_configtests fail onmaintoo in an environment without thetransformersextra) andruff check/ruff format --checkon the changed source file.Issue reference
Fixes #2242
Checklist