Skip to content

fix(analyzer): use the PatternRecognizer default when deny_list_score is omitted in a registry config - #2246

Open
L4XB wants to merge 1 commit into
data-privacy-stack:mainfrom
L4XB:fix/deny-list-score-default
Open

L4XB wants to merge 1 commit into
data-privacy-stack:mainfrom
L4XB:fix/deny-list-score-default

Conversation

@L4XB

@L4XB L4XB commented Sep 9, 2026

Copy link
Copy Markdown

Change Description

A custom deny-list recognizer that omits deny_list_score currently gets a different score depending on how it is loaded: PatternRecognizer(...) and RecognizerRegistry.add_pattern_recognizer_from_dict(...) give the constructor default 1.0, while RecognizerRegistryProvider (YAML / dict registry config) gives 0.0, because CustomRecognizerConfig.deny_list_score declared default=0.0 and that default was dumped and passed to the recognizer. A deny-list match with score 0.0 is filtered by any positive score_threshold, so a YAML deny list without an explicit score never fired through the provider path.

This PR:

  • changes CustomRecognizerConfig.deny_list_score to Optional[float] = Field(default=None, ge=0.0, le=1.0);
  • gives CustomRecognizerConfig the same model_dump(exclude_none=True) override the other pass-through config models already have, so an omitted field is left out of the kwargs that reach PatternRecognizer.from_dict and the constructor default applies. Other Optional fields on the custom config (patterns, context, supported_language, country_code, ...) already default to None in the recognizer constructor, and the loader reads them with .get(...), so leaving them out of the dump does not change their handling;
  • documents the default in docs/analyzer/recognizer_registry_provider.md.

Behavior change (existing users): a registry config (YAML or dict) whose custom recognizer has a deny_list but no deny_list_score now yields deny-list matches with score 1.0 instead of 0.0, i.e. those terms are now detected instead of being dropped below the threshold. Configs that set deny_list_score explicitly are unchanged. Detection patterns and scores of the predefined recognizers are not touched.

Tests:

  • model-level: an omitted deny_list_score is None and absent from the dump; an explicit value is dumped unchanged;
  • through RecognizerRegistryProvider (new-style entry and legacy supported_language entry): the constructed recognizer has deny_list_score == 1.0, its deny-list pattern has score 1.0, the same as an in-code PatternRecognizer, and analyze("Dear Mr. Smith") returns the match with score 1.0; an explicit deny_list_score: 0.4 is honored.

Ran uv run pytest tests/test_yaml_recognizer_models.py tests/test_recognizer_registry_provider.py (104 passed; the two *_chunker_config tests fail on main too in an environment without the transformers extra) and ruff check / ruff format --check on the changed source file.

Issue reference

Fixes #2242

Checklist

  • I have reviewed the contribution guidelines
  • I agree to follow this project's Code of Conduct
  • I confirm that I have the right to submit this contribution and that it does not knowingly contain proprietary or confidential code.
  • My code includes unit tests
  • All unit tests and lint checks pass locally
  • My PR contains documentation updates / additions if required

… is omitted in a registry config

A custom deny-list recognizer loaded through RecognizerRegistryProvider
got deny_list_score=0.0 when the YAML/dict entry omitted the field,
because CustomRecognizerConfig declared default=0.0 and dumped it. The
same entry created in code or via add_pattern_recognizer_from_dict gets
the PatternRecognizer default of 1.0, and a 0.0 deny-list match never
passes a positive score threshold.

Default the field to None and dump the custom config with
exclude_none=True, like the other pass-through config models, so an
omitted score reaches PatternRecognizer as "not set" and an explicit
score is honored unchanged.

Fixes data-privacy-stack#2242

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is narrowly scoped, aligns provider behavior with existing constructor defaults, and is covered by both model-level and provider-path tests.

Pull request overview

This PR fixes a scoring inconsistency for custom deny-list recognizers loaded via RecognizerRegistryProvider by ensuring that an omitted deny_list_score does not get serialized as 0.0 and passed to PatternRecognizer, allowing the constructor default (1.0) to apply consistently across all construction paths.

Changes:

  • Update CustomRecognizerConfig.deny_list_score to Optional[float] = None and ensure CustomRecognizerConfig.model_dump() excludes None by default so omitted YAML keys don’t override constructor defaults.
  • Add model-level tests to verify omitted vs explicit deny_list_score serialization behavior.
  • Add provider-level tests to confirm the registry/provider path yields the same deny-list score behavior as in-code PatternRecognizer construction, and update the provider docs accordingly.
File summaries
File Description
presidio-analyzer/presidio_analyzer/input_validation/yaml_recognizer_models.py Makes deny_list_score optional and ensures omitted fields don’t get dumped as explicit kwargs.
presidio-analyzer/tests/test_yaml_recognizer_models.py Adds unit tests asserting omitted deny_list_score is absent from dumps and explicit values are preserved.
presidio-analyzer/tests/test_recognizer_registry_provider.py Adds integration tests verifying provider-loaded custom deny-list recognizers use the PatternRecognizer default score when omitted, and honor explicit scores.
docs/analyzer/recognizer_registry_provider.md Documents that deny_list_score is optional and defaults to PatternRecognizer’s 1.0 when omitted.
Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Omitted deny_list_score yields 0.0 through RecognizerRegistryProvider but 1.0 through PatternRecognizer and add_pattern_recognizer_from_dict

2 participants