feat(analyzer): report a regex capture group instead of the whole match

[v0ropaev]

Change Description

Adds an optional capture_group to Pattern, so a PatternRecognizer can report one capture group instead of the whole regex match. For example password:\s*(?P<value>\S+) with capture_group="value" detects hunter2, not password: hunter2.

Root cause. PatternRecognizer.__analyze_patterns always takes match.span(), so whatever a regex needs as an anchor ends up inside the entity. The only ways around it today are a variable-width lookbehind or a subclass with its own matching loop (which is what the reporter of #1120 ended up doing).

What changes

Surface	Change
Pattern	new kwarg capture_group: Optional[Union[int, str]] = None, a group number or name. Checked at construction against the compiled regex: bool, negative, out-of-range and unknown names raise ValueError. Messages never include matched text
Pattern.to_dict	emits capture_group only when it is set, so serialised output of existing patterns stays byte-identical (checked for all 108 predefined pattern recognizers)
PatternRecognizer	uses match.span(capture_group). Matches where the group doesn't participate are dropped by the existing empty-match check, and validate_result / invalidate_result receive the group text
CustomRecognizerConfig (YAML)	validates capture_group at parse time when it is present. YAML without it is validated exactly as before
ad-hoc REST recognizers	work through the existing from_dict, no loader change
docs	api-docs.yml Pattern schema, a "Detecting part of a regex match" section in adding_recognizers.md, recognizer_registry_provider.md

Behaviour changes

• None unless capture_group is set: match.span(0) is match.span(), and an /analyze response with return_decision_process is identical to main.
• Configurations that set capture_group used to fail (TypeError in Pattern.__init__, HTTP 400 for ad-hoc recognizers) and now work.
• With a narrowed span, words the regex matched before the group become surrounding text, so a label like password: that is also a context word raises the score. Documented and covered by a test.
• The group is validated against the regex without flags. If the flags used at analysis time remove it (e.g. re.VERBOSE turning #... into a comment), the pattern is skipped with a warning (pattern name and group only), once per analyze call and regardless of the input text. A group number can also shift under such flags, so the docs recommend named groups there.
• AnalysisExplanation is unchanged, since the score formula is unchanged.

Design decisions

• One group per pattern. Improve PatternRecognizer to respect multiple match groups #739 (reporting every capture group) was closed as not planned, and reading groups implicitly would silently change results for the many regexes that wrap the whole pattern in ( ... ), including the zip example in the docs. Several groups → several patterns on the same regex.
• capture_group rather than use_group from the issue: I think it reads better in YAML and JSON. Happy to rename.
• Why not just lookbehind. The regex module does support (?<=password:\s*)\S+, and a few predefined recognizers use exactly that. Still, a group keeps the regex portable and readable, is settable from YAML/REST without lookbehind tricks, fails early with a clear message, and hands the validation hooks only the value.

Not in scope

• Recognizers that replace the matching loop (IbanRecognizer) ignore capture_group. Documented.
• Flag-aware validation. The flags come from the recognizer, the registry config or the request, and Pattern doesn't see any of them at construction.
• The HTTP status for an invalid capture_group in an ad-hoc recognizer is 500, same as an invalid score today (ValueError → 500 in app.py).

Verification

In presidio-analyzer, Python 3.12, uv sync --locked --all-extras --group dev plus en_core_web_lg / en_core_web_sm:

• pytest tests/test_pattern.py tests/test_pattern_recognizer.py tests/test_yaml_recognizer_models.py tests/test_recognizer_registry_provider.py tests/test_analyzer_request.py tests/test_context_support.py: 211 passed. The same tests against main's sources: 31 failed.
• Full suite the way CI runs it (pytest --cov=presidio_analyzer): 3498 passed, 13 skipped (main: 3463 passed, 13 skipped; the difference is the 35 new tests).
• diff-cover coverage.xml --compare-branch=origin/main --fail-under=90: 41 changed lines, 100%.
• ruff check with ruff 0.9.2 as in CI: clean.
• e2e tests/test_api_analyzer.py -k ad_hoc against a local analyzer started with the Dockerfile's conf: 5 passed (the new test gets a 400 on main). I didn't run the Docker e2e itself.
• api-docs.yml passes openapi-spec-validator.
• Not run locally: the 3.10 / 3.11 / 3.13 / 3.14 legs and the mkdocs build.

The new tests cover spans for a number, a name, 0 and unset; non-participating and empty groups; what the hooks receive; two patterns on one regex; the flags warning (logged once per call, no PII in the log); the to_dict / from_dict round trip; YAML through RecognizerRegistryProvider; ad-hoc request dicts; and the context-word interaction.

Issue reference

Fixes #1120

Checklist

• I have reviewed the contribution guidelines
• I agree to follow this project's Code of Conduct
• I confirm that I have the right to submit this contribution and that it does not knowingly contain proprietary or confidential code.
• My code includes unit tests
• All unit tests and lint checks pass locally
• My PR contains documentation updates / additions if required

[Copilot]

🔵 Needs a closer look

A moderate warning-handling issue remains in pattern_recognizer.py.

Pull request overview

Adds optional regex capture-group reporting so recognizers can return a subgroup instead of the full match.

Changes:

• Adds validation and serialization for numeric or named capture_group.
• Applies selected spans across Python, YAML, REST, and analyzer flows.
• Adds tests and documentation while preserving whole-match behavior when unset.
• Moderate finding (1 vote): Flags-incompatible patterns produce no warning when there are no matches; validation should occur before iterating matches.

File summaries

File	Description
presidio-analyzer/tests/test_yaml_recognizer_models.py	Tests YAML capture-group validation.
presidio-analyzer/tests/test_recognizer_registry_provider.py	Tests provider integration.
presidio-analyzer/tests/test_pattern.py	Tests validation and serialization.
presidio-analyzer/tests/test_pattern_recognizer.py	Tests matching behavior and flags.
presidio-analyzer/tests/test_context_support.py	Tests context interaction.
presidio-analyzer/tests/test_analyzer_request.py	Tests ad-hoc recognizers.
presidio-analyzer/presidio_analyzer/pattern.py	Adds capture-group support.
presidio-analyzer/presidio_analyzer/pattern_recognizer.py	Reports selected group spans and handles flags.
presidio-analyzer/presidio_analyzer/input_validation/yaml_recognizer_models.py	Validates configured groups.
e2e-tests/tests/test_api_analyzer.py	Verifies REST behavior.
docs/api-docs/api-docs.yml	Documents the API schema.
docs/analyzer/recognizer_registry_provider.md	Documents YAML configuration.
docs/analyzer/adding_recognizers.md	Documents capture-group usage.

Review details

Suppressed comments (1)

presidio-analyzer/presidio_analyzer/pattern_recognizer.py:232

• Because the invalid-group check is inside for match in matches, a flags-incompatible pattern emits no warning when the whole regex has no matches. The documentation promises a warning for this configuration and once-per-call behavior, so inspect compiled_regex.groups/groupindex before iterating and skip the pattern there; otherwise the misconfiguration remains silent on inputs without a candidate.

                group = 0 if pattern.capture_group is None else pattern.capture_group
                for match in matches:
                    try:
                        start, end = match.span(group)
                    except IndexError:

• Files reviewed: 13/13 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[v0ropaev]

Good catch from the Copilot review (it ended up in the suppressed comments, so replying here): the check for a group removed by the regex flags lived inside for match in matches, so a misconfigured pattern only warned when the text happened to contain a match of the whole regex. On an input without a candidate it stayed silent, which contradicts what the docs promise.

Fixed in ff16851:

• the group is now checked against the compiled regex (groups / groupindex) once per pattern, before finditer, and the pattern is skipped with the same warning. The match loop is back to a plain match.span(group) with no try/except IndexError;
• the check only runs when capture_group is set, so patterns without it never touch the compiled regex's groups (the existing regex-timeout tests patch re.compile with a MagicMock and keep passing);
• test_when_regex_flags_remove_capture_group_then_pattern_is_skipped_with_warning is parametrized over a numbered and a named group and now also asserts the warning on text with no candidate at all. That assertion fails on the previous commit.

Verification: the 6 touched test modules 211 passed; full presidio-analyzer suite with --cov 3498 passed, 13 skipped; diff-cover --fail-under=90 41 changed lines, 100%; ruff check (0.9.2) clean. PR description numbers updated.

[Copilot]

🟢 Approval recommended

No unresolved blocking issues were identified.

Review details

• Files reviewed: 13/13 changed files
• Comments generated: 0 new
• Review effort level: Lite