feat(analyzer): report a regex capture group instead of the whole match

docs/analyzer/adding_recognizers.md

@@ -98,6 +98,42 @@ engine = AnalyzerEngine(registry=registry)
 engine.analyze(...)
 ```
 
+### Detecting part of a regex match
+
+By default, a pattern detects the whole regex match.
+To detect only part of it, such as a password without the `password:` label before it,
+set `capture_group` on the `Pattern` to the number or the name of a capture group in the regex:
+
+```python
+from presidio_analyzer import Pattern, PatternRecognizer
+
+password_pattern = Pattern(
+    name="password (value only)",
+    regex=r"password:\s*(?P<value>\S+)",
+    score=0.5,
+    capture_group="value",
+)
+password_recognizer = PatternRecognizer(
+    supported_entity="PASSWORD", patterns=[password_pattern]
+)
+
+# Detects "hunter2" (start=13, end=20) instead of "password: hunter2"
+results = password_recognizer.analyze(text="my password: hunter2", entities=["PASSWORD"])
+print(results)
+```
+
+Notes:
+
+- `capture_group` is optional. It accepts a group number (`0` is the whole match) or a group name, and is checked against the regex when the `Pattern` is created.
+- This check does not apply regex flags. If the flags used at analysis time change the groups of the regex (for example, with `re.VERBOSE` the text after `#` is a comment), a pattern whose group no longer exists detects nothing and logs a warning, and a group number can refer to a different group. Group names are never renumbered, so prefer them in this case.
+- Matches in which the group does not participate, such as an optional group that did not match, are skipped.
+- A pattern detects a single group. To detect several groups of the same regex, define one pattern per group.
+- `validate_result` and `invalidate_result` receive the text of the group.
+- Words matched by the regex outside the group count as surrounding text, so they can raise the score of the result if they are context words of the recognizer.
+- Recognizers that replace the pattern matching logic of `PatternRecognizer` (for example, `IbanRecognizer`) ignore `capture_group`.
+
+The same field can be set in [ad-hoc recognizers](#creating-ad-hoc-recognizers) and in [recognizers loaded from YAML](./recognizer_registry_provider.md#the-recognizer-parameters).
+
 ### Creating a new `EntityRecognizer` in code
 
 To create a new recognizer via code:

docs/analyzer/recognizer_registry_provider.md

@@ -105,7 +105,19 @@ The recognizer list comprises of both the predefined and custom recognizers, for
   In addition to the language code, this field also contains a list of context words, which increases confidence in the detection in case it is found in the surroundings of a detected entity (as seen in the credit card example above).
   - `type`: this could be either predefined or custom. As this is optional, if not stated otherwise, the default type is custom.
   - `name`: Different per the type of the recognizer. For predefined recognizers, this is the class name as defined in presidio, while for custom recognizers, it will be set as the name of the recognizer.
-  - `patterns`: a list of objects of type `Pattern` that contains a name, score and regex that define matching patterns.
+  - `patterns`: a list of objects of type `Pattern` that contains a name, score and regex that define matching patterns. A pattern can also set an optional `capture_group`: the number or the name of a capture group in the regex. When set, only the text matched by this group is detected instead of the whole match, and matches in which the group does not participate are skipped. An unknown group fails validation when the configuration is loaded. This check does not apply `global_regex_flags`. See [detecting part of a regex match](./adding_recognizers.md#detecting-part-of-a-regex-match). Example:
+
+    ```yaml
+    - name: PasswordRecognizer
+      type: custom
+      supported_language: en
+      supported_entity: PASSWORD
+      patterns:
+        - name: password (value only)
+          regex: "password:\\s*(?P<value>\\S+)"
+          score: 0.5
+          capture_group: value
+    ```
   - `enabled`: enables or disables the recognizer.
   - `supported_entity`: the detected entity associated by the recognizer.
   - `deny_list`: A list of words to detect, in case the recognizer uses a predefined list of words.

docs/api-docs/api-docs.yml

@@ -570,6 +570,12 @@ components:
           type: number
           format: double
           description: "Detection confidence of this pattern (0.01 if very noisy, 0.6-1.0 if very specific)"
+        capture_group:
+          oneOf:
+            - type: integer
+              minimum: 0
+            - type: string
+          description: "Number or name of a capture group in the regex. If set, only the text matched by this group is returned as the detected entity, and matches in which the group does not participate are skipped. If omitted, the whole match is returned. The group is checked against the regex without regex flags: if the flags used at analysis time (for example re.VERBOSE) remove the group, the pattern detects nothing."
 
 
     PatternRecognizer:

e2e-tests/tests/test_api_analyzer.py

@@ -444,6 +444,45 @@ def test_given_ad_hoc_pattern_recognizer_context_raises_confidence():
     )
 
 
+@pytest.mark.api
+def test_given_ad_hoc_pattern_recognizer_with_capture_group_then_only_the_group_is_returned():
+    request_body = r"""
+     {
+         "text": "John Smith drivers license is AC432223. Zip code: 10023",
+         "language": "en",
+         "ad_hoc_recognizers":[
+             {
+                "name": "Zip code Recognizer",
+                "supported_language": "en",
+                "patterns": [
+                    {
+                    "name": "zip code (after label)",
+                    "regex": "zip code: (\\d{5})",
+                    "score": 0.01,
+                    "capture_group": 1
+                    }
+                ],
+                "supported_entity":"ZIP"
+            }
+        ]
+     }
+     """
+
+    response_status, response_content = analyze(request_body)
+
+    expected_response = """
+     [
+         {"entity_type": "PERSON", "start": 0, "end": 10, "score": 0.85, "analysis_explanation":null},
+         {"entity_type": "US_DRIVER_LICENSE", "start": 30, "end": 38, "score": 0.6499999999999999, "analysis_explanation":null},
+         {"entity_type": "ZIP", "start": 50, "end": 55, "score": 0.01, "analysis_explanation":null}
+     ]
+     """
+    assert response_status == 200
+    assert equal_json_strings(
+        expected_response, response_content
+    )
+
+
 @pytest.mark.api
 def test_given_ad_hoc_deny_list_recognizer_the_right_entities_are_returned():
     request_body = r"""

presidio-analyzer/presidio_analyzer/input_validation/yaml_recognizer_models.py

@@ -4,6 +4,7 @@
 
 from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
 
+from presidio_analyzer import Pattern
 from presidio_analyzer.input_validation import validate_language_codes
 from presidio_analyzer.recognizer_registry.recognizers_loader_utils import (
     PredefinedRecognizerNotFoundError,
@@ -446,6 +447,12 @@ def validate_patterns(cls, patterns: Optional[List[Dict]]) -> Optional[List[Dict
                 raise ValueError(f"Pattern score should be a float: {pattern}")
             if not (0.0 <= pattern["score"] <= 1.0):
                 raise ValueError(f"Pattern score should be between 0 and 1: {pattern}")
+            if pattern.get("capture_group") is not None:
+                # Build the Pattern to check the group against the compiled regex
+                try:
+                    Pattern.from_dict(pattern)
+                except (TypeError, ValueError) as e:
+                    raise ValueError(f"Invalid pattern {pattern['name']!r}: {e}") from e
         return patterns
 
     @model_validator(mode="after")

presidio-analyzer/presidio_analyzer/pattern.py

@@ -1,5 +1,5 @@
 import json
-from typing import Dict
+from typing import Dict, Mapping, Optional, Union
 
 import regex as re
 
@@ -11,26 +11,69 @@ class Pattern:
     :param name: the name of the pattern
     :param regex: the regex pattern to detect
     :param score: the pattern's strength (values varies 0-1)
+    :param capture_group: optional number or name of a capture group in the
+    regex. If set, the span of this group is detected instead of the whole
+    match, and matches in which the group does not participate are skipped.
+    Defaults to None (the whole match).
     """
 
-    def __init__(self, name: str, regex: str, score: float):
+    def __init__(
+        self,
+        name: str,
+        regex: str,
+        score: float,
+        capture_group: Optional[Union[int, str]] = None,
+    ):
         self.name = name
         self.regex = regex
         self.score = score
+        self.capture_group = capture_group
         self.compiled_regex = None
         self.compiled_with_flags = None
 
-        self.__validate_regex(self.regex)
+        self.__validate_regex(self.regex, self.capture_group)
         self.__validate_score(self.score)
 
     @staticmethod
-    def __validate_regex(pattern: str) -> None:
-        """Validate that the regex pattern is valid."""
+    def __validate_regex(
+        pattern: str, capture_group: Optional[Union[int, str]]
+    ) -> None:
+        """Validate that the regex pattern is valid and defines the capture group."""
         try:
-            re.compile(pattern)
+            compiled_regex = re.compile(pattern)
         except re.error as e:
             raise ValueError(f"Invalid regex pattern: {e}")
 
+        if capture_group is not None:
+            Pattern.__validate_capture_group(
+                capture_group, compiled_regex.groups, compiled_regex.groupindex
+            )
+
+    @staticmethod
+    def __validate_capture_group(
+        capture_group: Union[int, str], groups: int, groupindex: Mapping[str, int]
+    ) -> None:
+        if isinstance(capture_group, bool) or not isinstance(capture_group, (int, str)):
+            raise ValueError(
+                "capture_group must be an int or a str, "
+                f"got {type(capture_group).__name__}"
+            )
+        if isinstance(capture_group, str):
+            if capture_group not in groupindex:
+                raise ValueError(
+                    f"capture_group {capture_group!r} is not a named group "
+                    f"in the regex. Named groups: {list(groupindex)}"
+                )
+        elif capture_group < 0:
+            raise ValueError(
+                f"capture_group must be a non-negative integer, got {capture_group}"
+            )
+        elif capture_group > groups:
+            raise ValueError(
+                f"capture_group {capture_group} is out of range: "
+                f"regex defines {groups} capture group(s)"
+            )
+
     @staticmethod
     def __validate_score(score: float) -> None:
         if score < 0 or score > 1:
@@ -45,6 +88,8 @@ def to_dict(self) -> Dict:
         :return: a dictionary
         """
         return_dict = {"name": self.name, "score": self.score, "regex": self.regex}
+        if self.capture_group is not None:
+            return_dict["capture_group"] = self.capture_group
         return return_dict
 
     @classmethod

presidio-analyzer/presidio_analyzer/pattern_recognizer.py

@@ -1,7 +1,7 @@
 import datetime
 import logging
 import os
-from typing import TYPE_CHECKING, Dict, List, Optional
+from typing import TYPE_CHECKING, Dict, List, Optional, Union
 
 import regex as re
 
@@ -139,6 +139,7 @@ def validate_result(self, pattern_text: str) -> Optional[bool]:
 
         :param pattern_text: the text to validated.
         Only the part in text that was detected by the regex engine
+        (the pattern's capture group if set, otherwise the whole match)
         :return: A bool indicating whether the validation was successful.
         """
         return None
@@ -151,6 +152,7 @@ def invalidate_result(self, pattern_text: str) -> Optional[bool]:
 
         :param pattern_text: the text to validated.
         Only the part in text that was detected by the regex engine
+        (the pattern's capture group if set, otherwise the whole match)
         :return: A bool indicating whether the result is invalidated
         """
         return None
@@ -190,6 +192,12 @@ def build_regex_explanation(
         )
         return explanation
 
+    @staticmethod
+    def __regex_has_group(compiled_regex, capture_group: Union[int, str]) -> bool:
+        if isinstance(capture_group, str):
+            return capture_group in compiled_regex.groupindex
+        return capture_group <= compiled_regex.groups
+
     def __analyze_patterns(
         self, text: str, flags: int = None
     ) -> List[RecognizerResult]:
@@ -212,6 +220,19 @@ def __analyze_patterns(
                 pattern.compiled_with_flags = flags
                 pattern.compiled_regex = re.compile(pattern.regex, flags=flags)
 
+            # Flags such as re.VERBOSE can change the groups in a regex
+            if pattern.capture_group is not None and not self.__regex_has_group(
+                pattern.compiled_regex, pattern.capture_group
+            ):
+                logger.warning(
+                    "Regex pattern '%s' has no capture group %r "
+                    "when compiled with the regex flags in use, skipping.",
+                    pattern.name,
+                    pattern.capture_group,
+                )
+                continue
+            group = 0 if pattern.capture_group is None else pattern.capture_group
+
             try:
                 matches = pattern.compiled_regex.finditer(
                     text, timeout=REGEX_TIMEOUT_SECONDS
@@ -224,10 +245,11 @@ def __analyze_patterns(
                 )
 
                 for match in matches:
-                    start, end = match.span()
+                    start, end = match.span(group)
                     current_match = text[start:end]
 
-                    # Skip empty results
+                    # Skip empty results, including a capture group that did not
+                    # participate in the match (its span is (-1, -1))
                     if current_match == "":
                         continue
 

presidio-analyzer/tests/test_analyzer_request.py

@@ -190,6 +190,34 @@ def test_analyzer_request_with_ad_hoc_recognizers(self):
         assert isinstance(request.ad_hoc_recognizers[0], PatternRecognizer)
         assert request.ad_hoc_recognizers[0].supported_entities == ["CUSTOM_ID"]
 
+    def test_analyzer_request_with_ad_hoc_recognizer_capture_group(self):
+        """Test that a pattern capture_group reaches the ad-hoc recognizer."""
+        req_data = {
+            "text": "Employee ID-12345",
+            "language": "en",
+            "ad_hoc_recognizers": [
+                {
+                    "supported_entity": "CUSTOM_ID",
+                    "supported_language": "en",
+                    "patterns": [
+                        {
+                            "name": "id_pattern",
+                            "regex": r"ID-(\d{5})",
+                            "score": 0.8,
+                            "capture_group": 1
+                        }
+                    ]
+                }
+            ]
+        }
+
+        request = AnalyzerRequest(req_data)
+
+        recognizer = request.ad_hoc_recognizers[0]
+        assert recognizer.patterns[0].capture_group == 1
+        results = recognizer.analyze(request.text, ["CUSTOM_ID"])
+        assert [(result.start, result.end) for result in results] == [(12, 17)]
+
     def test_analyzer_request_without_ad_hoc_recognizers(self):
         """Test that ad_hoc_recognizers is empty list when not provided."""
         req_data = {

presidio-analyzer/tests/test_context_support.py

@@ -216,3 +216,47 @@ def test_when_context_custom_recognizer_then_succeed(spacy_nlp_engine, mock_nlp_
     assert len(results_without_context) == len(results_with_context)
     for res_wo, res_w in zip(results_without_context, results_with_context):
         assert res_wo.score < res_w.score
+
+
+def test_when_capture_group_narrows_span_then_matched_words_before_group_are_context(
+    spacy_nlp_engine, lemma_context
+):
+    """Words matched by the regex before the capture group act as context.
+
+    The result starts at the capture group, so "password" (matched by the
+    regex but outside the group) is a preceding word for context enhancement.
+    """
+    text = "my password: hunter2"
+    patterns = [
+        Pattern("password value", r"password:\s*(\S+)", 0.4, capture_group=1)
+    ]
+    recognizer_with_context = PatternRecognizer(
+        supported_entity="PASSWORD", patterns=patterns, context=["password"]
+    )
+    recognizer_without_context = PatternRecognizer(
+        supported_entity="PASSWORD", patterns=patterns
+    )
+    nlp_artifacts = spacy_nlp_engine.process_text(text, "en")
+
+    results_with_context = lemma_context.enhance_using_context(
+        text,
+        recognizer_with_context.analyze(text, ["PASSWORD"], nlp_artifacts),
+        nlp_artifacts,
+        [recognizer_with_context],
+    )
+    results_without_context = lemma_context.enhance_using_context(
+        text,
+        recognizer_without_context.analyze(text, ["PASSWORD"], nlp_artifacts),
+        nlp_artifacts,
+        [recognizer_without_context],
+    )
+
+    assert [(result.start, result.end) for result in results_with_context] == [
+        (13, 20)
+    ]
+    assert results_without_context[0].score == 0.4
+    assert results_with_context[0].score == pytest.approx(0.75)
+    assert (
+        results_with_context[0].analysis_explanation.supportive_context_word
+        == "password"
+    )