feat(detection): add regex and validator entity detection

[lipikaramaswamy]

Priority Level

Medium (Nice to have)

Is your feature request related to a problem?

NeMo Anonymizer currently relies on GLiNER, LLM validation, and LLM augmentation for entity detection. This works well for contextual entities, but users cannot express deterministic formats through supported configuration, and strongly structured identifiers still depend on model behavior.

Several entity classes have conservative candidate patterns and deterministic validity checks. Users also need a supported way to detect organization-specific identifiers by supplying an entity label, regular expression, and optional local validator.

Describe the solution you'd like

Add regex-backed entity detection as a native seed source alongside GLiNER.

• Provide a curated registry of built-in recognizers. Built-ins are enabled by default and run only when their output labels are in the effective detection label set. Detect(builtin_regexes=False) explicitly disables them.
• Begin with benchmark-backed recognizers for credit_debit_card, email, ipv4, ipv6, mac_address, and url.
• Allow built-in recognizers to combine candidate regexes with local parsers, structural checks, or checksums based on documented standards.
• Add public RegexRule configuration with a label, pattern, optional validator, and validate_with_llm setting.
• Accept a direct validator callable for local Python execution. Serialized and exported workflows use stable validator names supplied by trusted installed packages.
• Default validate_with_llm=True for every rule. When set to False, a candidate is accepted after regex and local validation; with no local validator, the regex match is authoritative.
• Merge regex and GLiNER candidates with deterministic duplicate, overlap, stable-ID, ordering, provenance, and validation-route semantics.
• Reject invalid or empty-matching patterns during configuration and bound regex execution and candidate counts at runtime.

Built-in usage requires no new rule configuration:

config = AnonymizerConfig(
    detect=Detect(entity_labels=["email", "ipv4", "credit_debit_card"]),
    replace=Redact(),
)

Custom rule usage:

def validate_support_case(candidate: RegexCandidate) -> bool:
    return not candidate.groups["number"].startswith("0000")


config = AnonymizerConfig(
    detect=Detect(
        entity_labels=["email", "support_case_id"],
        regex_rules=[
            RegexRule(
                label="support_case_id",
                pattern=r"(?<![A-Z0-9])CASE-(?P<number>[0-9]{8})(?![A-Z0-9])",
                validator=validate_support_case,
                validate_with_llm=False,
            )
        ],
    ),
    replace=Redact(),
)

Custom validator callables receive a typed candidate containing the original value, offsets, named capture groups, bounded context, and rule identity. They return either bool or a typed result containing a rejection reason and optional normalized value. The original source value and offsets remain authoritative.

Whether a decorator should optionally attach a stable validator name and version will be discussed during PR review. Direct callables remain the primary local Python experience.

Acceptance criteria:

• Public typed configuration supports built-ins, custom label/regex rules, optional validators, and validate_with_llm per rule.
• Built-ins activate only for labels in the effective detection scope; custom-label behavior is defined for implicit and explicit entity_labels.
• Rules with validate_with_llm=True enter the existing contextual validation path; rules with False bypass that payload after local acceptance.
• Direct callable validators work locally. Exported workflows resolve stable validator names from installed packages and fail preflight when a validator is unavailable.
• An exact same-label/span deterministic acceptance is not forced back through LLM validation by a duplicate GLiNER candidate.
• Configuration rejects malformed, duplicate, empty, and empty-matching rules with errors identifying the offending value.
• Runtime safeguards cover pathological patterns and excessive candidate production.
• Behavior-focused tests use fabricated data and cover matching, built-in and custom validation, both LLM routes, merge conflicts, serialization, workflow integration, and unchanged behavior when built-ins are disabled.
• Boundary tests cover structured values directly adjacent to Han characters without relying on whitespace or Unicode \b behavior.
• Quality evaluation reports per-label precision and recall, detector disagreement, language/script/locale slices, latency, and LLM candidate/call volume.
• Public configuration and detection documentation are updated, including geography, script, and standards caveats.
• The bundled anonymizer skill is updated because the public Detect surface changes.

Describe alternatives you've considered

• Continue using GLiNER and LLM augmentation alone. This does not provide deterministic support for structured or organization-specific identifiers.
• Merge the benchmark implementation from PR feat: add regex-backed detection benchmark strategies #183. That work is intentionally benchmark-focused and does not provide the public configuration or production integration contract.
• Treat validator callables as directly serializable data. The selected design separates local callable execution from stable installed identifiers used by exported workflows.
• Replace GLiNER for regex-covered labels. Keeping both sources initially provides disagreement evidence and model recovery of unusual valid formats.

Additional context

• PR feat: add regex-backed detection benchmark strategies #183 explores regex-backed benchmark strategies and provides reusable test ideas.
• PR docs: add devnote for hybrid entity detection #258 documents the existing hybrid detection pipeline and benchmark context.
• Issue Expose per-label detection examples in the public configuration API #259 proposes per-label prompt examples; examples complement deterministic rules but do not enforce a format.
• The internal validated-pii-benchmark provides deterministic curation evidence and fixtures for card, email, IPv4, IPv6, MAC, and URL validation.
• The Chinese GLiNER findings guide motivates non-whitespace boundary tests, exact-offset checks, and language/script/locale evaluation slices.

The implementation plan is stored at plans/262/hybrid-regex-detection.md and will be linked from the implementation PR. It cites primary documentation for recognizer design and each built-in validator.