Expose per-label detection examples in the public configuration API

[lipikaramaswamy]

Priority Level

Medium (Nice to have)

Is your feature request related to a problem?

Entity-label examples are an important quality lever for datasets containing domain-specific formats, such as vendor API keys, account handles, or organization-specific identifiers. Today these examples are stored in the internal anonymizer.engine.constants.ENTITY_LABEL_EXAMPLES mapping.

Users can technically mutate that mapping before calling run() or preview(), but this is not a supported API. It also changes process-global state, can leak between jobs with different requirements, may affect detection, substitution, and evaluation unexpectedly, and does not automatically activate a newly added label in the default detection set.

This came up while tuning detection for developer-agent task transcripts. Narrow positive examples could help distinguish vendor-prefixed API keys and actual account handles, while broad examples such as generic hexadecimal strings, UUIDs, and complete file paths could reinforce false positives such as message_hash values or repository names.

Describe the solution you'd like

Expose per-label positive examples as supported, per-run user configuration. One possible API is:

config = AnonymizerConfig(
    detect=Detect(
        entity_labels=["api_key", "user_name", "email"],
        entity_label_examples={
            "api_key": [
                "sk-ant-api03-abc123def456xyz",
                "OPENAI_API_KEY=sk-proj-abc123def456",
            ],
            "user_name": ["jsmith", "alice.chen", "@trudy-spies"],
        },
    ),
    replace=Annotate(),
)

Suggested behavior:

• Configuration is resolved independently for each run and never mutates module-level defaults.
• User examples merge with built-in examples by default. The merge/replace behavior should be documented explicitly.
• Label names are normalized consistently with Detect.entity_labels.
• When entity_labels is explicit, example-map keys must belong to that list; unmatched keys should produce a clear validation error.
• Adding examples for a new label must not silently add that label to the active detection set.
• Resolved examples are passed to the detection validator and augmenter prompts.
• Any use by substitute or evaluation workflows should be intentional and documented rather than occurring as a side effect of global state.
• Examples are documented as positive examples, not as a mechanism for expressing guaranteed exclusions.

Acceptance criteria:

• A public typed configuration field supports per-label examples.
• Two configurations with different examples can run in the same process without cross-run contamination.
• Validation covers empty labels/examples, normalized duplicate keys, and keys outside an explicit entity_labels list.
• Detection prompt tests verify that configured examples appear only for the corresponding active labels.
• Public documentation includes a dataset-specific example and explains when deterministic exclusions or data_summary are more appropriate.
• The bundled anonymizer agent skill is updated if its public configuration template is affected.

Describe alternatives you've considered

• Mutating anonymizer.engine.constants.ENTITY_LABEL_EXAMPLES. This works only as an internal, process-global workaround and is unsuitable as a supported recommendation.
• Encoding positive examples in data_summary. This mixes dataset context with label ontology and provides no structured validation.
• Using only custom label names. Label names constrain the ontology but do not demonstrate domain-specific value formats.
• Using a larger validation model. This may improve contextual reasoning but increases cost and does not replace explicit domain examples.

Additional context

This complements, rather than replaces, the existing quality controls:

• Explicit entity_labels constrain which labels augmentation may emit.
• data_summary supplies contextual guidance, including negative distinctions, but does not enforce exclusions.
• validation_excerpt_window_chars provides additional surrounding context.
• Deterministic rules remain preferable for guaranteed exclusions such as known non-sensitive JSON fields.

[lipikaramaswamy]

Proposed implementation

Keep the initial scope detection-only. Custom examples should be supplied to the validator and augmenter prompts without changing substitute or evaluation behavior as an accidental side effect.

1. Add the public configuration field

Add a typed field to Detect in src/anonymizer/config/anonymizer_config.py:

entity_label_examples: dict[str, list[str]] = Field(
    default_factory=dict,
    description=(
        "Additional positive examples for entity labels. Examples are merged "
        "with built-in examples for the current run."
    ),
)

Validate and normalize label keys consistently with entity_labels. Reject blank labels, blank examples, and example keys outside an explicitly supplied entity_labels list. Deduplicate examples while preserving order.

Adding examples for a new label must not implicitly activate that label; the label must also appear in entity_labels.

2. Resolve examples per run

Introduce a pure helper that copies the built-in mapping and applies the configured additions:

def resolve_entity_label_examples(
    overrides: dict[str, list[str]],
) -> dict[str, list[str]]:
    resolved = deepcopy(ENTITY_LABEL_EXAMPLES)
    for label, examples in overrides.items():
        resolved[label] = list(
            dict.fromkeys([*resolved.get(label, []), *examples])
        )
    return resolved

The helper must return run-local state and must never mutate ENTITY_LABEL_EXAMPLES or another configuration instance.

3. Thread the resolved mapping through detection

Pass config.detect.entity_label_examples through the existing detection call chain:

Anonymizer
  -> EntityDetectionWorkflow.run()/preview()
  -> build_detection_config()/build_detection_builder_for_seed()
  -> _get_validation_prompt()
  -> _get_augment_prompt()
  -> _format_label_examples()

Update _format_label_examples() to receive its mapping explicitly rather than reading mutable module state:

def _format_label_examples(
    labels: list[str],
    examples_by_label: dict[str, list[str]],
) -> str:
    ...

The external/exported detection-builder paths need the same plumbing so local execution and at-scale execution produce equivalent prompts.

4. Preserve workflow boundaries

For this first change, do not automatically feed detection-specific overrides into LlmReplaceWorkflow, detection judges, or type-fidelity judges. Those workflows currently derive lookup tables from the built-in mapping at module import. If custom examples should later affect those stages, expose and document that behavior intentionally rather than coupling it through global mutation.

5. Tests

Add coverage for:

• configuration serialization and default-factory isolation;
• normalized keys and order-preserving example deduplication;
• rejection of blank labels/examples;
• rejection of keys outside an explicit entity_labels list;
• custom-label examples when the custom label is explicitly active;
• merging additions with built-in examples;
• validator and augmenter prompts containing the resolved examples only for active labels;
• two runs/configurations with different examples not contaminating one another; and
• equivalent propagation through run, preview, and exported detection-builder paths.

Likely test locations include tests/config/test_anonymizer_config.py, the detection prompt/workflow tests, and interface/export tests.

6. Documentation and public-surface follow-up

Document that these are positive examples, not negative constraints. Recommend data_summary for contextual guidance and deterministic rules for guaranteed exclusions. Include an example pairing explicit entity_labels with matching entity_label_examples keys.

Because this changes the public Detect surface, also check and update skills/anonymizer/SKILL.md and its output template as required by the repository's public-API synchronization contract.