Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
62f9626
test: add structured OpenShell intake prototype
binaryaaron Aug 17, 2026
a252898
Add private row verification seam
binaryaaron Aug 17, 2026
d85f09c
fix: enforce private row terminal accounting
binaryaaron Aug 17, 2026
b593034
fix: harden private row verification
binaryaaron Aug 17, 2026
3ff6bfc
fix: close private row verification gaps
binaryaaron Aug 17, 2026
b4f316d
Extract private pandas execution seam
codex Aug 18, 2026
519ec5a
fix: harden compiled invocation snapshot
codex Aug 18, 2026
d893cdf
fix: map private verification failures at public boundary
binaryaaron Aug 18, 2026
6d11028
docs: propose extensible SDK companion plans
binaryaaron Aug 18, 2026
e626117
docs: refine companion SDK contracts
binaryaaron Aug 18, 2026
fae06f6
feat: add private Plan A protection spike
binaryaaron Aug 18, 2026
d628a18
fix: tighten private protection release checks
binaryaaron Aug 18, 2026
4990d00
fix malformed private protection segments
binaryaaron Aug 18, 2026
d6c1d9e
chore: tighten draft PR scope
binaryaaron Aug 18, 2026
a77da47
Harden private protection provenance boundary
binaryaaron Aug 18, 2026
92a1f02
fix: suppress private DataDesigner task traces
binaryaaron Aug 18, 2026
61842d5
docs: align companion plan evidence and gates
binaryaaron Aug 18, 2026
e98d813
test: validate Intake formats through private flow
binaryaaron Aug 19, 2026
7cc4760
test: validate protected Intake delivery contracts
binaryaaron Aug 19, 2026
4154400
test: export protected Sandbox sessions to Intake
binaryaaron Aug 19, 2026
8893095
docs: add local Intake dogfood runbook
binaryaaron Aug 19, 2026
4586b82
feat: add private graph-native execution seam
binaryaaron Aug 20, 2026
fbe2aea
docs: replace companion plans with graph proposal
binaryaaron Aug 20, 2026
702f43a
fix: fail closed on incomplete graph redaction
binaryaaron Aug 20, 2026
d7ee655
Implement Phase 4 hierarchical terminal accounting
binaryaaron Aug 25, 2026
bb79cda
Implement target-context workframe execution
binaryaaron Aug 26, 2026
64bcd63
Harden Phase 5 context execution boundaries
binaryaaron Aug 26, 2026
1d24a7e
Harden Phase 5 context cleanup lifecycle
binaryaaron Aug 26, 2026
53ef74f
Polish Phase 5 context cleanup ownership
binaryaaron Aug 26, 2026
5bf61c6
feat: add private anchored mention resolution
binaryaaron Aug 27, 2026
69a8105
fix: enforce the private Phase 6 Redact contract
binaryaaron Aug 27, 2026
2dfaee1
chore: apply current Ruff formatting
binaryaaron Aug 27, 2026
e578840
test: remove optional IPython dependency
binaryaaron Aug 27, 2026
1aa8994
fix: reject coerced Phase 6 provider values
binaryaaron Aug 27, 2026
29bddad
fix: validate frozen Phase 6 role policy
binaryaaron Aug 27, 2026
b1e9805
docs: reconcile graph SDK phase status
binaryaaron Aug 31, 2026
6fcb7da
feat: freeze private Phase 7 contract
binaryaaron Sep 1, 2026
5ea5a29
refactor: add typed accounting task subjects
binaryaaron Sep 1, 2026
77a4edf
feat: add private Phase 6 Substitute handoff
binaryaaron Sep 1, 2026
deb56f6
feat: compile private Phase 7 manifests
binaryaaron Sep 1, 2026
9271f5e
test: freeze private Phase 7 reference model
binaryaaron Sep 1, 2026
6569466
feat: validate private Phase 7 substitutions
binaryaaron Sep 2, 2026
18ae19d
feat: add private Phase 7 candidate backend
binaryaaron Sep 2, 2026
2de3f0f
feat: add private Phase 7 planner ledger
binaryaaron Sep 3, 2026
f47cb2a
feat: add private Phase 7 runtime
binaryaaron Sep 3, 2026
5a7b1b2
feat: wire private stable substitute protection
binaryaaron Sep 3, 2026
4403aa3
test: prove phase 7 privacy convergence
binaryaaron Sep 3, 2026
6508c34
test: format phase 7 privacy allowlists
binaryaaron Sep 3, 2026
2690ed2
fix: bind phase 7 release to accounting
binaryaaron Sep 3, 2026
94b6f11
fix: localize phase 7 application faults
binaryaaron Sep 3, 2026
923d870
fix: seal phase 7 task prerequisites
binaryaaron Sep 3, 2026
7db42f7
fix: attest phase 7 effective readiness
binaryaaron Sep 3, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions STYLEGUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,13 @@ except Exception as exc:
raise AnonymizerWorkflowError("Workflow failed")
```

Privacy-sensitive public boundaries are the narrow exception. When an
underlying exception may contain input values, private correlations, prompts,
or backend details, translate it to a generic canonical interface error after
leaving the active exception handler. The public error must have neither an
accessible ``__cause__`` nor ``__context__``; ``raise ... from None`` inside an
``except`` block only suppresses display and still retains ``__context__``.

Don't use defensive `try/except` on trusted internal calls that shouldn't fail — only catch at module boundaries. `RewriteWorkflow._run_final_judge` is the intentional exception: it's explicitly non-critical and catches broadly, logging with `exc_info=True` and substituting safe defaults.

**Error messages** must identify the actual bad value. Use `!r` to make interpolated values unambiguous:
Expand Down
248 changes: 248 additions & 0 deletions docs/development/anonymizer-performance-and-experimentation.md

Large diffs are not rendered by default.

20 changes: 20 additions & 0 deletions docs/development/extensible-sdk-companion-plans.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

# Extensible SDK planning (superseded)

Status: superseded on 2026-08-20. This path remains as a compatibility landing
page for existing repository links; it is not a second architecture plan.

Use these documents instead:

- [Technical Proposal — Graph-native Anonymizer SDK](graph-native-anonymizer-sdk-technical-proposal.md)
defines the proposed architecture, migration sequence, ownership boundary,
and promotion gates.
- [Evidence — Intake workload validation](intake-workload-validation-evidence.md)
records the published Intake behavior, dated dogfood observations, inferred
Platform compatibility, and unresolved adopter decisions that inform the
proposal.

The repository history retains the former planning report. Do not use that
historical text as current architecture or evidence.
616 changes: 616 additions & 0 deletions docs/development/graph-native-anonymizer-sdk-rfc.md

Large diffs are not rendered by default.

256 changes: 256 additions & 0 deletions docs/development/graph-native-anonymizer-sdk-technical-proposal.md

Large diffs are not rendered by default.

115 changes: 115 additions & 0 deletions docs/development/intake-workload-validation-evidence.md

Large diffs are not rendered by default.

593 changes: 593 additions & 0 deletions docs/development/phase-4-hierarchical-terminal-accounting-design.md

Large diffs are not rendered by default.

697 changes: 697 additions & 0 deletions docs/development/phase-5-target-context-workframe-design.md

Large diffs are not rendered by default.

679 changes: 679 additions & 0 deletions docs/development/phase-6-anchored-mention-resolution-design.md

Large diffs are not rendered by default.

732 changes: 732 additions & 0 deletions docs/development/phase-7-stable-substitute-design.md

Large diffs are not rendered by default.

81 changes: 81 additions & 0 deletions docs/development/plan-a-design-spike.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
<!-- SPDX-License-Identifier: Apache-2.0 -->

# Historical private protection design-spike record

Status: historical branch-local implementation evidence. This record predates
and is subordinate to the current
[graph-native technical proposal](graph-native-anonymizer-sdk-technical-proposal.md);
its former “Plan A” vocabulary is retained only where needed to describe the
implemented checkpoint and test lineage.

This record covers the private, synchronous protection slice only. It does not
authorize a public SDK, Intake integration, release decision, or alternate
engine path.

The Intake team is the named adopter. The project owner accepted the canonical
cause-free `AnonymizerWorkflowError` mapping for pipeline failures that cross
the private row-verification boundary. The customer or consumer PII boundary
remains unresolved and must be recorded by the consuming-product owner before
a production validation placement is selected.

## Ownership verdicts

Ownership: stay; owner=anonymizer.interface._protection._compile_protection_plan; evidence=the helper consumes AnonymizerConfig, ModelSelection, and ModelConfig and returns a distinct private plan; reason=cross-domain release-policy compilation is not a same-type config transform.

Ownership: stay; owner=anonymizer.interface._protection._build_operation_plan; evidence=the helper consumes a compiled plan and protection records and returns an invocation-private operation plan; reason=batch admission bounds and correlation span several domain values.

Ownership: stay; owner=anonymizer.interface._protection._ProtectionFlow._execute; evidence=the lifecycle caller coordinates the pandas runtime, invocation verifier, and terminal accounting; reason=effectful runtime coordination does not belong on a frozen domain value.

Ownership: stay; owner=anonymizer.interface._protection._failure; evidence=the helper constructs one closed safe-failure domain value from static enum and stage inputs; reason=the private protection domain module owns its stable failure taxonomy.

Ownership: stay; owner=anonymizer.interface._protection release-policy helpers; evidence=_has_accepted_detections and _redact_release_passed inspect verified engine entity values to derive a protection disposition and enforce the compiled Redact predicate; reason=these helpers coordinate engine schema values with protection-domain policy rather than transform one config model.

Ownership: stay; owner=anonymizer.interface._protection._SafeRepr; evidence=all new private domain values inherit the content-free rendering mixin; reason=one domain-local rendering policy prevents record content and references from entering repr, logs, or errors.

Ownership: stay; owner=anonymizer.engine.ndd.adapter.NddAdapter.private_execution; evidence=the policy coordinates DataDesigner invocation artifacts and ambient measurement, message-trace, and task-trace collection across all engine workflows while preserving run_workflow as the sole execution boundary; reason=filesystem and collector isolation belong at the DataDesigner adapter boundary, with _ProtectionFlow activating the policy only for private execution.

## Test strategy

Focused contract tests cover private value bounds, closed compilation outcomes,
pre-admission batch rejection, terminal accounting, lifecycle overlap and close,
safe rendering, and adversarial engine results. A focused integration test uses
the synthetic detector and the real local Redact/pandas runtime seam. Tests
assert returned outcomes and absence of private data, not internal call counts.

The flow borrows the facade runtime. Closing it rejects new admission and lets
an already admitted synchronous invocation drain; it never closes the borrowed
DataDesigner or provider resources. The current runtime does not establish hard
cancellation or deterministic dependency teardown, and this spike makes no such
claim. Execution completion remains distinct from external data-handling and
release or commit authority.

The implemented private execution path gives each DataDesigner `create()` call
an invocation-scoped temporary artifact root, loads its output before cleanup,
and disables ambient Anonymizer measurement plus DataDesigner message and task
traces for that scope. `preview()` retains its existing in-memory behavior.
Adapter and flow canaries cover durable-root and collector isolation, backend
exception and failed-record diagnostic safety, and fail-closed terminal
accounting.

The compiled plan deep-copies execution inputs and fingerprints the complete
allowlisted Plan A semantic snapshot, including selected models, model configs,
detection settings, replacement settings, profile, versions, and limits.
Execution verifies that fingerprint before effects. Receipts bind the plan
digest and a fresh content-independent attempt identity. Failure retry safety
is `unknown` and retry ownership is `Unassigned`; this spike makes no stronger
retry claim before taxonomy review.

## Intake-format validation

The synthetic validation corpus covers ATIF v1.0 and v1.7, an
extension-bearing chat-completion request and response, a real OTLP protobuf
batch, and an Intake-shaped local CHAIN-to-LLM trace. Every declared target
runs through the private Plan A flow with deterministic local detection and
`Redact`. Tests verify reconstruction, structural and topology preservation,
closed field handling, and complete-item withholding after invalid OTLP spans
or non-success outcomes.

This is adapter and execution evidence only. It does not run the Intake
service, persist records, call an external provider, use customer data, or
establish ATIF, chat-completion, or OTLP production support.
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ anonymizer-chunked-validation = "anonymizer.engine.workflow_columns.detection.pl

[dependency-groups]
dev = [
"opentelemetry-proto>=1.27.0,<2",
"pre-commit>=4.0.0,<5",
"pytest>=9.0.3,<10",
"pytest-cov>=7.0,<8",
Expand Down
2 changes: 2 additions & 0 deletions src/anonymizer/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
from anonymizer.interface.errors import (
AnonymizerError,
AnonymizerIOError,
AnonymizerWorkflowError,
InvalidConfigError,
InvalidInputError,
)
Expand Down Expand Up @@ -52,6 +53,7 @@ def __getattr__(name: str) -> object:
"AnonymizerError",
"AnonymizerInput",
"AnonymizerIOError",
"AnonymizerWorkflowError",
"Annotate",
"DEFAULT_ENTITY_LABELS",
"Detect",
Expand Down
20 changes: 20 additions & 0 deletions src/anonymizer/engine/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,26 @@
# Input
COL_TEXT = "__nemo_anonymizer_text_input__"

# Private target/context workframes. These columns never enter public results.
COL_TARGET_WORK_ID = "__anonymizer_private_row_correlation__"
COL_TASK_ID = "__anonymizer_private_task_identity__"
COL_ATTEMPT_ID = "__anonymizer_private_attempt_identity__"
COL_CONTEXT_BINDING_ID = "__anonymizer_context_binding_id__"
COL_CONTEXT_OWNER_WORK_ID = "__anonymizer_context_owner_work_id__"
COL_CONTEXT_ORDINAL = "__anonymizer_context_ordinal__"
COL_CONTEXT_TEXT = "__anonymizer_context_text__"

# Private Phase 6 provider workframes. These columns never enter public results.
COL_PHASE6_CONTEXT = "__anonymizer_phase6_context__"
COL_PHASE6_CANDIDATES = "__anonymizer_phase6_candidates__"
COL_PHASE6_AUGMENTED = "__anonymizer_phase6_augmented__"
COL_PHASE6_VALIDATION = "__anonymizer_phase6_validation__"

# Private Phase 7 candidate workframes. These columns never enter public results.
COL_PHASE7_INVOCATION_ID = "__anonymizer_phase7_invocation_identity__"
COL_PHASE7_CANDIDATE_REQUEST = "__anonymizer_phase7_candidate_request__"
COL_PHASE7_CANDIDATE_BUNDLE = "__anonymizer_phase7_candidate_bundle__"

# Step 1: GLiNER detection
COL_RAW_DETECTED = "_raw_detected_entities"

Expand Down
65 changes: 48 additions & 17 deletions src/anonymizer/engine/detection/detection_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

import logging
from copy import deepcopy
from dataclasses import dataclass
from dataclasses import dataclass, field
from pathlib import Path
from typing import cast

Expand Down Expand Up @@ -42,7 +42,7 @@
_jinja,
)
from anonymizer.engine.detection.postprocess import EntitySpan, group_entities_by_value
from anonymizer.engine.ndd.adapter import FailedRecord, NddAdapter
from anonymizer.engine.ndd.adapter import FailedRecord, NddAdapter, _FailedRowEvidence
from anonymizer.engine.ndd.model_loader import resolve_model_alias, resolve_model_aliases
from anonymizer.engine.prompt_utils import substitute_placeholders
from anonymizer.engine.schemas import (
Expand Down Expand Up @@ -75,6 +75,7 @@
class EntityDetectionResult:
dataframe: pd.DataFrame
failed_records: list[FailedRecord]
failed_row_evidence: tuple[_FailedRowEvidence, ...] = field(default=(), repr=False)


class EntityDetectionWorkflow:
Expand Down Expand Up @@ -123,7 +124,11 @@ def detect_and_validate_entities(
preview_num_records=preview_num_records,
)
detected_df = detection_result.dataframe.copy()
return EntityDetectionResult(dataframe=detected_df, failed_records=detection_result.failed_records)
return EntityDetectionResult(
dataframe=detected_df,
failed_records=detection_result.failed_records,
failed_row_evidence=detection_result.failed_row_evidence,
)

def _build_detection_spec(
self,
Expand Down Expand Up @@ -347,7 +352,11 @@ def identify_latent_entities(
workflow_name="latent-entity-detection",
preview_num_records=preview_num_records,
)
return EntityDetectionResult(dataframe=latent_result.dataframe, failed_records=latent_result.failed_records)
return EntityDetectionResult(
dataframe=latent_result.dataframe,
failed_records=latent_result.failed_records,
failed_row_evidence=latent_result.failed_row_evidence,
)

def run(
self,
Expand Down Expand Up @@ -407,9 +416,14 @@ def run(
)
final_df = latent_result.dataframe.copy()
final_failures = [*detected_result.failed_records, *latent_result.failed_records]
final_failure_evidence = (
*detected_result.failed_row_evidence,
*latent_result.failed_row_evidence,
)
else:
final_df = detected_result.dataframe.copy()
final_failures = detected_result.failed_records
final_failure_evidence = detected_result.failed_row_evidence

# When entity_labels is explicitly provided (even if it matches DEFAULT_ENTITY_LABELS),
# the augmenter is strict and out-of-scope labels are filtered.
Expand All @@ -425,6 +439,7 @@ def run(
result = EntityDetectionResult(
dataframe=final_df,
failed_records=final_failures,
failed_row_evidence=final_failure_evidence,
)
measurement.update(
output_row_count=len(result.dataframe),
Expand All @@ -440,19 +455,35 @@ def _inject_detector_params(
labels: list[str],
gliner_detection_threshold: float,
) -> list[ModelConfig]:
resolved = deepcopy(model_configs)
for config in resolved:
if config.alias != selected_models.entity_detector:
continue
if config.inference_parameters.extra_body is None:
config.inference_parameters.extra_body = {}
config.inference_parameters.extra_body["labels"] = labels
config.inference_parameters.extra_body["threshold"] = gliner_detection_threshold
config.inference_parameters.extra_body["chunk_length"] = 384
config.inference_parameters.extra_body["overlap"] = 128
config.inference_parameters.extra_body["flat_ner"] = False
break
return resolved
return _inject_detector_params(
model_configs=model_configs,
selected_models=selected_models,
labels=labels,
gliner_detection_threshold=gliner_detection_threshold,
)


def _inject_detector_params(
*,
model_configs: list[ModelConfig],
selected_models: DetectionModelSelection,
labels: list[str],
gliner_detection_threshold: float,
) -> list[ModelConfig]:
"""Return detached GLiNER model configs for one detector workflow."""
resolved = deepcopy(model_configs)
for config in resolved:
if config.alias != selected_models.entity_detector:
continue
if config.inference_parameters.extra_body is None:
config.inference_parameters.extra_body = {}
config.inference_parameters.extra_body["labels"] = labels
config.inference_parameters.extra_body["threshold"] = gliner_detection_threshold
config.inference_parameters.extra_body["chunk_length"] = 384
config.inference_parameters.extra_body["overlap"] = 128
config.inference_parameters.extra_body["flat_ner"] = False
break
return resolved


def _resolve_detection_labels(entity_labels: list[str] | None) -> list[str]:
Expand Down
4 changes: 4 additions & 0 deletions src/anonymizer/engine/execution/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Private pandas execution coordination for the public interface."""
Loading
Loading