diff --git a/STYLEGUIDE.md b/STYLEGUIDE.md index ebfa40cf..83474b1e 100644 --- a/STYLEGUIDE.md +++ b/STYLEGUIDE.md @@ -55,6 +55,13 @@ except Exception as exc: raise AnonymizerWorkflowError("Workflow failed") ``` +Privacy-sensitive public boundaries are the narrow exception. When an +underlying exception may contain input values, private correlations, prompts, +or backend details, translate it to a generic canonical interface error after +leaving the active exception handler. The public error must have neither an +accessible ``__cause__`` nor ``__context__``; ``raise ... from None`` inside an +``except`` block only suppresses display and still retains ``__context__``. + Don't use defensive `try/except` on trusted internal calls that shouldn't fail — only catch at module boundaries. `RewriteWorkflow._run_final_judge` is the intentional exception: it's explicitly non-critical and catches broadly, logging with `exc_info=True` and substituting safe defaults. **Error messages** must identify the actual bad value. Use `!r` to make interpolated values unambiguous: diff --git a/docs/development/anonymizer-performance-and-experimentation.md b/docs/development/anonymizer-performance-and-experimentation.md new file mode 100644 index 00000000..16919f5d --- /dev/null +++ b/docs/development/anonymizer-performance-and-experimentation.md @@ -0,0 +1,248 @@ + + + +# Anonymizer Performance and Experimentation + +*Cross-context optimization program* + +Status: Proposed as part of the complete RFC development and research plan. RFC acceptance permits branch-local experiments under this program; it does not qualify an optimization, approve an execution profile, establish a product SLO, select a deployment form, or authorize production use. + +Canonical architecture: SDK Redesign tab and repository development documents. + +## Decision + +Establish a repeatable performance and experimentation program for Anonymizer across batch, bounded-invocation, inline, and bounded-microbatch workloads. Use the graph-native semantic contracts to test alternative implementations and execution choices without silently changing identity, authorization, accounting, verification, release, privacy claims, or downstream ownership. + +This tab owns hypotheses, workload definitions, measurement methods, experiment records, comparison evidence, and promotion recommendations. The SDK Redesign RFC owns semantic contracts, compilation, task accounting, release rules, compatibility, and the permitted internal extension mechanism. Performance evidence may recommend promotion; it does not authorize a product profile, deployment, privacy claim, public API, or product SLO. + +## Why This Work Is Separate + +Performance is a property of the complete Anonymizer system, not a streaming-only feature. Improvements to detection, models, prompts, stage execution, batching, serialization, resource reuse, or deployment may benefit DataFrame batch calls, one-at-a-time inline calls, bounded microbatches, and future integrations wherever the measured workload assumptions hold. + +The term streaming is intentionally avoided here. A bounded invocation is one finite request whose datums, relationships, limits, and selected contract are known at compilation. A streaming transport would additionally require ordering, backpressure, checkpointing, delivery, retry, recovery, and partial-result semantics; this program does not define those behaviors. + +## Goals + +* Measure end-to-end and per-stage latency, throughput, predictability, resource use, reliability, model use, and quality. +* Support controlled experiments with detectors, models, prompts, parsers, physical stage execution, batching, runtime implementation, and deployment prototypes. +* Make results reproducible against pinned inputs, environments, revisions, profiles, and evidence schemas. +* Require correctness, privacy, compatibility, and failure-behavior gates alongside performance evidence. +* Promote only changes whose supported scope, claim, owner, rollout, rollback, and requalification rules are explicit. +* Reuse improvements across workload contexts only where their measured assumptions and qualified contract apply. + +### Non-Goals + +* Approve a fast profile, regex detector, smaller model, stage consolidation, batching policy, or deployment form in advance. +* Promise a universal latency, throughput, cost, or availability target. +* Treat baseline parity as proof of privacy or detection completeness. +* Create a user-extensible Anonymizer plugin surface or permit live undeclared stage substitution. +* Design a streaming transport or move queues, retries, persistence, reconstruction, retention, or delivery into Anonymizer. +* Replace the existing Anonymizer measurement contract with a second benchmark system. + +## Vocabulary and Claim Classes + +Experiment arm — A benchmark-only configuration used to test a hypothesis. It carries no support or deployment claim. + +Candidate implementation — An implemented alternative that has not passed promotion gates. + +Supported execution profile — A versioned product contract with accepted semantic, correctness, privacy, compatibility, failure, performance, ownership, rollout, rollback, and requalification evidence. + +Same-profile optimization — A change that claims the same supported behavior. It must prove compatibility and non-inferiority plus absolute correctness and privacy gates. + +Equivalent alternative implementation — A different implementation of the same semantic tasks. It requires conformance evidence before substitution. + +Restricted profile — A deliberately narrower entity, input, strategy, quality, or release contract. It must declare unsupported cases and cannot inherit another profile’s privacy claim. + +Router profile — A profile whose compiled graph predeclares eligibility, routing, fallback, and terminal behavior among supported paths and records the actual route. + +Deployment or scheduling experiment — A change to placement, batching, concurrency, or transport overhead that does not by itself change the protection contract. + +Research probe — An investigation that produces evidence but is not eligible for product use. + +## Architecture Guardrails + +Compilation fixes one closed conditional semantic graph and its authorized implementations before invocation effects. Runtime may traverse only predeclared applicability, skip, failover, fallback, or retry edges and must record the selected route and terminal outcome. This statement does not authorize retry, failover, or fallback in profiles whose accepted design excludes them. + +Semantic tasks remain accounting and verification units even when one physical call implements several tasks. Consolidation is eligible only when every task outcome remains keyed, attributable, schema-valid, and independently reconcilable. If safe localization is impossible, failure withholds the wider affected scope under the compiled release contract. + +Every DataDesigner-backed task executes through NddAdapter.run\_workflow(). Pure local implementations do not need DataDesigner, but they participate in the same compiled task, terminal-outcome, verification, and release accounting. + +Phase 4 release semantics remain authoritative: every expected task and datum reaches a terminal outcome; dependencies and flat atomic groups apply fixed-point withholding; group predicates qualify publication; and only released atomic groups contain output. Public failed\_records remains a compatibility surface where applicable and is not graph identity. + +Single-datum, bounded-microbatch, and batch execution normally change scheduling only. They must preserve datum identity, authorized context, semantic outcomes, failure policy, and release scope. Provider output position and DataFrame row order are never identity. + +## Workload Model + +Every benchmark must identify the workload rather than reporting one aggregate performance number. Required dimensions include: + +* Strategy and qualified contract, such as Redact, Substitute, or Rewrite. +* Entity-bearing and no-entity inputs, because no-entity paths may bypass model work. +* Text-size, entity-density, label, language, Unicode, context-size, and relationship strata. +* Single datum, bounded microbatch, or batch execution. +* Batch size, offered concurrency, achieved concurrency, and provider rate limits. +* Local and provider-backed stages. +* Cold-start and warm steady-state execution. +* Success, unsupported, inapplicable, failed, timed-out, cancelled, lost, and inconsistent outcomes where the selected contract defines them. +* Deployment prototype, if any, with transport time reported separately from core processing. + +## Existing Measurement Surface + +The anonymizer.measurement package, docs/development/observability.md, and tools/measurement are the canonical measurement and benchmark surfaces. They already record runs, stages, row throughput, DataDesigner workflows, direct benchmark model workflows, requests and tokens, failures, safety and quality fields, parameter sweeps, sealed artifacts, and sanitized W\&B summaries. + +This program extends that contract. It does not assume that current aggregate and median reports establish per-datum tail latency, queue delay, concurrency saturation, open-loop load behavior, or deployment SLOs. New metrics and analysis require schema, privacy, completion-seal, and compatibility review. + +## Required Graph Observation Contract + +The SDK redesign must expose versioned, content-free observations through the existing opt-in measurement surface for preflight, workframe construction, semantic stages, backend calls, reconciliation, cleanup, and release. When measurement is active, each entered boundary records a monotonic duration and closed outcome. Where available and privacy-reviewed, observations also carry bounded or bucketed workload dimensions, resource use, allowlisted numeric or bucketed provider usage, selected route, semantic and implementation profile versions, reason codes, and protection-quality proxies. + +This is the common measurement surface for the current path and experimental alternatives such as regex-first detection, specialized models, physical call consolidation, and batching. An experiment may change implementation and scheduling, but it must preserve the semantic accounting events required by its claimed profile. Missing lifecycle observations are missing evidence, not zero cost or success. + +Public results and observations remain separate. Caller trace IDs may establish distributed-trace parentage, but they do not become datum identity, private work IDs, or metric labels. Target/context text, prompts, entities, replacements, source identifiers, graph identifiers, private work IDs, endpoints, credentials, and unbounded or content-derived dimensions are forbidden. Measurement failure cannot fabricate terminal evidence or change the release set. + +## Measurement Method + +Use controlled component probes to isolate detector, model, prompt, parser, DataFrame, serialization, graph-building, initialization, resource-reuse, and transport overhead. Use end-to-end benchmarks to determine whether a component improvement survives the complete Anonymizer path. A microbenchmark win is not promotion evidence by itself. + +Freeze before execution: + +* Baseline and candidate code revisions, configurations, profile manifests, prompt and parser versions, rule sets, thresholds, and exact model or endpoint revisions. +* Materialized corpus and workload version, including provenance and held-out role. +* Hardware, runtime, dependency versions, provider, endpoint, region, and resource limits. +* Warmup policy, cold/warm posture, batching, concurrency, measurement boundary, timeouts, retry posture, repetition count, and stopping rule. +* Primary performance hypothesis and whether the candidate claims equivalence, restriction, routing, deployment-only change, or research evidence. +* Mandatory correctness, privacy, quality, reliability, compatibility, and resource gates. + +Remote-model comparisons should use identical materialized inputs and a paired, interleaved, or counterbalanced schedule where practical. Report uncertainty and environmental limitations. Do not publish tail percentiles without a defined sampling unit and enough observations to support them. + +## Metrics + +* End-to-end latency and per-stage or per-workflow service time, using a named sampling unit. +* Queue wait, service time, and total datum latency separately for bounded microbatches. +* Throughput, offered load, achieved load, saturation, latency variance, and timeout behavior. +* CPU, memory, accelerator, initialization, serialization, and instrumentation overhead. +* Provider request count, token use, failures, and rate-limit behavior. +* Detection, transformation, utility, leakage, release-conformance, and FailedRecord evidence appropriate to the selected strategy. +* Report resource consumption before monetary cost. A currency estimate must name the currency, date, rate card, provider and model revision, excluded costs, retry treatment, and calculation method. + +## Correctness and Privacy Gates + +Performance and privacy are not interchangeable scores. A faster candidate does not pass by compensating for worse privacy or correctness with lower latency. + +Baseline parity is necessary for a same-profile claim but is not sufficient: a baseline comparison can preserve every baseline miss. Qualification also requires absolute gates on an independently labelled held-out corpus. Missing or inadequate ground truth produces insufficient evidence, not success. + +Evidence should include exact and relaxed span results, per-label floors, transformation coverage, original-value leakage, unsupported and inapplicable cases, adversarial formats, Unicode and normalization cases, overlap behavior, and strategy-specific utility or relational consistency. LLM judges may supplement but do not replace independently adjudicated labels. + +## Candidate Experiment Families + +The following are unapproved experiment families, not a roadmap commitment or supported behavior. + +### Regex-first hybrid detection + +A candidate experiment may use deterministic rules to seed or replace the initial candidate detector while retaining declared downstream validation, augmentation, finalization, verification, accounting, and release behavior. It must prove exact target offsets, rule provenance, Unicode and normalization handling, overlap behavior, adversarial complexity bounds, per-label quality, end-to-end leakage, and failure semantics. + +This description does not assert that regex is faster, sufficiently complete, safe for any entity class, or equivalent to the current detector. + +### Regex-only restricted detection + +A separate research arm may test a locally executable path for a narrow declared label and input domain. It would be a restricted contract, not a fast form of the full pipeline. It must return unsupported or otherwise fail closed outside its qualified scope and cannot treat no rule match as proof that no PII exists. + +No regex-only profile is approved by this tab. + +### Smaller or specialized models + +Candidate experiments may bind a smaller, specialized, local, fine-tuned, or alternate provider model to a declared semantic task. Model identity, prompt, parser, label set, thresholds, provider policy, and actual route must be recorded. Performance evidence must be paired with stage and end-to-end quality, failure, and privacy evidence. + +### Physical stage consolidation + +Experiments may reduce orchestration or provider overhead by combining physical calls. Consolidation does not remove semantic tasks. Promotion requires preserved keyed outputs, terminal outcomes, failure attribution, cancellation behavior, FailedRecord reconciliation, and measurement visibility. The existing combined rewrite work is evidence that fewer workflows do not by themselves prove a speedup or compatibility. + +### Batching, concurrency, and runtime + +Experiments may explore bounded batching, concurrency, DataFrame construction, serialization, tokenization, graph compilation, initialization, connection reuse, and instrumentation overhead. They must measure queueing, saturation, fairness, memory, provider limits, partial failure, and cancellation without using row position as identity or exposing one datum as unauthorized context for another. + +### Deployment prototypes + +The program may measure externally supplied in-process, local-service, container, or hosted prototypes. Such measurements do not select a deployment form or assign hosting, autoscaling, transport retry, persistence, backpressure, or delivery ownership. + +## Reported Product Interest and Latency Notes + +The separate Streaming Mode discovery note lists OpenShell, Switchyard, Relay, CrowdStrike, and Fortinet. Unless stronger evidence is linked, these names indicate reported interest, not accepted requirements, supported integrations, or product commitments. + +The same note records “30 ms latency expectation” under CrowdStrike and “A few hundred ms” under Fortinet. These are unvalidated discovery notes, not SLOs, benchmark results, or Anonymizer requirements. They lack a named accountable owner, percentile, start and stop boundary, payload and entity density, concurrency, environment, warm/cold posture, model and provider, error budget, quality and privacy objective, and accepted failure policy. + +Do not use these figures as targets until the relevant owner supplies a measurable requirement and approves its context. Until then, experiments may investigate representative low-latency workloads without claiming that they satisfy either note. + +## Evidence Status for Product Inputs + +Reported interest — Preliminary or unattributed input; not a requirement. + +Candidate workload — A named owner supplies workflow placement, request shape, representative bounds, and intended outcome. + +Validated workload evidence — A pinned corpus and environment produce dated, reproducible observations. + +Accepted requirement — The accountable owner approves measurable performance, quality, privacy, and failure criteria. + +Supported integration — Implementation, semantic conformance, operational, public-surface, and deployment gates pass. + +A use-case record should name its source, product owner, technical owner, workflow insertion point, request shape and bounds, measurement definition, throughput and concurrency, quality and privacy objective, failure and fallback policy, trust boundary, deployment constraints, representative corpus, validation date and revision, and unresolved decisions. + +## Experiment Lifecycle + +* Classify the experiment arm and claim before implementation. +* Freeze the hypothesis, baseline, candidate, corpus, environment, measurement method, and acceptance gates. +* Validate instrumentation and run correctness and privacy preflight checks. +* Run controlled component and end-to-end comparisons. +* Analyze distributions, strata, failures, uncertainty, quality, privacy, and resource use. +* Record a result of rejected, inconclusive, further research, candidate for qualification, or qualified for a separately authorized rollout. +* For promotion, name the approving authorities, supported scope, rollout, rollback, monitoring, drift, and requalification triggers. + +## Promotion Evidence + +A promotion package should include: + +* Immutable profile or candidate manifest and distinct version dimensions for code, semantic graph, prompts, parsers, rules, thresholds, exact models or endpoints, provider policy, schemas, retry or fallback policy, and release predicate. +* Pinned benchmark corpus, environment, raw sealed measurements, analysis, uncertainty, and limitations. +* Component and end-to-end performance results. +* Baseline comparison plus absolute correctness, privacy, quality, compatibility, failure-attribution, cancellation, and release evidence. +* Named product, semantic, privacy, compatibility, architecture or API, deployment, and operational authorities where applicable. +* Rollout, rollback, monitoring, drift detection, and requalification plan. + +If an authority or evidence class is missing, the candidate remains an experiment or candidate implementation. + +## Initial Experiment Backlog + +* Establish reproducible baselines for current Redact, Substitute, and Rewrite paths where each path is already qualified. +* Measure single-datum non-model overhead and compare it with complete end-to-end time. +* Prototype an unapproved regex-first hybrid arm for declared labels and exact target offsets. +* Prototype a separate unapproved regex-only restricted Redact research arm. +* Evaluate smaller or specialized models for individual semantic tasks. +* Measure physical stage consolidation while preserving semantic accounting and failure attribution. +* Explore bounded batching and concurrency across the latency, throughput, memory, and failure frontier. +* Measure deployment overhead only after a deployment owner supplies a prototype and measurement boundary. + +Backlog order does not imply priority, approval, implementation authorization, or product commitment. + +## Open Decisions + +* Who owns performance, quality, privacy, and product-promotion thresholds? +* Which corpus is authoritative, how is it governed and versioned, and where may sensitive examples live? +* What constitutes equivalence for nondeterministic Rewrite output? +* Which labels, languages, and adversarial cases require independent critical gates? +* What harness and sampling rules are required before inline tail-latency or load-saturation claims are permitted? +* Which deployment forms are supported, observed, or out of scope, and who owns each? +* May caching or deduplication be investigated, and under what isolation, retention, invalidation, and version-binding policy? +* How are model, provider, prompt, dependency, hardware, and workload drift detected and requalified? +* Should a restricted path reject, declare unsupported, or route to a broader profile when its declared scope does not apply? No behavior is selected here. + +## References + +* [Graph-native Anonymizer SDK RFC](graph-native-anonymizer-sdk-rfc.md). +* [Intake workload validation evidence](intake-workload-validation-evidence.md). +* Streaming Mode discovery document: https://docs.google.com/document/d/1eYsTD49wBbIrE\_321JuFeptZTzs-zMz4VotDvQG44Uk/edit +* Repository observability contract: docs/development/observability.md +* Repository measurement tools: tools/measurement/README.md +* Combined rewrite experiment plan: plans/237/combined-rewrite-graph.md + +## Next Action + +Review this program boundary and identify owners for the first baseline and workload corpus. Do not select or implement a candidate profile, adopt a latency target, or infer adopter acceptance from this tab. diff --git a/docs/development/extensible-sdk-companion-plans.md b/docs/development/extensible-sdk-companion-plans.md new file mode 100644 index 00000000..1e9540f5 --- /dev/null +++ b/docs/development/extensible-sdk-companion-plans.md @@ -0,0 +1,20 @@ + + + +# Extensible SDK planning (superseded) + +Status: superseded on 2026-08-20. This path remains as a compatibility landing +page for existing repository links; it is not a second architecture plan. + +Use these documents instead: + +- [Technical Proposal — Graph-native Anonymizer SDK](graph-native-anonymizer-sdk-technical-proposal.md) + defines the proposed architecture, migration sequence, ownership boundary, + and promotion gates. +- [Evidence — Intake workload validation](intake-workload-validation-evidence.md) + records the published Intake behavior, dated dogfood observations, inferred + Platform compatibility, and unresolved adopter decisions that inform the + proposal. + +The repository history retains the former planning report. Do not use that +historical text as current architecture or evidence. diff --git a/docs/development/graph-native-anonymizer-sdk-rfc.md b/docs/development/graph-native-anonymizer-sdk-rfc.md new file mode 100644 index 00000000..9466e53b --- /dev/null +++ b/docs/development/graph-native-anonymizer-sdk-rfc.md @@ -0,0 +1,616 @@ + + + +# Graph-native Anonymizer SDK + +*AIRE Engineering RFC* + +Author(s): TBD + +Status: Under Review — acceptance of the complete RFC plan requested + +Category: Architecture / SDK + +Draft Date: 2026-08-20 + +Review Date: In progress + +Target Closing Date: Not set + +Implementation: https://github.com/NVIDIA-NeMo/Anonymizer/pull/253 + +Implementation baseline: `codex/anonymizer-openshell-intake` at `29bddad51fdc879c5e5c677857c1d2561f4528ee`. The review candidate includes this RFC, the phase designs, and the private Phases 1–6 source and tests. Phase 7 implementation remains unauthorized. This tab is the review mirror. + +## Decision Requested + +Accept, request revisions to, or reject this RFC as one development and research plan for the graph-native Anonymizer SDK. The decision covers the proposed semantic architecture, the ordered branch-development phases, the strongly typed graph SDK direction, and the separate performance and experimentation program. + +Acceptance records the project decision on the complete plan. It permits iterative implementation, experiments, and evidence gathering on this development branch, subject to the phase order, prerequisites, and operator checkpoints in the plan. It does not approve the proposed public API for publication, authorize production Intake or OpenShell integration, establish a product SLO or deployment profile, permit stable promotion, or make a privacy-boundary or “zero PII” claim. + +Phase checkpoints are branch execution controls, not separate project acceptance decisions. A checkpoint records that the operator has authorized the next bounded implementation or research step on this branch. Evidence review determines whether its prerequisites have passed; later product, public-API, adopter, customer, and release gates remain with the owners named below. + +* Phase 4: the authorized branch-local implementation and its evidence gates completed on 2026-08-25; RFC acceptance, public-API approval, production integration, and promotion remain pending their separate decisions. +* Phase 5: the private branch-local implementation and hardening landed on 2026-08-26 in `bb79cda` through `53ef74f`; its frozen reference model and focused context, lifecycle, privacy, and compatibility evidence are present on the branch. +* Phase 6: the private branch-local implementation and hardening landed on 2026-08-27 in `5bf61c6` through `29bddad`; its frozen reference model and focused mention, resolution, role, Redact, lifecycle, privacy, and compatibility evidence are present on the branch. +* Phase 7: reviewed design only; branch implementation authorization remains pending and sequenced after Phases 4–6 and its versioned semantic and execution contract. + +## Revision History + +* 0.1 — 2026-08-20 — Recorded the graph-native proposal, branch-local Phases 1–3, and the ordered migration. +* 0.2 — 2026-08-20 — Added independently reviewed Phase 5 and Phase 6 designs, synchronized Phase 4 and Phase 7 acceptance status, and reorganized the review mirror as a complete RFC. +* 0.3 — 2026-08-21 — Proposed the strongly typed graph SDK, added benchmark and trace examples, and moved product authorization and field policy outside Anonymizer. +* 0.4 — 2026-08-21 — Defined preflight, private work-ID terminology, content-free lifecycle observations, cleanup outcomes, and the retention-capability boundary. +* 0.5 — 2026-08-21 — Made acceptance apply to the complete RFC plan, distinguished project acceptance from branch execution checkpoints, and recorded implementation and test status. +* 0.6 — 2026-08-25 — Recorded the completed private Phase 4 implementation, frozen conformance corpus, repository verification, and remediation-council closeout without changing RFC or later-phase authority. +* 0.7 — 2026-08-25 — Recorded authenticated Arc acceptance, bounded rejection of non-UTF-8 datum values, and the explicit per-datum stage-predecessor contract without changing later-phase or publication authority. +* 0.8 — 2026-08-31 — Reconciled Phase 5 and Phase 6 status with the private branch implementation, reference-model, test, and CI evidence without changing public, production, Phase 7, or promotion authority. + +## Review History + +* Phase 5 architecture and test-strategy councils — Complete — 2026-08-21 focused re-review closed with zero unresolved Critical or Warning findings. +* Phase 6 architecture and test-strategy councils — Complete — 2026-08-20 — Zero unresolved Critical or Warning findings. +* Phase 4 implementation-remediation council — Complete — 2026-08-25 — All nine remediation claims independently verified; zero unresolved material findings. +* Phase 4 authenticated review-only Arc — Complete — 2026-08-25 — Reviewer accepted the focused remediation with zero findings; formatting, type, full repository, and strict documentation validations passed. +* Project and Anonymizer semantic owner — Pending — Acceptance of the complete RFC development and research plan requested. +* Additional adopter, customer, public-API, and runtime approvals remain scoped to the gates that name them. + +## Development Status + +PR 253 is an active development and research PR. Its protected branch contains the private Phases 1–6 implementation plus the evolving RFC and phase designs; it is not an implementation of the complete RFC and does not expose a public graph SDK. + +| Phase | Branch state | Verification state | Next branch checkpoint | +| --- | --- | --- | --- | +| 1–3 | Implemented privately | 38 focused graph and private-protection tests passed; the 2026-08-20 repository run reported 1,408 passed and 11 opt-in Intake tests skipped | Preserve compatibility while later phases replace the internal system of record | +| 4 | Implemented privately under its authorized branch checkpoint | Frozen `phase4-stream-v4` corpus passed 397,542 canonical traces over 8,278 admitted graphs; focused accounting, race, process-loss, privacy, compatibility, formatting, type, documentation, and full repository checks passed; remediation council closed with zero unresolved material findings | Preserve the verified boundary; later adoption, publication, and promotion require their separate gates | +| 5 | Implemented privately and hardened | Frozen Phase 5 reference-model evidence and focused context admission, execution, reconciliation, cleanup, privacy, and public-compatibility tests are present; 2026-08-31 PR checks pass | Preserve the qualified private boundary; publication and production use require separate gates | +| 6 | Implemented privately and hardened | Frozen Phase 6 reference-model evidence and focused mention, resolution, role-policy, Redact, backend, lifecycle, privacy, and public-compatibility tests are present; 2026-08-31 PR checks pass | Preserve the qualified private Redact boundary; Phase 7 requires its own contract and authorization | +| 7 | Designed and independently reviewed; implementation authorization pending | Test strategy reviewed, but implementation evidence does not yet exist | Freeze the Phase 7 semantic and execution contract and obtain separate authorization | +| 8–11 | RFC plan only | No phase implementation evidence | Refine and authorize bounded branch checkpoints in order | + +For phases that remain proposals, “test strategy reviewed” means reviewers found the proposed evidence plan sufficient to begin the corresponding branch work when authorized. It does not mean that phase has been implemented or its tests have passed. + +## Problem + +The current Anonymizer API treats a DataFrame row as the practical unit of protection and accepts source-formatted data only after callers flatten it into tabular text. That boundary works for independent records, but it cannot faithfully express related-record workloads in which context, replacement coherence, dependencies, and atomic release differ. + +External workloads such as trace trees, trajectories, and structured request/response records need source-neutral protection semantics without moving codecs, source identity, persistence, retries, deduplication, reconstruction, cleanup, or delivery into Anonymizer. Unsupported relationships must fail closed; silently treating related datums as independent rows would erase the property the integration needs to preserve. + +## Background + +Published Anonymizer exposes DataFrame-oriented `run()`, `preview()`, and `evaluate()` paths. `NddAdapter.run_workflow()` is the engine boundary for executing DataDesigner workflows. The migration must preserve those public contracts while replacing the row-local internal system of record. + +Private branch-local Phases 1–3 define immutable, non-serializable graph values, validate a trivial independent-datum profile, lower it through the existing pandas runtime, hydrate graph outcomes, and map them back through the private compatibility flow. Phase 4 extends that seam with explicit dependency DAGs, flat exact atomic partitions, and exhaustive terminal accounting. Phase 5 adds bounded target/context workframes, and Phase 6 adds anchored mentions, explicit-evidence resolution, versioned role results, and exact local Redact verification. Coherence planning and links remain unsupported and fail closed. + +Verification on 2026-08-20 reported 1,408 passed and 11 skipped tests with one warning. The focused graph and private-protection suites passed 38 tests. The 11 opt-in Intake dogfood tests skipped because their external environment was not enabled; historical operator runs remain bounded observations, not product guarantees. + +Branch-local Phase 4 verification on 2026-08-25 reported 1,508 passed and 11 skipped tests with one pre-existing deprecation warning. The frozen `phase4-stream-v4` corpus passed 397,542 canonical traces over 8,278 admitted graphs with manifest digest `e778147bf77909ddb94117fe7e6c230de57e46a722fad49c563b36f0b5660efa`. Formatting, type checking, documentation, privacy-canary, compatibility, fault, concurrency, and mutation checks passed. An independent remediation verifier closed all nine council findings with no remaining material findings, and the maintained authenticated review-only Arc accepted the focused remediation with zero findings. + +Admission now converts non-UTF-8 datum IDs and text into the bounded `malformed_graph` rejection rather than allowing an encoding exception to escape. Synthetic multi-stage plans use a fixed per-datum earlier-stage predecessor, matching the independent reference model; a stage result still closes only after all of its tasks are terminal. This is an accounting contract, not authorization for later semantic stages. + +The separate [Intake workload evidence](intake-workload-validation-evidence.md) records ATIF, OTLP, and chat-completion ingestion findings. Those formats demonstrate different hierarchy, structure, and partial-acceptance pressures, but they remain one Intake runtime and do not satisfy the independent-semantic-runtime gate. + +## Goals + +* Make an immutable ProtectionGraph the private semantic system of record and define a small strongly typed public authoring contract for review. +* Represent context, replacement coherence, dependencies, and atomic release independently. +* Preserve the current DataFrame API and DataDesigner execution boundary during migration. +* Reconcile backend work through opaque invocation-local identity rather than source IDs, content, indexes, or row position. +* Fail closed when requested semantics, backend compatibility, attribution, terminal evidence, or release verification are incomplete. +* Keep source codecs and durable operational lifecycle responsibilities downstream. +* Qualify each semantic capability through an independent reference model, bounded conformance envelope, mutation tests, and privacy checks. + +### Non-Goals + +* Implement or publish the proposed graph API before its public-API and implementation-evidence gates pass. +* Claim exhaustive detection, absence of all PII, or an approved “before Intake” boundary. +* Move source codecs, platform jobs, storage, durable retries, deduplication, retention, purge, reconstruction, or delivery into Anonymizer. +* Treat multiple Intake formats, a process-backed Python host, test adapters, or OpenShell telemetry as a materially different semantic runtime. +* Permit unsupported related-record semantics to degrade to independent rows. +* Authorize production Intake or OpenShell changes, stable promotion, or revisions to “samples.” + +## Terminology and Definitions + +* Atomic group — A flat, exact set of target datums whose qualified outputs are released or withheld together. +* Coherence scope — The set within which aliases and replacement assignments must remain consistent. +* Context scope — The ordered, explicitly declared set of related datums that may inform one target decision. +* Datum — An immutable source-neutral text value with graph-scoped identity. +* Entity cluster — A deterministic set of accepted mentions joined only by explicit same-subject evidence. +* Mention — A detected entity anchored to an exact half-open character interval in one authoritative target datum. +* NDD — NVIDIA DataDesigner, used for LLM column generation through `NddAdapter.run_workflow()`. +* ProtectionGraph — The immutable semantic input containing source-neutral datums and reviewed relationships. +* PreparedProtection — The immutable process-local result of successful graph admission; the only value accepted by public `protect()`. +* ProtectionResult — The immutable mapping from every admitted target key to one closed terminal target outcome. +* Replacement role — A versioned classification of a mention’s replacement function; distinct from detector label, cluster identity, and replacement slot. +* Workframe — A temporary bounded DataFrame projection used to execute one graph stage; not a semantic output unit. +* Bounded invocation — One finite request whose datums, relationships, limits, and selected protection contract are known at compilation. +* Inline integration — A future integration pattern in which an existing system submits a bounded invocation and waits for qualified output or a typed non-success outcome. +* Bounded microbatch — A finite host-formed execution group used to explore throughput and queueing trade-offs; it does not define graph identity, context, atomic groups, or release units. +* Streaming transport — A long-lived or unbounded flow with ordering, backpressure, checkpointing, delivery, retry, recovery, and partial-result semantics. This RFC does not propose one. +* Execution profile — A closed, versioned internal selection of semantic tasks, authorized implementations, schemas, limits, conditional routes, failure behavior, release predicate, and release claim, fixed before invocation effects. +* Deployment form — In-process SDK, service, container, or hosted packaging. Deployment form is not a protection semantic and remains separately governed. + +## Requirements + +### REQ 1 Preserve Public Compatibility + +The migration MUST preserve current public constructors, configuration types, signatures, defaults, result columns and attributes, failed\_records shape, trace\_dataframe behavior, `evaluate()` behavior, CLI behavior, and canonical errors unless a separate public-API review approves a change. + +### REQ 2 Use the Graph as the Semantic Record + +The graph MUST remain authoritative before lowering and after hydration. DataFrame rows, batches, provider responses, and context fragments MUST NOT become semantic identity or release units. + +### REQ 3 Preserve the DataDesigner Boundary + +Every DataDesigner workflow MUST execute through `NddAdapter.run_workflow()`. Graph stages MAY declare DataDesigner column configurations and use temporary DataFrames, but MUST NOT call DataDesigner execution APIs directly. + +### REQ 4 Keep Scopes Independent + +Context scope, coherence scope, dependency, and atomic group MUST remain separate declarations. Source adjacency, equal content, common context, or membership in one relationship MUST NOT imply another. + +### REQ 5 Preserve Datum and Attempt Identity + +Datum identity MUST be immutable and graph-scoped. Private runtime work IDs MUST be random and invocation-local. Text, labels, offsets, source IDs, DataFrame indexes, row order, model-returned IDs, and content-derived hashes MUST NOT satisfy graph or attempt correlation. + +### REQ 6 Compile Before Effects + +The complete requested graph and its declared capabilities MUST be validated into one immutable compiled plan before graph-invocation effects. The executor MUST consume that plan and MUST NOT widen it from live authoring input, host state, source reconstruction state, or observed rows. + +### REQ 7 Account for Every Terminal Outcome + +Every expected task, stage, target datum, dependency, atomic group, and invocation MUST close with an exhaustive typed outcome. Missing, failed, cancelled, blocked, lost, duplicated, stale, foreign, or inconsistent evidence MUST NOT be represented as success or raw-input fallback. + +### REQ 8 Release Only Qualified Complete Groups + +Only policy-qualified success MAY contain output. Release MUST occur after exact reconciliation, strategy verification, publication-critical cleanup, and fixed-point dependency and atomic-group withholding. + +### REQ 9 Bound Context and Preserve Product Ownership + +Context MUST use a private immutable compiled projection with separate target and context frames, exact binding reconciliation, explicit count and byte ceilings, and provider retention disabled for the first profile. The integrating product MUST authorize source access and field use before graph construction; Anonymizer MUST NOT accept or interpret product authorization tokens. + +### REQ 10 Anchor Mentions and Patches Exactly + +Private graph mentions MUST name exact target offsets and source slices. Context MAY inform reviewed decisions but MUST NOT supply a mention endpoint or replacement span. Private Redact MUST apply exactly one mention-keyed patch per accepted mention and verify output by authoritative source-plus-patches reconstruction. + +### REQ 11 Keep Resolution Evidence-Based + +Every mention MUST begin in a singleton cluster. Clusters MAY merge only through accepted versioned same-subject evidence keyed by current mention tokens. Equal text, equal labels, source identity, context membership, and response order MUST NOT merge clusters. + +### REQ 12 Preserve Ownership Boundaries + +Anonymizer MUST own source-neutral protection semantics and sanitized outcomes. Source adapters and downstream systems MUST retain codecs, source identity, field policy, reconstruction, persistence, durable retry, deduplication, retention, cleanup, and delivery. + +### REQ 13 Gate Public Promotion + +Any public graph or session surface MUST receive separate public-API authorization. Stable promotion MUST additionally pass the privacy, provenance, lifecycle, capability, artifact, governance, and materially different runtime gates. + +### REQ 14 Keep Integration Form Separate from Protection Semantics + +A DataFrame adapter and any future bounded-invocation, inline, bounded-microbatch, service, or transport adapter MUST lower once into the same source-neutral graph model and typed outcome model for the selected contract. Packaging and transport MUST NOT redefine datum identity, context declarations, task accounting, atomic groups, release semantics, or downstream ownership. A new public integration form requires separate public-API review. + +### REQ 15 Compile Closed Conditional Profiles + +Before invocation effects, the compiler MUST fix one immutable versioned profile and closed conditional graph, including semantic tasks, declared applicability and routing edges, dependencies, authorized implementations, schemas, limits, failure semantics, release predicate, and release claim. Runtime MUST NOT introduce an undeclared stage, implementation, fallback, or claim from latency, provider availability, row content, or other live conditions. + +### REQ 16 Preserve Accounting Across Physical Optimization + +Batching, concurrency, local execution, provider aggregation, and physical call consolidation MAY change effect scheduling but MUST preserve compiled dependencies, keyed outcomes, failure attribution, terminal accounting, strategy verification, and Phase 4 dependency and atomic-group release behavior. Every DataDesigner-backed task MUST continue to execute through `NddAdapter.run_workflow()`. + +### REQ 17 Make Execution Observable and Measurable + +Every graph phase and execution boundary MUST expose versioned, content-free observations sufficient to measure latency, throughput, resource use, route selection, terminal accounting, reconciliation, cleanup, and protection-quality proxies. Instrumentation MUST preserve semantic behavior and MUST NOT place content, source identity, private work IDs, credentials, endpoints, or unbounded dimensions in telemetry. + +## Proposal + +### Overview + +Adopt an immutable ProtectionGraph as Anonymizer’s private semantic system of record. Keep current public APIs as compatibility facades, keep `NddAdapter.run_workflow()` as the sole DataDesigner execution boundary, and use DataFrames only as temporary stage workframes. + +This separates protection semantics from the accidental boundaries of a DataFrame row and a source format. It allows context, replacement coherence, dependencies, and atomic release to vary independently while preserving downstream ownership of source and operational concerns. + +### Semantic Model + +ProtectionGraph contains source-neutral datums plus independently declared context scopes, coherence scopes, dependencies, and atomic groups. The semantic phases are immutable and explicit: + +`ProtectionGraph → DetectedGraph → ResolvedGraph → PlannedGraph → TransformedGraph → VerifiedGraph` + +DetectedGraph contains accepted datum-anchored mentions. ResolvedGraph contains deterministic entity clusters and evidence. PlannedGraph contains dispositions and replacement assignments. TransformedGraph contains mention-keyed patches or keyed group revisions. VerifiedGraph contains exhaustive datum and atomic-group outcomes. + +Each executor stage consumes only the immediately preceding immutable phase. A grouped operation must return a complete keyed group result or fail; it cannot silently execute as unrelated row operations. + +### Execution Architecture + +The execution path is: + +`source-owned value → source adapter validation and projection → ProtectionGraph → temporary graph workframe → existing pandas workflows → NddAdapter.run_workflow() → exact reconciliation and typed hydration → source adapter reconstruction → downstream persistence and delivery` + +Compilation is pure against declared capabilities. Opening an invocation binds providers, credentials, resources, and private invocation-local work IDs. Runtime correlation never reuses caller or source identifiers. + +Only allowlisted content-free identifiers, bounded counts, reason codes, and verification statements may enter public receipts or diagnostics. Raw target or context text, entities, prompts, replacement values, graph IDs, and content-derived hashes remain private. + +### Product Experimentation and Integration + +The graph-native design creates one stable semantic trunk for product experimentation and future integration. Identity, exact context projection, semantic-task outcomes, terminal accounting, verification, and release remain graph-defined while qualified implementations and execution envelopes may vary within a compiled contract. + +An existing system may eventually submit one bounded invocation and receive qualified atomic-group outputs or typed non-success outcomes through an adapter. The integrating product authorizes source access and field use before that adapter constructs a graph. The adapter and integrating system retain source decoding, field policy, reconstruction, persistence, retries, deduplication, retention, cleanup, and delivery. This RFC does not select an in-process, service, container, or Platform deployment form. + +#### Independent Axes + +* Strategy and release claim: Redact, Substitute, or Rewrite. +* Qualified pipeline profile: the current full path or a separately reviewed alternative implementation or restricted contract. +* Execution envelope and backend: single datum, bounded microbatch, or batch; local or DataDesigner-backed task. +* Deployment form: in-process SDK, local service, container, or hosted service, subject to separate public and operational decisions. + +A bounded microbatch normally changes scheduling, not graph identity, context declarations, atomic groups, or release semantics. Streaming transport remains out of scope because it would require its own ordering, backpressure, checkpointing, recovery, and delivery contract. + +#### Closed Conditional Profiles + +Compilation fixes one closed conditional semantic graph and its authorized implementations before effects. Runtime may follow only predeclared applicability, skip, failover, fallback, or retry edges and must record the selected route and terminal outcome. This rule does not add retry, failover, or fallback to phases whose accepted design excludes them. + +Profiles remain an Anonymizer-owned closed catalog. Narrow internal capabilities may support independently implemented stages where real substitution pressure exists, but this RFC does not create a user-extensible Anonymizer plugin surface or a public profile selector. + +#### Semantic Tasks and Physical Effects + +Semantic tasks are accounting and verification units, not necessarily provider calls. One physical call may serve several tasks only when every task outcome remains keyed, attributable, schema-valid, and independently reconcilable. If physical consolidation prevents safe fault localization, it is not a compatible optimization and must withhold the wider affected scope under the compiled contract. + +Every DataDesigner-backed task continues to execute through `NddAdapter.run_workflow()`. Pure local implementations participate in the same graph accounting even when they do not call DataDesigner. + +#### Performance Program + +The separate Performance and Experimentation tab owns benchmark methodology, experiment arms, candidate implementations, workload evidence, comparison results, and promotion recommendations. Candidate examples such as regex-first detection, restricted rule-based detection, smaller or specialized models, physical call consolidation, batching, and deployment prototypes are not approved behavior. Performance evidence cannot authorize a privacy claim, product profile, deployment, public API, or SLO. + +#### Observability and Profiling + +The graph lifecycle exposes versioned observations through the existing opt-in measurement surface for preflight, workframe construction, semantic stages, backend calls, reconciliation, cleanup, and release. When measurement is active, each entered boundary records a start/terminal pair with monotonic duration. Observations may also carry bounded or bucketed workload dimensions, profile and implementation versions, selected route, terminal outcomes, reason codes, reconciliation and cleanup status, and allowlisted numeric or bucketed provider-usage fields. This common contract allows full-path and alternative implementations to be compared without changing graph semantics. + +Public results and observability remain separate surfaces. Anonymizer may join a caller's distributed trace as a child operation, but caller trace IDs never become datum identity, private work IDs, or metric labels. Measurement failure cannot supply missing terminal evidence, fabricate success, or change the release set. The separate Performance and Experimentation tab owns benchmark interpretation and promotion evidence; the SDK redesign owns the lifecycle events and privacy boundary that make those measurements possible. + +### Phase 4: Hierarchical Terminal Accounting + +Phase 4 adds a private one-shot ledger for stage tasks, target datums, dependencies, invocations, and atomic groups. Dependencies form an explicit DAG. Atomic groups form a flat exact partition. Cycles, nesting, overlap, coverage gaps, unknown references, and implicit singleton completion are rejected before graph-invocation effects. + +The ledger reconciles keyed terminal evidence and distinguishes failed, cancelled, blocked, lost, and inconsistent outcomes. Post-dispatch cancellation without trusted stop evidence is lost. Phase 4 performs no automatic retry. Release occurs only after exhaustive reconciliation and fixed-point dependency and group withholding. + +Status: the authorized private branch-local implementation and its evidence gates completed on 2026-08-25. RFC acceptance, public-API approval, production integration, later-phase authorization, and promotion remain pending their separate decisions. + +### Phase 5: Target and Bounded-Context Workframes + +Phase 5 compiles one bounded projection for every output-bearing target datum. Target text and explicitly declared context remain in separate logical frames and bind to one Phase 4 target task through immutable compiled binding identities and fresh private `target_work_id`, `context_binding_id`, and `attempt_id` values. These work IDs are invocation-local correlation values, not credentials or public graph identifiers. + +Context is an immutable original-text snapshot. It creates no dependency, output, mention, cluster, coherence scope, or release rule. Context-reference cycles are permitted when explicitly declared and bounded because they are not scheduling edges. An invalid binding, exceeded limit, or incompatible backend fails closed rather than dropping context or falling back to an independent row. + +The first profile requires an immutable typed context execution contract with explicit limits, closed backend artifact classes and closure attestations, and a backend-attested `retention_disabled` posture. It is a backend-compatibility contract, not product authorization, proof of provider behavior, or a claim that Anonymizer controls provider retention. Reconciliation proves exact target-task, target-row, context-binding, context-row, ordinal, owner, and consumed-evidence bijections. Publication-critical cleanup closes every Anonymizer-owned frame and work-ID map and reconciles every required execution-boundary closure attestation after private task and datum outcomes are terminal but before release. It does not rewrite those absorbing outcomes. Confirmed owned or attested backend cleanup failure closes the invocation as `failed(cleanup_failed)`; missing or contradictory cleanup evidence closes it as `inconsistent(cleanup_unconfirmed)`. Both map every public target to the corresponding non-success outcome and embargo all output. + +Status: reviewed and implemented privately on the branch. Commits `bb79cda` through `53ef74f` landed and hardened bounded context admission, separate workframes, exact reconciliation, cleanup, reference-model, privacy, and compatibility evidence on 2026-08-26. This branch-local completion does not authorize public APIs, production integration, or promotion. + +### Phase 6: Anchored Mentions and Resolution + +Phase 6 converts untrusted candidate evidence into immutable mentions anchored only to exact target offsets. Candidate lineage is closed: each current candidate receives exactly one keep, reclass, or drop decision before finalization. Missing, duplicate, stale, foreign, contradictory, value-only, overlapping, or source-slice-mismatched evidence fails closed. + +The compiler creates exactly one resolver task per target. Resolution waits for the complete set of required mention-finalization predecessors, then accepts only versioned same-subject or distinct-subject evidence keyed by eligible current mention tokens. Every mention starts as a singleton; deterministic connected components merge only accepted same-subject evidence and reject contradictions. + +The first transform profile is private Redact only. It creates one exact patch per accepted mention, applies patches once against authoritative source offsets without evolving-output search or value fallback, and verifies the returned text through source-plus-patches reconstruction and atomic-group predicates. + +Status: reviewed and implemented privately on the branch. Commits `5bf61c6` through `29bddad` landed and hardened anchored mention admission, explicit-evidence resolution, the frozen Redact role policy, mention-keyed verification, reference-model, privacy, and compatibility evidence on 2026-08-27. This branch-local completion does not authorize Substitute, public APIs, production integration, or promotion. + +### Phase 7: Stable Substitute + +Phase 7 adds a flat coherence partition, replacement-slot planning, and a bounded ephemeral ledger. An entity cluster may map to several type-appropriate replacement slots. Each slot keeps one assignment within one explicitly declared coherence scope and invocation. + +Stable means invocation-bounded consistency, not global permanence, restart recovery, cross-worker consistency, transactional delivery, or durable idempotency. Planning creates one complete provisional bundle per scope and exposes no replacement map until qualified atomic-group release. Collision, concurrency, rollback, leakage, lifetime, cleanup, and state authority must fail closed. + +Status: reviewed design; branch implementation authorization pending. Execution remains sequenced after Phases 4–6 and after its owned versioned semantic and execution contract is frozen. + +### Ownership + +* Anonymizer — Source-neutral graph validation; detection, resolution, planning, transformation, verification, dependency and group accounting; sanitized outcomes. +* Source adapter — Codec; closed field policy; source identity; graph projection; bounded reconstruction state; schema validation; reconstruction and output mapping. +* Execution host — Providers, credentials, endpoints, model provisioning, resource ceilings, backend capabilities, and ephemeral state lifetime. +* NeMo Platform Anonymizer plugin — Current public-facade integration, authentication, filesets, providers, jobs, storage, cancellation, artifacts, and delivery lifecycle. +* Future Intake integration — Ingress, partial acceptance, source-item persistence, durable retries, retention, cleanup, destination deduplication, and delivery. +* Integrating event-driven host (proposed boundary) — Event selection, queueing, saturation and omission policy, worker or plugin lifecycle, and subscriber delivery. +* Shared governance — Policy schemas, conformance corpora, thresholds, support matrix, and compliance-facing claims; owner remains unresolved. + +### Failure, Cancellation, and Release + +Cancellation is an ordered event, not proof that backend work stopped. Pre-dispatch cancellation performs no provider work. After dispatch, a task is cancelled only with trusted stop evidence; otherwise it is lost. Accepted terminal evidence cannot be rewritten by later cancellation or late results. + +Attribution defects localize only when the complete unaffected bijection remains provable. Foreign, swapped, cross-target, plan-mismatched, or contradictory evidence that destroys attribution closes the invocation as inconsistent and withholds all groups. + +Publication-critical finalization must close all owned workframes, stores, artifacts, and work-ID maps before release. Cleanup qualification occurs after private task and datum terminal acceptance and does not rewrite those outcomes. A cleanup failure or inability to prove cleanup instead closes the invocation with the specified global non-success reason and exposes no public target text. This proves logical lifecycle closure, not secure memory erasure or absence from unowned provider traces. + +### Verification Strategy + +Each phase must first define a pure reference model independent of pandas, DataDesigner, production reducers, and the Phase 4 ledger. A frozen finite conformance generator must publish its event alphabet, independence relation, exact graph and trace counts, computed bounds, versions, and SHA-256 manifest digest before comparing executor outcomes. + +Tests must cover exact-limit and one-over-limit admission; declaration and row permutations; duplicate, missing, stale, foreign, cross-target, and contradictory evidence; cancellation and terminal races; worker death; cleanup and publication failure; opaque-ID renaming; DataFrame compatibility; FailedRecord shape; privacy canaries; and mutations that violate every critical invariant. + +Passing the Python reference model and process-backed lifecycle tests does not satisfy the materially different semantic runtime gate. + +## Open Questions + +* Which customer or consuming-product owner will approve the earliest boundary that unprotected target and context data may cross? +* What closed field roles, representative bounds, and source commit units will Intake owners approve for ATIF, OTLP, and chat-completion workloads before graph construction? +* What opaque provenance and receipt contract will span source items, graph datums, invocations, processes, and artifacts without exposing private correlation or content? +* Who owns policy schemas, conformance corpora, support thresholds, and compliance-facing claims? +* Which materially different semantic runtime will implement the agreed conformance subset before stable promotion? +* When cancellation and cleanup become observable, what async operation surface—if any—should receive separate public review? +* Which parts of the proposed graph SDK should remain experimental until private semantics, lifecycle, diagnostics, and privacy gates pass? + +## User Experience Impact + +### Overall UX + +The private migration changes no current Anonymizer or NeMo Platform user workflow. Existing DataFrame, file, preview, run, evaluation, CLI, trace, and failed-record behavior remains the compatibility contract. + +The proposed additive graph SDK lets adopters preserve hierarchy, bounded context, stable substitutions, dependencies, and complete-output requirements without flattening source records. Publication remains gated on private implementation evidence, public-API approval, adapter ownership, cancellation review, and the stable-promotion gates. + +### Public API Changes + +Status: included in the RFC plan; not implemented or approved for publication. Publication requires separate public-API review. + +The public contract uses three immutable generic values and one mandatory state transition: + +```text +ProtectionGraph[TargetKey] + -> PreparedProtection[TargetKey] + -> ProtectionResult[TargetKey] +``` + +`prepare()` is the public preflight operation and the only graph admission boundary. It validates and freezes the exact graph, configuration, profile, schemas, routes, dependencies, limits, context projection, stable-substitution partition, and complete-output partition without model calls or other invocation effects. `protect()` accepts only a prepared value: + +```python +prepared = anonymizer.prepare( + graph, + config=config, +) + +result = anonymizer.protect(prepared) +``` + +`PreparedProtection` is immutable, process-local, and non-serializable. It contains no credentials, live provider session, attempt, or result. The same prepared value may open more than one independent invocation, but reuse does not imply caching, idempotency, or identical generated substitutes. + +Invalid or unsupported input raises a typed `ProtectionRejected` before an invocation exists. An admitted invocation returns exactly one terminal outcome per target. The public target outcome is a closed union: + +```python +TargetOutcome = ( + Protected + | Withheld + | Failed + | Cancelled + | Lost + | Blocked + | Inconsistent +) +``` + +Only `Protected` contains text. A locally successful target whose `require_complete` peer does not qualify becomes `Withheld`; raw input is never returned as fallback. Bounded enums or closed values carry public reason codes, and exhaustive pattern matching distinguishes known failure, proven cancellation, uncertain lost execution, dependency blocking, and inconsistent evidence. + +Immediately before invocation effects, `protect()` verifies that the bound backend still satisfies the capabilities frozen during preflight. A mismatch raises `ProtectionRejected`; it creates no invocation and never drops context or selects a context-free fallback. After execution begins, exact reconciliation uses private invocation-local `target_work_id`, `context_binding_id`, and `attempt_id` values. Missing or contradictory execution evidence produces typed target outcomes rather than a preflight rejection. + +The graph authoring vocabulary is: + +* `texts` — immutable text datums keyed by the caller's target-key type; +* `targets` — the ordered output-bearing datum keys; +* `context` — ordered, explicitly declared read-only context per target; +* `depends_on` — explicit target scheduling dependencies; +* `stable_substitutions` — exact target partitions within which the same resolved entity receives the same invocation-bounded substitute; and +* `require_complete` — exact target partitions for which protected text is exposed only when every member qualifies. + +Context, dependencies, stable substitutions, and complete-output requirements never imply one another. Direct graph construction requires complete valid partitions. Source adapters may materialize reviewed defaults before compilation. + +#### DataFrame benchmark + +The checked-in `repo-data-smoke` benchmark selects the `biography` column from `docs/data/NVIDIA_synthetic_biographies.csv`. Its rows are independent source records, so the DataFrame adapter creates one target and singleton partitions per selected row: + +```python +import pandas as pd + +from anonymizer import Anonymizer, AnonymizerConfig, Redact +from anonymizer.graph import ProtectionGraph + + +rows = pd.read_csv( + "docs/data/NVIDIA_synthetic_biographies.csv" +).head(5) + +graph = ProtectionGraph.from_dataframe( + rows, + target_column="biography", +) + +anonymizer = Anonymizer() +prepared = anonymizer.prepare( + graph, + config=AnonymizerConfig(replace=Redact()), +) +result = anonymizer.protect(prepared) +``` + +`target_column` names the selected field's graph role. Existing `AnonymizerInput.text_column` and benchmark `text_column` retain their published names. The factory does not infer context, dependencies, or cross-row relationships. Equal text and duplicate or null DataFrame indexes never merge datums. + +Existing callers should continue to use `run()` for the simplest DataFrame-in/DataFrame-out workflow. A graph-native caller that needs exact DataFrame reconstruction uses a `DataFrameAdapter`; the adapter retains source identity and reconstruction state outside `ProtectionGraph`. + +#### Whole-trace protection + +One graph may represent an entire bounded trace. This example uses parent text as context, keeps substitutions stable across the trace, and exposes no protected span text unless every target qualifies: + +```python +span_ids = ( + "trace:7/span:root", + "trace:7/span:http", + "trace:7/span:database", +) + +graph = ProtectionGraph( + texts={ + "trace:7/span:root": root_text, + "trace:7/span:http": http_text, + "trace:7/span:database": database_text, + }, + targets=span_ids, + context={ + "trace:7/span:http": ( + "trace:7/span:root", + ), + "trace:7/span:database": ( + "trace:7/span:root", + "trace:7/span:http", + ), + }, + stable_substitutions=( + span_ids, + ), + require_complete=( + span_ids, + ), +) +``` + +Putting the trace in one graph does not require all-or-none output. An adapter may instead declare one singleton `require_complete` set per span when its reconstruction contract permits partial protected traces. Trace hierarchy alone implies neither context nor completeness. + +The integrating product authorizes source access and field use before constructing either graph. Anonymizer receives no product authorization token. Preflight enforces exact bounded context handling, and execution rejects an incompatible backend posture before effects. A `retention_disabled` profile is a checked backend requirement, not an Anonymizer guarantee of provider behavior or physical deletion. + +### Documentation Changes + +Repository development documents remain canonical. This Google document remains a review mirror. Public product documentation and the bundled Anonymizer skill change only if a separately approved public API or behavior changes. + +## Implementation Details + +### Private Types and Validation + +Use immutable closed values for the proposed public graph, prepared protection, target outcomes, and rejection reasons, and for private graph phases, compiled plans, capabilities, identities, outcomes, and evidence. Dynamic DataFrames, dictionaries, serialized payloads, and compatibility models cross one validation boundary before becoming trusted graph values. + +Use Pydantic where user-facing validation or serialization is required and frozen dataclasses for immutable engine values. Keep `PreparedProtection` and private graph phases non-serializable. A portable graph artifact requires a separate versioned-schema decision. + +### Workframes and NDD + +Use COL\_\* constants for every internal column. Use `_jinja()` for shared DataFrame column references and `substitute_placeholders()` for dynamic prompt values. Preserve the no-context legacy workflow and prompt shape where compatibility tests require it. + +Graph stages may declare DataDesigner column configurations, but `NddAdapter.run_workflow()` remains the sole execution boundary. Reconciliation starts from the immutable compiled plan, not from rows or work IDs observed after lowering. Instrumentation may measure immediately around that boundary but cannot bypass it or change its semantic result. + +### Compatibility + +The public facade continues to compile only profiles that have been qualified for compatibility. Public Substitute and Rewrite remain on their legacy semantic paths until separately reviewed graph profiles preserve their filtering, collision repair, prompt, trace, evaluation, and repair behavior. + +### Deferred to Implementation + +Exact private module and class placement, helper decomposition, batch sizing within frozen ceilings, and performance tuning may be resolved during implementation and code review. + +Semantic grammar, requirement strength, rejection precedence, capability versions, privacy boundaries, role policies, release predicates, and ownership gates are not implementation details. Their named owners must freeze or approve them before the phase that consumes them. + +## Implementation Phases + +### Phases 1–3: Private Compatibility Foundation + +Status: branch-local implementation. Phases 1–3 established immutable trivial graphs, independent-datum validation, temporary pandas lowering, typed hydration, and private Redact compatibility. Phase 4 added dependency DAGs, flat exact multi-datum atomic groups, and exhaustive terminal accounting. Later branch-local phases now add bounded context and anchored mention resolution; coherence planning, grouped rewrite, links, and public graph APIs remain unsupported. + +### Phase 4: Hierarchical Terminal Accounting + +Status: the authorized private branch implementation and evidence gates completed on 2026-08-25. It delivers an immutable compiled accounting plan, one-shot ledger, dependency DAG, flat atomic partition, exhaustive outcomes, exact reconciliation, cancellation/loss rules, and fixed-point release. RFC acceptance, public-API approval, production integration, later-phase authorization, and promotion remain separate gates. + +### Phase 5: Target and Context Workframes + +Status: reviewed and implemented privately on 2026-08-26 in `bb79cda` through `53ef74f`. The branch delivers immutable context scopes, bounded separate frames, original-text snapshots, a typed context execution contract, exact work-ID reconciliation, a retention-disabled first profile, observable lifecycle cleanup, and the versioned content-free observation contract. Product authorization and field policy remain outside Anonymizer. + +### Phase 6: Anchored Mentions and Private Redact + +Status: reviewed and implemented privately on 2026-08-27 in `5bf61c6` through `29bddad`. The branch delivers exact target-offset mentions, closed candidate lineage, one resolver task per target, deterministic evidence-based clustering, versioned role results, mention-keyed Redact patches, and exact reconstruction. + +### Phase 7: Stable Substitute + +Status: reviewed design; branch implementation authorization pending. If separately authorized after Phases 4–6 and contract freeze, deliver flat coherence planning, type-appropriate replacement slots, bounded ephemeral ledger, deterministic collision handling, complete provisional bundles, and qualified release. + +### Phase 8: Grouped Rewrite + +Status: proposed. Add keyed group rewrite, evaluation, and repair with no independent-row fallback. + +### Phase 9: Result Compatibility + +Status: proposed. Move legacy result materialization behind compatibility adapters while retaining public behavior. + +### Phase 10: Bounded Inspection + +Status: proposed. Add bounded explain, inspect, and diagnose views, then prepare graph/session records for separate review. + +### Phase 11: Lifecycle and Independent Runtime + +Status: proposed. Validate lifecycle behavior through a process-backed host and the agreed conformance subset through a materially different semantic runtime. The Python host supplies lifecycle evidence only. + +## Related Proposals + +* Implementation and review PR: https://github.com/NVIDIA-NeMo/Anonymizer/pull/253 +* Phase 4 hierarchical terminal-accounting design: repository file docs/development/phase-4-hierarchical-terminal-accounting-design.md +* Phase 5 target/context workframe design: repository file docs/development/phase-5-target-context-workframe-design.md +* Phase 6 anchored-mention resolution design: repository file docs/development/phase-6-anchored-mention-resolution-design.md +* Phase 7 stable Substitute design: repository file docs/development/phase-7-stable-substitute-design.md +* [Intake workload evidence](intake-workload-validation-evidence.md). +* [Performance and experimentation program](anonymizer-performance-and-experimentation.md) — proposed cross-context program; no candidate profile or product SLO is approved. + +## Alternate Solutions + +### Alternative 1: Keep DataFrame Rows as the Semantic Unit + +Pros: smallest internal change; preserves the current mental model. + +Cons: conflates context, coherence, dependency, and atomic release; cannot faithfully represent related-record workloads. + +Reason rejected: flattening would silently weaken requested semantics and make source format or row position an accidental part of protection identity. + +### Alternative 2: Put Source Formats and Delivery Lifecycle in Anonymizer + +Pros: one component could appear to own end-to-end processing. + +Cons: couples protection semantics to ATIF, OTLP, chat, Intake, Relay, and future formats; duplicates platform responsibilities; expands the privacy and reliability boundary. + +Reason rejected: Anonymizer should own source-neutral protection semantics, while adapters and downstream systems retain codecs and durable effects. + +### Alternative 3: Implement the Proposed Graph API Before Private Qualification + +Pros: adopters could integrate against the reviewed graph concepts immediately. + +Cons: would freeze semantics, lifecycle, diagnostics, provenance, capability, and artifact contracts before implementation and conformance evidence exist. + +Reason rejected: this RFC may settle the candidate public contract, but private implementation and evidence must precede publication and compatibility commitments. + +### Alternative 4: Start with Durable Cross-Worker Replacement State + +Pros: could provide restart and distributed consistency earlier. + +Cons: requires governed storage, transactional semantics, recovery, idempotency, retention, and operational ownership beyond the current SDK boundary. + +Reason rejected for the first profile: Phase 7 deliberately qualifies invocation-bounded stability. Durable consistency remains a separate architecture decision. + +## Appendix A: Status Labels + +* Published current behavior — Supported by an immutable public repository revision. +* Branch-local implementation — Present on the draft research branch or explicitly identified working tree; not published behavior or a public contract. +* Dated dogfood observation — A bounded result from a named test or environment on a stated date; not a product guarantee. +* Proposal — Architecture to adopt or work to perform; not current behavior. +* Unresolved gate — A decision, assumption, authority, or proof still required. A provisional assumption is not approval. + +## Appendix B: Gates Before Stable Public Promotion + +* Materially different runtime — Independent semantic implementation of a declared capability subset and shared conformance outcomes — Anonymizer and adopter architecture review. +* Privacy boundary — Earliest boundary unprotected content may cross, named actors, residual risk, and release criteria — Customer or consuming-product owner routed by the Intake team. +* Provenance — Opaque identity and receipt contract across invocation, process, and artifact boundaries — Anonymizer and adopter architecture review. +* Related-record semantics — Hierarchical accounting, no silent flattening, and conformance for context, coherence, dependency, and atomic groups — Anonymizer architecture review. +* Stable Substitute — Collision, concurrency, rollback, leakage, scope lifetime, cleanup, and state-lifetime evidence — Anonymizer semantic and execution owners. +* Lifecycle — Bounded resources, readiness, cancellation, cleanup, crash, and version behavior — Runtime owners. +* Observability — Versioned content-free lifecycle events, bounded dimensions, privacy allowlist, non-interference, and profiling coverage — Anonymizer semantic and measurement owners. +* Public graph or session surface — Separate publication review and explicit authorization — Public-API owners. +* Stable public artifacts — Versioned schemas, capability negotiation, reconstruction, OpenAPI and SDK regeneration, and cross-repository tests — Anonymizer and Platform owners. +* Governance — Owners for policy, corpus, thresholds, support matrix, and compliance-facing claims — Unresolved. + +## Appendix C: Evidence Base + +* Published Anonymizer facade: https://github.com/NVIDIA-NeMo/Anonymizer/blob/3eab7d1b6005b85e7d415b704e27a20dc41ba71e/src/anonymizer/interface/anonymizer.py +* Published `NddAdapter.run_workflow()` boundary: https://github.com/NVIDIA-NeMo/Anonymizer/blob/3eab7d1b6005b85e7d415b704e27a20dc41ba71e/src/anonymizer/engine/ndd/adapter.py +* Branch-local graph model: src/anonymizer/engine/execution/graph.py +* Branch-local graph runtime: src/anonymizer/engine/execution/graph\_runtime.py +* Branch-local protection service: src/anonymizer/engine/execution/protection\_service.py +* Branch-local compatibility flow: src/anonymizer/interface/\_protection.py +* [Separate Intake workload evidence](intake-workload-validation-evidence.md) + +## Next Decision + +Project reviewers should accept the complete RFC development and research plan or identify required revisions. Acceptance permits branch-local implementation and experimentation only through the ordered prerequisites and operator checkpoints described above; it does not approve public API publication or any production integration. Private Phases 4–6 are implemented with branch-local evidence. Phase 7 remains after those completed prerequisites and still requires its contract freeze, semantic and execution owner approval, and separate operator authorization. diff --git a/docs/development/graph-native-anonymizer-sdk-technical-proposal.md b/docs/development/graph-native-anonymizer-sdk-technical-proposal.md new file mode 100644 index 00000000..e6623aec --- /dev/null +++ b/docs/development/graph-native-anonymizer-sdk-technical-proposal.md @@ -0,0 +1,256 @@ + + + +# Technical Proposal — Graph-native Anonymizer SDK + +Status: proposed architecture within the complete development and research plan presented by the companion [Graph-native Anonymizer SDK RFC](graph-native-anonymizer-sdk-rfc.md). RFC acceptance permits ordered branch-local implementation and evidence gathering through operator checkpoints. It does not approve publication of a public graph contract, a production Intake integration, or a release claim. “Samples” are outside this rewrite and require separate review. + +## Decision + +**[Proposal]** Adopt an immutable `ProtectionGraph` as Anonymizer’s semantic system of record. Keep the current public APIs as compatibility facades, keep `NddAdapter.run_workflow()` as the sole boundary for executing DataDesigner workflows, and use DataFrames only as temporary stage workframes. + +**[Proposal]** This decision separates protection semantics from two accidental boundaries: a DataFrame row and a source format. Context, replacement coherence, and atomic release can then vary independently without moving codecs, platform jobs, persistence, retries, deduplication, or delivery into Anonymizer. + +**[Proposal and unresolved gate]** Accept the architecture and ordered phases as one RFC plan. The operator may then authorize bounded implementation and research checkpoints on this branch as their prerequisites are met. The proposed public graph contract still requires separate public-API approval before publication, and stable promotion additionally requires a materially different semantic runtime to implement the reviewed contract and acceptance of the privacy and provenance decisions described below. + +## How to read the status labels + +Every consequential claim uses one of these labels: + +- **[Published current behavior]** Behavior supported by an immutable public repository revision. +- **[Branch-local implementation]** Behavior in the protected review candidate at branch baseline `88b59e2a4366be09aa7af802fa0a8f81afa8440d`; not published behavior or a public contract. +- **[Dated dogfood observation]** A bounded observation from a named test or environment on a stated date; not a product guarantee. +- **[Proposal]** Architecture to adopt or work to perform; not current behavior. +- **[Unresolved gate]** A decision, assumption, or proof still required. A provisional assumption is not approval. + +## Contract to preserve + +**[Published current behavior]** Anonymizer exposes DataFrame-oriented `run()`, `preview()`, and `evaluate()` paths, and `NddAdapter.run_workflow()` is the engine boundary that executes DataDesigner workflows. The replace and rewrite pipelines use this adapter rather than creating or previewing DataDesigner workflows directly. [Facade and `run()` path](https://github.com/NVIDIA-NeMo/Anonymizer/blob/3eab7d1b6005b85e7d415b704e27a20dc41ba71e/src/anonymizer/interface/anonymizer.py#L151-L257) · [DataDesigner execution boundary](https://github.com/NVIDIA-NeMo/Anonymizer/blob/3eab7d1b6005b85e7d415b704e27a20dc41ba71e/src/anonymizer/engine/ndd/adapter.py#L267-L328) + +**[Proposal]** Preserve the signatures, constructors and config types, defaults, result columns and attributes, `failed_records` shape, and CLI behavior of the current public facade throughout the internal migration. `trace_dataframe` and current `evaluate()` behavior remain pandas compatibility contracts until an additive graph-level design is separately reviewed. + +**[Proposal]** Preserve `NddAdapter.run_workflow()` as the sole DataDesigner execution boundary. A graph stage may declare DataDesigner column configurations and may batch work in a DataFrame, but it must execute through the adapter. + +**[Proposal]** Treat a successful protection outcome as a qualified execution result, not proof that detection was exhaustive or that output contains no PII. A release predicate can verify only its reviewed strategy and policy postconditions. + +## Semantic model + +**[Proposal]** `ProtectionGraph` contains source-neutral text datums, links, and three independent scopes: + +```text +context scope — which related datums may inform a decision +coherence scope — where aliases and replacements must remain consistent +atomic group — which protected outputs succeed or fail together +``` + +A DataFrame row must not stand in for all three. For example, a trace can use parent spans as context, a trace-wide coherence scope, and per-span atomic groups. A trajectory can instead require bounded turn context, trajectory-wide replacement coherence, and a whole-trajectory atomic group. Unsupported related-record semantics fail closed; they never fall back to independent rows. + +**[Proposal]** The semantic phases are immutable and explicit: + +```text +ProtectionGraph + -> DetectedGraph # accepted, datum-anchored mentions + -> ResolvedGraph # entity clusters and alias evidence + -> PlannedGraph # dispositions and replacement assignments + -> TransformedGraph # patches or keyed group revisions + -> VerifiedGraph # exhaustive datum and atomic-group outcomes +``` + +Each phase should have its own closed type. Detection offsets remain anchored to the target datum. Entity clusters and replacement slots remain distinct, so related aliases can refer to one subject while using type-appropriate replacements. A grouped rewrite must return a keyed group result or fail; it cannot silently run as unrelated row rewrites. + +## Execution architecture + +**[Proposal]** The graph is authoritative before and after each stage. A private executor lowers the ready frontier to a temporary workframe, calls the existing pandas/DataDesigner backend, reconciles private invocation-local work IDs, and hydrates typed results into the next graph phase. + +```text +source-owned value + -> source adapter: validate, project, retain reconstruction state + -> ProtectionGraph + -> graph stage: lower to temporary DataFrame workframe + -> existing pandas workflows + -> NddAdapter.run_workflow() + -> graph stage: verify and hydrate typed outcomes + -> source adapter: reconstruct + -> downstream persistence and delivery +``` + +The text equivalent is: adapters project source data into graph semantics; Anonymizer runs source-neutral protection through temporary DataFrames; adapters reconstruct protected source data; downstream systems own durable effects. + +**[Proposal]** Caller or source identifiers never become engine correlation IDs. Runtime uses private `target_work_id`, `context_binding_id`, and `attempt_id` values that are random and invocation-local, not credentials or public graph identifiers. Public receipts contain only allowlisted, content-free identifiers and verification statements; they do not include raw entities, prompts, source content, private work IDs, or content-derived hashes. + +**[Proposal]** Terminal accounting is exhaustive by datum and atomic group. Only a policy-qualified success contains output. Rejected, failed, cancelled, unknown, missing, duplicated, or inconsistent outcomes withhold the affected atomic group. A transport break that returns no run record also withholds output; raw input is never the fallback. + +## Compatibility migration + +**[Branch-local implementation]** Private phases 1–3 exist in `src/anonymizer/engine/execution/graph.py`, `src/anonymizer/engine/execution/graph_runtime.py`, `src/anonymizer/engine/execution/protection_service.py`, and `src/anonymizer/interface/_protection.py`. The implementation defines immutable, non-serializable graph values; validates a trivial independent-datum profile; lowers it through the existing pandas runtime; hydrates graph outcomes; and maps those outcomes back to the private compatibility flow. + +**[Branch-local implementation]** Phases 1–3 established a trivial compiler for singleton context, coherence, and atomic scopes. Phase 4 extended that compiler with explicit dependency DAGs and flat exact multi-datum atomic partitions. At the Phase 4 checkpoint it still rejected links, added context, and multi-datum coherence with explicit private validation codes. The branch-local accounting tests are in `tests/engine/execution/test_graph_runtime.py` and `tests/engine/execution/test_hierarchical_accounting.py`. + +**[Branch-local implementation — phases 5–6]** Phase 5 now adds bounded target/context admission, separate workframes, exact private-ID reconciliation, cleanup, and a frozen independent reference model. Phase 6 now adds anchored mentions, closed candidate lineage, explicit-evidence resolution, versioned role results, mention-keyed local Redact verification, and a separate frozen reference model. Multi-datum coherence, graph Substitute, grouped Rewrite, links, and public graph APIs remain unsupported. + +**[Branch-local implementation]** No public graph API exists. The graph slice qualifies only its private Redact release profile; it does not qualify Substitute or Rewrite for graph execution and does not change current public signatures. + +**[Branch-local verification — 2026-08-20]** The verifier reran `uv run --frozen pytest -q` against the current uncommitted worktree: **1,408 passed and 11 skipped** with one warning. The focused graph and private-protection tests passed **38 tests**. The 11 opt-in Intake dogfood tests skipped because their external environment was not enabled; the dated observations below remain historical operator-run evidence, not results from this suite invocation. + +**[Branch-local verification — 2026-08-25]** The Phase 4 implementation passed the frozen `phase4-stream-v4` corpus of **397,542 canonical traces over 8,278 admitted graphs** and the full repository suite of **1,508 passed and 11 skipped** with one pre-existing deprecation warning. Formatting, type checking, documentation, privacy-canary, compatibility, process-loss, concurrency, and mutation checks passed. The independent implementation-remediation council closed all nine findings with zero unresolved material findings, and the maintained authenticated review-only Arc accepted the focused remediation with zero findings. + +**[Branch-local verification — 2026-08-31]** The protected PR head includes the Phase 5 implementation and hardening commits `bb79cda` through `53ef74f` and the Phase 6 implementation and hardening commits `5bf61c6` through `29bddad`. Frozen Phase 5 and Phase 6 reference-model manifests and focused admission, runtime, reconciliation, lifecycle, privacy, Redact, and compatibility suites are present. Current PR checks pass on Python 3.11–3.13, along with formatting, type, DCO, linked-issue, conventional-commit, and documentation checks. This is branch-local evidence, not publication or production authorization. + +The ordered migration is: + +1. **[Branch-local implementation — phases 1–3]** Encapsulate the pandas backend, extract source-neutral protection services, and prove trivial-graph compatibility. These phases remain the compatibility seam extended by Phase 4. +2. **[Branch-local implementation — phase 4 hard gate complete]** A private one-shot ledger provides hierarchical stage-task, datum, dependency, invocation, and atomic-group terminal accounting according to the [phase 4 design and test strategy](phase-4-hierarchical-terminal-accounting-design.md). Dependencies form an explicit DAG. Atomic groups form a flat exact partition; cycles, nesting, overlap, coverage gaps, and unsupported cardinalities are rejected before graph-invocation effects. Phase 4 performs no automatic task retry. Post-dispatch cancellation without trusted stop evidence is lost. Release occurs only after exhaustive reconciliation and fixed-point dependency and group withholding. Its authorized private implementation, repository evidence gates, and independent remediation review completed on 2026-08-25. This completion grants no later-phase, public-API, product, integration, or promotion authority. +3. **[Branch-local implementation — phase 5 complete]** Separately framed target and bounded context workframes landed in `bb79cda` through `53ef74f` according to the [phase 5 design and test strategy](phase-5-target-context-workframe-design.md). The private branch includes the frozen reference model and focused context, lifecycle, privacy, and compatibility evidence. This completion grants no public-API, product, integration, or promotion authority. +4. **[Branch-local implementation — phase 6 complete]** Target-anchored mentions, deterministic evidence-based entity clustering, versioned replacement-role classification, and exact local Redact verification landed in `5bf61c6` through `29bddad` according to the [phase 6 design and test strategy](phase-6-anchored-mention-resolution-design.md). The private branch includes the frozen reference model and focused mention, resolution, role, Redact, lifecycle, privacy, and compatibility evidence. This completion grants no Substitute, public-API, product, integration, or promotion authority. +5. **[Proposal — phase 7; branch implementation authorization pending]** Add coherence-scope replacement planning and a bounded ephemeral ledger according to the [stable Substitute design, branch decision record, and test strategy](phase-7-stable-substitute-design.md). Stable Substitute means stable assignment inside one explicitly declared scope; it does not imply durable idempotency, restart recovery, or cross-worker consistency. Independent architecture and test-strategy reviews are complete. Implementation requires separate operator authorization after phases 4–6 and after its owned versioned semantic and execution contract is frozen. +6. **[Proposal — phase 8]** Add keyed group rewrite, evaluation, and repair with no independent-row fallback. +7. **[Proposal — phase 9]** Move legacy result materialization behind compatibility adapters while retaining public behavior. +8. **[Proposal — phase 10]** Add bounded explain, inspect, and diagnose views. Prepare bounded graph/session records for review after their semantics, diagnostics, cancellation, and cleanup are verified. +9. **[Proposal — phase 11]** Validate lifecycle behavior through a process-backed host and validate the agreed conformance subset through a materially different semantic runtime. The process-backed Python host supplies lifecycle evidence only; it does not satisfy the second-runtime gate. + +**[Unresolved gate]** Public exposure, including an experimental graph or session surface, requires separate authorization. Freeze and promote a stable portable contract only after that authority exists, phase 11 passes, and the privacy, provenance, lifecycle, and capability gates below pass. + +## Phase 5 and phase 6 design scope + +**[Branch-local implementation]** Phase 5 compiles one private, bounded target/context projection per output +target. It preserves original datum identity and keeps target, context, dependency, atomic, +and later coherence semantics separate. Context-only datums never become output. The +integrating product authorizes source access and field use before graph construction. +Anonymizer enforces the compiled projection, limits, private work-ID reconciliation, +observable cleanup of owned state, required execution-boundary closure attestations, and a +first profile that requires provider retention to be disabled; it does not accept product +authorization tokens or claim to control provider retention or deletion. + +**[Branch-local implementation]** Phase 6 converts untrusted candidate evidence into exact target-offset +mentions, then resolves those mentions from explicit typed same-subject and distinct-subject +evidence. It accounts for exactly one resolver task per target and qualifies only a private +Redact profile through exact source-plus-patches reconstruction. Context may inform reviewed +tasks but cannot create a mention endpoint, replacement span, coherence scope, or release +rule. The [phase 5](phase-5-target-context-workframe-design.md) and +[phase 6](phase-6-anchored-mention-resolution-design.md) subordinate designs define the +admission, lifecycle, failure, privacy, reference-model, and exhaustive test requirements. +Their branch-local implementations preserve the reviewed prerequisites and gates. Their completion does not authorize public or production use or bypass the separate Phase 7 checkpoint. + +## Phase 7 stable Substitute design scope + +**[Proposal]** Stable Substitute is explicitly in scope for the RFC plan. Phase 7 follows phases 4–6 and adds coherence-scope replacement planning plus a bounded ephemeral ledger. Independent architecture and test-strategy review completed on 2026-08-20 with zero unresolved Critical or Warning findings. The [decision record](phase-7-stable-substitute-design.md#branch-decision-record) preserves the reviewed architecture, but implementation still requires a separate operator checkpoint after the earlier gates and the required contract freeze. + +The first proposed profile is invocation-bounded. It plans one complete provisional bundle per flat coherence scope, preserves distinct entity-cluster and type-appropriate replacement-slot identity, and releases assignments only through qualified atomic-group output. Failed, cancelled, lost, inconsistent, colliding, partial, or unverifiable planning exposes no replacement map or raw-input fallback. + +The subordinate design defines admission, linearization, cancellation, collision, cleanup, leakage, compatibility, and conformance requirements. Durable state, retries, deduplication, reconstruction, persistence, and delivery remain downstream or separately governed. + +## SDK and runtime boundary + +**[Proposal]** Dynamic inputs—including DataFrames, dictionaries, serialized payloads, and compatibility models—must cross one validation boundary into trusted nominal graph values. Internal graph phases, lifecycle states, and expected terminal outcomes should use immutable closed variants with exhaustive handling. Missing, cancelled, lost, blocked, and inconsistent are distinct states rather than values collapsed into `None`, a boolean, or an untyped error bag. + +**[Proposal]** Keep the owned graph grammar closed. Add a narrow protocol only where the executor consumes a genuinely replaceable capability with independent implementations; do not turn lifecycle states or graph alternatives into an open plugin hierarchy. Preserve request, result, provenance, and applicability as separate typed axes. + +**[Proposal]** The compiled plan earns its boundary by proving validation before effects and carrying that proof into execution. The executor must consume the compiled plan rather than re-read or reinterpret the original authoring input. The DataFrame facade remains a translation-only compatibility boundary over the same semantic path where that path is qualified. Public Substitute temporarily remains on its legacy row-local semantic path until a separately reviewed compatibility profile can preserve its filtering, collision repair, and trace behavior. + +**[Proposal]** Expected outcomes that callers must inspect belong in typed variants. Invalid contracts, violated lifecycle rules, I/O failures, and exceptional runtime failures use precise cause-preserving exceptions at their owning boundary. Retryability and partial-failure policy remain explicit domain decisions; an exception type alone does not make an operation safe to retry. + +**[Proposal]** Subject to separate public-API review before publication, the small SDK consists of `ProtectionGraph[TargetKey]`, mandatory `prepare()` into an immutable process-local `PreparedProtection[TargetKey]`, `protect()` of prepared values only, and `ProtectionResult[TargetKey]` with a closed target-outcome union. The exact proposed vocabulary, benchmark example, trace example, and rejection contract are defined in the companion RFC. `prepare()` is the preflight boundary and remains pure against declared capabilities. Immediately before effects, `protect()` rejects a bound backend that no longer satisfies the prepared capability contract; otherwise it binds providers, credentials, resources, and fresh private work IDs. + +**[Proposal]** Every graph phase and execution boundary exposes a versioned, content-free observation contract for latency, throughput, resource use, selected route, terminal accounting, reconciliation, cleanup, and protection-quality proxies through the existing opt-in measurement surface. Instrumentation joins caller traces without reusing caller IDs as datum or work identity, and it cannot change semantic outcomes or place content, source identity, credentials, endpoints, private work IDs, or unbounded dimensions in telemetry. The SDK redesign owns these lifecycle events; the separate performance program owns benchmark interpretation and promotion recommendations. + +**[Proposal]** An in-process session may provide bounded process-local replacement consistency. It does not provide durable idempotency, restart recovery, cross-worker consistency, transactional output-and-ledger commit, or platform job recovery. Those claims require a durable service and a governed state backend. + +**[Unresolved gate]** Async operation handles are deferred until cancellation and cleanup are observable in the underlying runtime. Cancelling a caller’s waiter does not by itself prove that model or workflow execution stopped. + +**[Unresolved gate]** Do not introduce an open engine protocol or select a wire format from the Python implementation alone. A process-backed Python host can validate lifecycle, IPC, crash, and cancellation behavior, but it is not a materially different semantic runtime. + +## Ownership + +**[Proposal]** The boundary follows this allocation: + +| Owner | Responsibility | +| --- | --- | +| Anonymizer | Source-neutral graph validation; detection, resolution, transformation, and release semantics; context, coherence, dependency, and atomic-group accounting; sanitized outcomes | +| Source adapter | Codec; closed field policy; projection; bounded reconstruction state; source identity; schema validation; reconstruction and output mapping | +| Execution host | Providers, credentials, endpoints, model provisioning, resource ceilings, backend capabilities, and ephemeral state lifetime | +| NeMo Platform Anonymizer plugin | Published public-facade integration; authentication, filesets, provider resolution, jobs, storage, cancellation, artifacts, and delivery lifecycle | +| Future Intake integration | Ingress, source-item persistence and partial acceptance, durable retries, retention, cleanup, destination deduplication, and delivery | +| Relay | Event selection, queueing, saturation and omission policy, worker or plugin lifecycle, and subscriber delivery | +| Shared governance | Policy schemas, conformance corpora, thresholds, support matrix, and compliance claims; owner remains unresolved | + +**[Proposal]** Source formats stay downstream. ATIF, OTLP, chat-completion, direct-span, OCSF, Intake, Relay, and OpenShell types do not enter Anonymizer core. Multiple formats handled by Intake exercise distinct workload shapes but remain one Intake runtime, not a second semantic implementation. + +**[Proposal]** For target graph/session adoption, source adapters may propose graph relations and atomic groups under a closed policy; Anonymizer validates support for the requested source-neutral semantics. The adapter maps the source-owned commit unit to graph atomic groups and retains the reconstruction manifest. Anonymizer does not select a codec, commit durable data, schedule retries, or deduplicate destination writes. Legacy `run()` and `preview()` retain their published file/DataFrame input behavior during migration. + +**[Proposal]** The integrating product authenticates and authorizes source access and field use before its adapter constructs a graph. Anonymizer owns exact bounded context handling after admission, but it does not interpret product identities, access tokens, or field-authorization claims. + +## NeMo Platform compatibility and future adoption + +**[Published current behavior]** At NeMo Platform commit `e1057736703bb8b167a4bd9013cea0caae2df63a`, the current Anonymizer plugin pins `nemo-anonymizer==0.3.3`, constructs the public `Anonymizer` facade, and calls public `run()`, `preview()`, and `validate_config()`. It persists and reconstructs DataFrame-shaped artifacts and owns the service, job, fileset, provider, storage, cancellation, and delivery lifecycle. This released plugin is distinct from the proposed Intake workload adapter; no current plugin-to-Intake graph integration exists. [Plugin dependency](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/plugins/nemo-anonymizer/pyproject.toml#L10-L20) · [Run job](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/jobs/run.py#L41-L141) · [Preview worker](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/functions/_preview_worker.py#L31-L73) + +**[Proposal, inferred compatibility]** Private phases 1–3 should require no Platform change if current constructors and config types, `run()`, `preview()`, `validate_config()`, result columns and attributes, and `failed_records` shape remain stable, because the plugin does not call private protection or graph symbols. This is a call-site inference, not an executed cross-repository compatibility test. + +**[Proposal]** Future graph/session adoption requires a Platform-owned adapter for cell-to-datum projection, outcome-to-row reconstruction, and source identity; capability negotiation; versioned artifacts and readers; cancellation and cleanup review; OpenAPI and SDK regeneration for any new endpoint schema; and cross-repository tests. The legacy endpoints remain compatibility adapters until that work is complete. + +**[Published current behavior]** The Platform plugin writes `dataset.parquet`, `trace.parquet`, `metadata.json`, and optional `failed_records.json` sequentially before `ctx.results.save`; no atomic artifact-bundle contract has been established. Preview uses `abandon_on_cancel=True`, so abandoning the async wait may leave the synchronous worker running. A future session design must therefore establish cancellation and cleanup behavior rather than inherit it by assumption. + +## Atomicity, retry, and deduplication + +**[Published current behavior]** Current Anonymizer `run()` and `preview()` return one Python result or raise; `failed_records` identify records dropped by a workflow in an otherwise returned result. This is Python result-publication behavior. It is not artifact atomicity—the Platform plugin writes result files sequentially before publication—and it is not a transaction over providers, telemetry, source ingestion, storage, or delivery. + +**[Proposal]** Anonymizer atomicity is scoped to declared graph atomic groups and the publication of their qualified outcomes. A source adapter maps source commit units to those groups and withholds or reconstructs accordingly. Intake decides its persistence and partial-acceptance behavior. + +**[Proposal]** Retry ownership remains downstream. A source adapter that retries must retain the exact protected payload when safe, preserve stable source identity, classify transport uncertainty, and verify the destination postcondition. Neither a repeated successful Anonymizer call nor one collapsed read-model row establishes transactional Intake idempotency. + +## Gates before stable public promotion + +| Gate | Evidence required | Authority or routing surface | +| --- | --- | --- | +| Materially different runtime | Independent semantic implementation of a declared capability subset; shared conformance outcomes | Anonymizer and adopter architecture review | +| Privacy boundary | Earliest boundary unprotected content may cross; named actors; residual-risk and release criteria | Customer or consuming-product owner, routed by the Intake team | +| Provenance | Reviewed opaque identity and receipt contract across invocation, process, and artifact boundaries | Anonymizer and adopter architecture review | +| Related-record semantics | Hierarchical accounting, no silent flattening, and conformance for context, coherence, dependency, and atomic groups | Anonymizer architecture review | +| Stable Substitute | Collision, concurrency, rollback, leakage, scope-lifetime, and state-lifetime evidence | Anonymizer semantic and execution owners | +| Lifecycle | Bounded resources, readiness, cancellation, cleanup, crash, and version behavior | Runtime owners | +| Any public graph/session surface | Separate publication review and explicit authorization | Public-API owners | +| Stable public artifacts | Versioned schemas, capability negotiation, reconstruction rules, OpenAPI/SDK regeneration, and cross-repository tests | Anonymizer and Platform owners | +| Governance | Owners for policy, corpus, support thresholds, and compliance-facing claims | Unresolved | + +**[Unresolved gate]** “Before Intake” is only a provisional planning assumption. It is not customer-approved. The accepted decision must distinguish the source adapter, optional edge component, Intake process, durable storage, operator-facing APIs and UI, and downstream consumers. + +**[Unresolved gate]** “Zero PII” has no accepted contract meaning. The design must not claim absence of all PII. A review may instead define which content, detection policy, release predicate, residual risk, and trust boundary make an output eligible for a particular use. + +## Explicit non-goals + +This proposal does not: + +- make a public graph API current behavior; +- treat multiple Intake formats, a process-backed host, a test adapter, or current OpenShell telemetry as a second semantic runtime; +- move source codecs, platform jobs, storage, retries, deduplication, retention, purge, or delivery into Anonymizer; +- claim transactional Intake idempotency or storage rollback; +- claim that “before Intake” is approved; +- permit unsupported related-record semantics to degrade to independent rows; +- claim exhaustive detection or absence of all PII; +- authorize production OpenShell changes; or +- revise or approve “samples.” + +## Evidence base + +- [Published Anonymizer facade](https://github.com/NVIDIA-NeMo/Anonymizer/blob/3eab7d1b6005b85e7d415b704e27a20dc41ba71e/src/anonymizer/interface/anonymizer.py#L151-L257) +- [Published `NddAdapter.run_workflow()` boundary](https://github.com/NVIDIA-NeMo/Anonymizer/blob/3eab7d1b6005b85e7d415b704e27a20dc41ba71e/src/anonymizer/engine/ndd/adapter.py#L267-L328) +- Branch-local graph model: `src/anonymizer/engine/execution/graph.py` +- Branch-local graph runtime: `src/anonymizer/engine/execution/graph_runtime.py` +- Branch-local protection service: `src/anonymizer/engine/execution/protection_service.py` +- Branch-local compatibility flow: `src/anonymizer/interface/_protection.py` +- [Phase 4 hierarchical terminal-accounting design and test strategy](phase-4-hierarchical-terminal-accounting-design.md) +- [Phase 5 target and bounded-context workframe design and test strategy](phase-5-target-context-workframe-design.md) +- [Phase 6 anchored-mention, resolution, and local-verification design and test strategy](phase-6-anchored-mention-resolution-design.md) +- [Phase 7 stable Substitute design and test strategy](phase-7-stable-substitute-design.md) +- [Separate Intake workload evidence](intake-workload-validation-evidence.md) + +## Next decision + +Project reviewers should accept the complete RFC development and research plan or identify +required revisions. Acceptance permits branch-local implementation and experimentation only +through its ordered prerequisites and operator checkpoints. Private Phases 4–6 are now +implemented with branch-local evidence. Phase 7 remains sequenced after those completed +prerequisites and still requires its owned versioned semantic and execution contract, +semantic and execution owner approval, and separate operator authorization. +The proposed public SDK contract still requires separate public-API review before +publication. RFC acceptance does not authorize a production Intake or OpenShell integration, +stable promotion, or any unresolved privacy or provenance decision. diff --git a/docs/development/intake-workload-validation-evidence.md b/docs/development/intake-workload-validation-evidence.md new file mode 100644 index 00000000..8902d7c7 --- /dev/null +++ b/docs/development/intake-workload-validation-evidence.md @@ -0,0 +1,115 @@ + + + +# Evidence — Intake workload validation + +Status: evidence record as of 2026-08-20. This tab validates workload pressure on the graph-native SDK proposal; it does not define Anonymizer architecture, claim production format support, approve a PII boundary, or make “samples” part of this rewrite. + +## Scope and interpretation + +The evidence covers NeMo Platform Intake at commit `e1057736703bb8b167a4bd9013cea0caae2df63a`, the Anonymizer draft research branch at `702f43a988cf3673d16f40be5c59bc784737e1a3`, and dated synthetic dogfood anchored to the revisions named below. Consequential claims use the same labels as the proposal: **[Published current behavior]**, **[Branch-local implementation]**, **[Dated dogfood observation]**, **[Proposal]**, and **[Unresolved gate]**. + +**[Proposal]** Several formats passing through one Intake service provide varied workload evidence. They do not constitute a second semantic runtime for Anonymizer’s stable-public-API gate. + +## Current Intake workload + +**[Published current behavior]** Intake exposes ATIF, OTLP/HTTP protobuf, and OpenAI-compatible chat-completion ingress. These routes normalize source data into `IntakeSpan` records. Content that may require protection occurs in `input`, `output`, and retained raw attributes, often as JSON serializations of objects, arrays, or scalars. [Format reference](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/references/ingest-formats.md#L32-L161) · [Normalized span model](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/domain.py#L41-L61) + +| Format | Published current behavior | Design pressure, not current support | +| --- | --- | --- | +| ATIF | Accepts ATIF v1.0–v1.7; one trajectory can normalize into a recursive span tree | Preserve hierarchy and order while testing bounded context, trajectory coherence, and reviewed atomic groups | +| OTLP | Accepts OTLP/HTTP protobuf at the traces endpoint; a mixed-validity request can report per-span errors while retaining valid spans | Preserve trace and parent identity; reconcile per-span acceptance with source-item reconstruction and atomic-group policy | +| Chat completion | Accepts a captured request/response and creates one LLM span | Preserve structured request/response fields, tool content, source identity, and stable retry fields under a closed policy | + +**[Published current behavior]** Intake currently provides semantic or model-dumped fidelity, not byte-for-byte source fidelity. ATIF retains validated fragments under `atif.raw`; OTLP retains selected unconsumed attributes, events, and instrumentation scope; chat-completion ingress stores parsed request and response objects as JSON strings. Therefore, any protection adapter must preserve parsed structure and field identity rather than treat every payload as prose. [ATIF boundary](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/ingest/atif.py#L100-L138) · [OTLP receiver](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/ingest/otlp.py#L57-L122) · [Chat normalization](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/ingest/chat_completions.py#L143-L163) + +**[Published current behavior]** Provider-neutral direct-span ingress and importers for MLflow, LangSmith, Phoenix, and Braintrust also exist. They remain outside this initial three-format validation scope until separately reviewed. Their existence supports an open-ended source vocabulary, not an Anonymizer enum of supported sources. + +## Platform plugin impact + +**[Published current behavior]** The current NeMo Platform Anonymizer plugin pins `nemo-anonymizer==0.3.3` and calls only the public `Anonymizer` facade: `run()`, `preview()`, and `validate_config()`. It accepts CSV/Parquet and fileset-backed inputs, resolves providers and secrets, runs jobs, and publishes DataFrame-shaped result artifacts. The plugin—not Anonymizer—owns authentication, filesets, provider resolution, job state, storage, cancellation, artifact publication, download, and delivery. This released plugin is distinct from the proposed Intake workload adapter; there is no current plugin-to-Intake graph integration. [Plugin dependency](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/plugins/nemo-anonymizer/pyproject.toml#L10-L20) · [Run job](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/jobs/run.py#L41-L141) · [Preview worker](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/functions/_preview_worker.py#L31-L73) + +**[Proposal, inferred compatibility]** The private phases 1–3 should be compatible with the current plugin if current constructors and config types, `run()`, `preview()`, `validate_config()`, result columns and attributes, and `failed_records` shape do not change. This follows from inspected call sites; it has not yet been established by a cross-repository test against the uncommitted graph slice. + +**[Proposal]** A future public graph/session integration, if separately authorized, is not a drop-in replacement. Platform would need projection and reconstruction, capability negotiation, versioned artifacts, cancellation and cleanup review, OpenAPI and SDK regeneration for new schemas, and cross-repository tests. Platform must retain its existing lifecycle responsibilities. Preview cancellation is specifically unresolved because abandoning an async wait may leave the synchronous worker running. + +## Dated validation observations + +The operator-backed observations in this section are historical run evidence retained in the checked-in runbook and test contracts. In the 2026-08-20 verification run, all 11 opt-in Intake dogfood tests skipped because the external Intake/ClickHouse/Sandbox environment was not enabled. + +**[Dated dogfood observation — 2026-08-19]** The branch-local hermetic corpus exercised synthetic ATIF v1.0 and v1.7, extension-bearing chat-completion JSON, real OTLP protobuf bytes, and an Intake-shaped CHAIN-to-LLM trace through the private Redact profile. Tests checked complete-item reconstruction, topology preservation, closed field policy, and withholding for invalid spans or non-success outcomes. They did not use provider-backed detection, customer data, or durable Intake commit and do not establish production format support. [Hermetic adapter tests](https://github.com/NVIDIA-NeMo/Anonymizer/tree/d39e74f17ef090e84ca9c4fb86f47e6cee2ecd4d/tests/streaming) + +**[Dated dogfood observation — 2026-08-19]** An opt-in test-only run used an operator-owned local Intake service backed by ClickHouse 26.3. Raw and protected ATIF v1.0, ATIF v1.7, chat-completion JSON, and OTLP/protobuf traversed public Intake routes. The protected read models omitted the specifically declared synthetic test values and retained the asserted topology and semantic fields. This bounded observation is not evidence that all PII was absent. The checked-in runbook retains the validated revisions, environment, and local instance identity; the opt-in tests retain the asserted route behavior. [Dogfood run record](https://github.com/NVIDIA-NeMo/Anonymizer/blob/d39e74f17ef090e84ca9c4fb86f47e6cee2ecd4d/tools/intake_dogfood_runbook.md#L15-L34) · [Validated instance](https://github.com/NVIDIA-NeMo/Anonymizer/blob/d39e74f17ef090e84ca9c4fb86f47e6cee2ecd4d/tools/intake_dogfood_runbook.md#L274-L291) · [Opt-in dogfood tests](https://github.com/NVIDIA-NeMo/Anonymizer/blob/d39e74f17ef090e84ca9c4fb86f47e6cee2ecd4d/tests/streaming/test_intake_dogfood.py#L211-L636) + +**[Dated dogfood observation — 2026-08-19]** A completed isolated Sandbox session was mapped by a closed test-only adapter to ATIF v1.0. Deterministic local detection protected the declared synthetic values before the test request reached Intake, and the resulting read model retained the asserted parent-child topology. This validates the tested execution and provisional boundary path; it does not validate provider-backed detection quality, production Sandbox support, or a customer-approved “before Intake” boundary. [Sandbox export test](https://github.com/NVIDIA-NeMo/Anonymizer/blob/d39e74f17ef090e84ca9c4fb86f47e6cee2ecd4d/tests/streaming/test_intake_dogfood.py#L549-L580) + +**[Dated dogfood observation — 2026-08-19]** A mixed-validity OTLP request exposed an atomicity mismatch. The test adapter rejected the complete request and emitted no bytes, whereas Intake accepted the same raw request, returned one per-span error, and persisted the two valid spans. Complete-request withholding is therefore only a test-adapter policy pending adopter review. It is not current Intake behavior and does not alter current Anonymizer result-publication behavior. [Atomicity-mismatch test](https://github.com/NVIDIA-NeMo/Anonymizer/blob/d39e74f17ef090e84ca9c4fb86f47e6cee2ecd4d/tests/streaming/test_intake_dogfood.py#L375-L449) + +**[Dated dogfood observation — 2026-08-19]** A pre-connect delivery failure left the exact protected bytes available for a test retry and created no observed Intake row. Exact-byte retries collapsed to one public read-model row for the tested ATIF and OTLP fixtures. The initial chat fixture produced two public rows because it omitted a source timestamp; preserving one positive, non-future integer `response.created` caused the tested exact-byte retry to collapse to one public chat row. [Delivery-failure and retry tests](https://github.com/NVIDIA-NeMo/Anonymizer/blob/d39e74f17ef090e84ca9c4fb86f47e6cee2ecd4d/tests/streaming/test_intake_dogfood.py#L450-L636) + +**[Dated dogfood observation — 2026-08-19]** That last observation is format-specific read-model behavior. It is not evidence of transactional Intake idempotency, rollback, one physical write, or atomic persistence. A duplicate that is hidden by the read model can still represent more than one underlying write. + +## Findings for the proposal + +**[Proposal]** The evidence supports modeling target data, related context, replacement coherence, and atomic release independently. ATIF hierarchy, OTLP parentage and partial acceptance, and chat request/response structure cannot be represented faithfully by assuming that every independent DataFrame row is the complete semantic unit. + +**[Proposal]** A source adapter should own the codec, a closed target/context/preserve/reject field policy, stable source identity, graph projection, reconstruction state, output mapping, protected-byte retention for safe retries, and destination postcondition checks. It must pass only source-neutral graph semantics into Anonymizer. + +**[Proposal]** One complete ingestion item is the candidate buffered output-commit unit for adapter design, but Intake’s mixed-validity OTLP behavior proves that this cannot be stated as accepted Intake policy. The adapter and Intake owners must review each format’s partial-success and reconstruction contract. + +**[Proposal]** Unsupported context, coherence, atomicity, or dependency semantics must be rejected. Treating related datums as independent rows would erase the workload property being validated. + +**[Proposal]** Phase 4 selects a flat exact atomic partition as Anonymizer's first supported release model. That capability does not choose Intake's source-item or per-span grouping: the adapter and Intake owners must still review that mapping, and nested or overlapping groups remain unsupported. + +**[Proposal]** The reviewed phase 5 design keeps target and context in separate source-neutral +frames under an immutable bounded capability. It does not select which ATIF, OTLP, or chat +fields become targets or context; the source adapter and adopter owners retain that field +policy. The first profile also requires provider retention to be disabled. Any future +retention-enabled profile needs separate customer-owned privacy-boundary authorization. + +**[Proposal]** The reviewed phase 6 design anchors every mention and patch to authoritative +target offsets. Context may inform a reviewed validation, augmentation, or resolution task, +but it cannot supply a mention endpoint or replacement span. Source-field mapping, +reconstruction, source commit units, persistence, retries, destination deduplication, and +delivery therefore remain downstream responsibilities. + +## Open gates + +**[Unresolved gate]** The customer or consuming-product owner has not selected the earliest boundary that unprotected content may cross. “Before Intake” remains a provisional test posture, not approval. The decision must name the source adapter, optional edge component, Intake process, durable storage, operator-facing APIs and UI, and downstream consumers. + +**[Unresolved gate]** No accepted “zero PII” definition exists. An enforceable contract would need a reviewed closed detection and transformation policy, a selected trust boundary, release criteria, and residual-risk treatment. Current evidence supports no claim that all PII is absent. + +**[Unresolved gate]** Intake owners must define representative bounds and closed field roles for every relevant `input`, `output`, and raw-attribute surface in ATIF, OTLP, and chat-completion workloads. + +**[Unresolved gate]** The adapter contract still needs a reviewed provenance mechanism across source item, graph datum, atomic group, process, and persisted artifact without exposing raw detected entities or internal correlation tokens. + +**[Unresolved gate]** Stable public promotion still needs a materially different semantic runtime. Multiple Intake formats, hermetic fixtures, a process-backed Python host, and test-only Sandbox or OpenShell adapters do not satisfy that gate. + +**[Unresolved gate]** Future Platform adoption needs versioned projection and artifact contracts, capability negotiation, cancellation and cleanup behavior, OpenAPI/SDK regeneration, and cross-repository tests. + +## Ownership retained downstream + +| Owner | Work that remains outside Anonymizer | +| --- | --- | +| Source adapter | Format validation; closed field policy; source IDs; projection; reconstruction; protected-byte retention; retry classification; destination postcondition | +| NeMo Platform Anonymizer plugin | Current public-facade integration; authentication; filesets; provider resolution; jobs; cancellation; storage; artifacts; delivery | +| Future Intake integration | Ingress; source-item persistence and partial acceptance; durable retries; retention; cleanup; destination deduplication; delivery | +| Customer or consuming-product owner | Trust boundary and accepted privacy objective | + +## Evidence references + +- [NeMo Platform formats at `e1057736703bb8b167a4bd9013cea0caae2df63a`](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/references/ingest-formats.md#L32-L161) +- [Normalized `IntakeSpan`](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/domain.py#L41-L61) +- [ATIF normalization](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/ingest/atif.py#L100-L138) +- [OTLP/HTTP protobuf receiver](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/ingest/otlp.py#L57-L122) +- [Chat-completion normalization](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/ingest/chat_completions.py#L143-L163) +- [Branch-local validation tests at `d39e74f17ef090e84ca9c4fb86f47e6cee2ecd4d`](https://github.com/NVIDIA-NeMo/Anonymizer/tree/d39e74f17ef090e84ca9c4fb86f47e6cee2ecd4d/tests/streaming) +- [Retained local dogfood run record](https://github.com/NVIDIA-NeMo/Anonymizer/blob/d39e74f17ef090e84ca9c4fb86f47e6cee2ecd4d/tools/intake_dogfood_runbook.md#L274-L291) +- [Technical proposal](graph-native-anonymizer-sdk-technical-proposal.md) +- [Phase 5 target and bounded-context workframe design](phase-5-target-context-workframe-design.md) +- [Phase 6 anchored-mention, resolution, and local-verification design](phase-6-anchored-mention-resolution-design.md) + +## Next evidence action + +Have the Intake owner review bounded workloads, field roles, source commit units, retry identity, and partial reconstruction for the three initial formats. Then run cross-repository compatibility tests and rerun opt-in dogfood only in an operator-owned environment under the selected privacy boundary. +These are future adapter-adoption gates; they do not block the authorized private Phase 4 branch implementation. The RFC and its public or production adoption remain under review. diff --git a/docs/development/phase-4-hierarchical-terminal-accounting-design.md b/docs/development/phase-4-hierarchical-terminal-accounting-design.md new file mode 100644 index 00000000..695fcfb4 --- /dev/null +++ b/docs/development/phase-4-hierarchical-terminal-accounting-design.md @@ -0,0 +1,593 @@ + + + +# Phase 4 design — Hierarchical terminal accounting + +Status: branch implementation checkpoint complete. The private branch-local implementation +and its evidence gates passed on 2026-08-25. This document is subordinate to the complete +development and research plan in the +[graph-native SDK RFC](graph-native-anonymizer-sdk-rfc.md). The checkpoint permits Phase 4 +work on this branch, but it is not a separate project acceptance decision and does not +authorize a public graph API, production Intake or OpenShell integration, or stable +promotion. + +Review status: independent architecture and test-strategy council review completed on +2026-08-20 with zero unresolved Critical or Warning findings. On 2026-08-20, the operator +authorized the Phase 4 branch checkpoint after review of its outcome algebra, policies, +boundaries, and test strategy. This is development authorization, not an approving GitHub +review or RFC acceptance. A separate implementation-remediation council completed on +2026-08-25 with zero unresolved material findings. + +## Decision + +**[Branch-local implementation]** Phase 4 adds a private, source-neutral execution ledger that accounts for +every admitted invocation, stage, logical task, datum, dependency, and atomic group. The +ledger permits output only after exhaustive reconciliation and atomic-group qualification. +It does not turn a DataFrame row, a backend response, or a source-format item into the +semantic identity of a datum. + +**[Branch-local implementation]** The first supported related-record profile uses explicit datum dependencies +that form a directed acyclic graph and atomic groups that form a flat, exact partition of +the target datums. Cycles, nesting, overlap, incomplete group coverage, implicit +dependencies, and unsupported task cardinalities are rejected before effects. Phases 5–10 +remain sequenced after the implemented Phase 4 passes the evidence gates below and their +branch checkpoints are authorized. + +## Scope and preserved boundaries + +The current branch-local phases 1–3 provide immutable private graph values, a compiler for +independent datums, temporary DataFrame lowering, invocation-private row correlation, and a +fail-closed Redact release profile. Phase 4 extends that seam; it does not replace the +published facade or claim that the graph architecture is current public behavior. + +The implementation must preserve these boundaries: + +- `_DatumId` remains immutable and graph-scoped through every graph phase. Source identity + stays in the adapter's private reconstruction map; DataFrame index, row order, text, and + caller identifiers are not datum identity. +- The public constructors, config types, defaults, `run()`, `preview()`, `evaluate()`, + `validate_config()`, CLI behavior, result columns and attributes, `trace_dataframe`, and + `failed_records` shape remain compatible. +- A stage may use a temporary DataFrame workframe and declare DataDesigner columns, but + `NddAdapter.run_workflow()` remains the sole boundary for executing DataDesigner + workflows. +- Anonymizer owns source-neutral validation, scheduling, terminal accounting, protection + verification, and release qualification. Downstream owners retain codecs, closed field + policy, source identity, projection, reconstruction, persistence, retries, + deduplication, cleanup, retention, and delivery. +- New phase-4 runtime tokens and diagnostics remain opaque, invocation-local, bounded, and + content-free. They must not expose source content, detected values, prompts, source IDs, + or content-derived hashes. The existing public `FailedRecord` representation is a + grandfathered compatibility surface, not phase-4 identity or a proposed diagnostic + design. + +Phase 4 does not add context or coherence semantics, grouped rewrite, durable state, +delivery, or an adapter-accessible provenance artifact. Those capabilities remain later +phases or unresolved gates. + +## Pure compilation before effects + +Compilation is the graph admission boundary. It consumes a proposed graph, limits, strategy, +and declared runtime capabilities and either returns one immutable compiled plan or one +bounded, content-free rejection. The compiler is pure: it does not use providers, +credentials, runtime resources, telemetry, workframes, or an execution context. It must +finish before phase-4 invocation identity, ledger or row-token creation, content-bearing +workframe lowering, backend dispatch, or any `NddAdapter.run_workflow()` call. + +The existing public facade is already an open host: construction initializes logging, +selects providers, and creates the DataDesigner adapter, and public call telemetry may begin +before private graph compilation. Phase 4 does not move or repeat those compatibility +effects. Its guarantee is that an internally generated compatibility graph is compiled +before any graph-invocation or protection effect and that rejection causes no additional +provider use, credential access, resource acquisition, content-bearing telemetry, +workframe, reconstruction, or publication. A future raw-graph/session entry would need a +separate reviewed pre-binding API; this phase does not introduce one. + +```text +proposed graph + limits + declared capabilities + -> pure validation and compilation + -> rejected # no invocation and no effects + or + -> immutable compiled plan + -> invocation-local execution + -> temporary ready-frontier workframes + -> existing pandas workflows + -> NddAdapter.run_workflow() + -> reconcile, verify, close ledger + -> atomic-group release decision +``` + +The graph runtime accepts only a compiled plan. It must not discover unsupported graph +semantics after it opens an execution context. After public input loading and preview +selection, the compatibility facade compiles its internally generated independent-row +workload to a no-dependency graph with singleton atomic groups. + +Validation uses deterministic precedence so malformed structure does not appear as a +capability decision: + +1. Reject a malformed outer value or unsupported private schema version. +2. Reject count and byte-limit violations. +3. Reject malformed or duplicate datum, dependency, task, stage, or group declarations. +4. Reject unknown references, wrong-purpose references, empty groups, duplicate members, + and atomic coverage gaps. +5. Reject self-dependencies, duplicate dependency edges, partial group overlap, and cycles. +6. Reject recognized but unsupported atomic nesting, relation, context, coherence, or task + cardinality. +7. Reject an unsupported strategy or runtime capability. + +The implementation may use more specific safe codes within these classes. The same invalid +input must select the same code regardless of declaration order or runtime configuration. + +## Dependency and atomic-group policy + +Dependencies are first-class values with `prerequisite` and `dependent` endpoints. They +express execution and release prerequisites between target datums. They are never inferred +from datum declaration order, a source parent-child relation, context, coherence, or atomic +membership. `_DatumLink.RELATED` must not be reinterpreted as a dependency. + +Every endpoint resolves to exactly one admitted target datum. Self-edges, duplicate edges, +dangling endpoints, and every directed cycle are invalid. Declaration order is only a +deterministic tie-break among ready tasks and a presentation-order input to the compatibility +adapter; it has no dependency meaning. + +Atomic groups form a flat exact partition: + +- Every target datum belongs to exactly one explicitly declared, non-empty group. +- Compilation assigns each group an opaque graph-scoped identity; tuple position and member + contents are not group identity. +- Members are unique target datum IDs. Equal duplicate groups and implicit singleton groups + are invalid. +- Partial overlap, such as `{a, b}` and `{b, c}`, is invalid. +- Strict nesting, such as `{a}` inside `{a, b}`, is recognized but unsupported in phase 4. +- Member order and group declaration order have no semantic effect. + +Flat partitioning can express the current design choices between independently releasable +datums and a whole-set fail-together boundary. Nested source commit units remain a downstream +concern unless a later Anonymizer capability defines their protection semantics. Intake's +mixed-validity OTLP behavior is evidence that an adapter owner must choose the mapping; it +is not an Anonymizer default. + +## Execution hierarchy and identity + +One invocation owns an immutable compiled plan and a one-shot ledger. A stage is a +compiler-owned Anonymizer accounting boundary with a fixed per-datum predecessor rule and +typed result; it is not a global dispatch barrier, DataDesigner task, column, scheduler +step, or `FailedRecord.step`. A later-stage task may become ready after its own fixed +predecessor succeeds while an unrelated earlier-stage task remains open. Requiring global +stage closure would deadlock a multi-stage plan whose dependent datum waits for its +prerequisite datum to become locally qualified across every stage. The production +phase-4 Redact profile has one effectful `protect` stage that wraps the existing complete +pandas detection-and-replacement path. Its release checks are pure reducers after that +stage. Synthetic ledger plans may use up to three semantic stages to prove hierarchy. Later +phases may add reviewed semantic stages, but changing stage inventory is a compiler decision, +not an inference from backend traces. + +Each stage expands to one logical task for every datum that requires it. A logical task is +identified by an opaque `(invocation, stage, datum)` identity; later phases must separately +review group-scoped tasks before admitting them. + +The executor derives a task DAG from fixed stage ordering and explicit datum dependencies. +It may batch any ready frontier into a DataFrame call, but a batch and its rows are transport +details, not accounting units. Each lowered row carries a fresh opaque task token. Hydration +must prove a bijection between the expected ready tasks and the returned terminal records; +it never joins by position, index, text, or caller identity. + +The hierarchy has these obligations: + +```text +invocation + stage + logical task: exactly one target datum and one terminal task outcome + datum: exactly one terminal datum outcome after all required tasks close + dependency: unresolved until its prerequisite closes, then satisfied or unsatisfied + atomic group: exactly one released or withheld decision +``` + +A stage result may close only after every logical task declared for it has a terminal outcome. A +datum may close only after every required task has a terminal outcome or has been +deterministically blocked. An invocation may return a result only after every admitted unit +is terminal and every dispatched backend attempt has either an accepted task-terminal +classification grounded in trusted run evidence—including localized +`inconsistent(missing)`—or a `lost` classification. + +## Closed outcome algebra + +Admission has two outcomes: `compiled`, which creates no execution evidence yet, and +`rejected`, which creates no invocation and performs no effect. Runtime states such as +`pending`, `ready`, `running`, and `cancellation_requested` are not terminal outcomes and +must never appear in a returned terminal result. + +Logical task outcomes are closed and mutually exclusive: + +| Outcome | Meaning | Output candidate | +| --- | --- | --- | +| `succeeded` | Exactly one trusted result passed task-level schema and provenance checks | May contribute | +| `failed` | Execution returned a known failure or task-level verification failed | None | +| `cancelled` | The task did not run, or the execution boundary proved that it stopped without a usable result | None | +| `lost` | Dispatch may have occurred but no trusted terminal record proves success, failure, or cancellation | None | +| `blocked` | A prerequisite closed without local qualification before this task was dispatched | None | +| `inconsistent` | Returned accounting was missing, duplicated, foreign, stale, contradictory, or otherwise unverifiable | None | + +A datum is `locally_qualified` only when every required logical task succeeded exactly once +and the datum-level release predicate passed. A datum-level predicate rejection is +`failed(release_predicate_failed)`. Other non-successes produce the corresponding terminal +`failed`, `cancelled`, `lost`, `blocked`, or `inconsistent` outcome. A locally qualified +datum holds an internal output candidate, but local qualification alone does not expose +output. + +A dependency is nonterminal `unresolved` until its prerequisite datum closes. It then has +the execution outcome `satisfied` when that datum is `locally_qualified` before atomic-group +propagation, or `unsatisfied` with only the prerequisite's bounded cause classes otherwise. +Dependencies never contain protected content. An unsatisfied dependency blocks an +undispatched dependent task. Release propagation separately follows prerequisite datum +eligibility, so a later atomic-peer failure can withhold an already-run dependent without +rewriting the dependency's execution outcome. + +An atomic group is either `released` or `withheld`. Propagation derives a separate +`release_eligible` bit without rewriting the datum's terminal outcome. A group is released +only when every member datum is `locally_qualified` and remains `release_eligible`, and the +group release predicate passes. `released` is the only terminal outcome that contains +outputs, and it contains the complete group in graph datum presentation order. `withheld` +contains no output; raw input is never substituted for a withheld result. + +Stages use the same non-success evidence classes as tasks and otherwise close as +`succeeded`. Their outcome summarizes child evidence and does not alter it. An invocation +closes as: + +- `completed` when exhaustive, internally consistent accounting produced a trusted group + result, even if some groups were withheld; +- `failed` when a known invocation-scoped failure prevented a normal group result; +- `cancelled` when invocation cancellation closed all unfinished work but no normal group + result is returned; +- `lost` when the execution boundary cannot establish a trusted invocation run record; or +- `inconsistent` when global reconciliation cannot prove which declared units the evidence + describes. + +Invocation-level `failed`, `cancelled`, `lost`, and `inconsistent` outcomes expose no graph +output. They still close every admitted child unit with a terminal outcome for private +accounting. + +When a datum or stage has several non-success children, its terminal variant follows +`inconsistent` → `lost` → `cancelled` → `failed` → `blocked`, while retaining an ordered, +deduplicated tuple of every bounded cause. Atomic groups retain the same cause tuple under +the single `withheld` variant. The precedence gives each scope one mutually exclusive +terminal variant without erasing concurrent evidence. + +The logical-task transition relation is closed: + +| Current state | Accepted event or guard | Next state | +| --- | --- | --- | +| `planned` | No earlier stage exists, or it succeeded; every direct dependency is `satisfied` | `ready` | +| `planned` | Any direct dependency closes as `unsatisfied` | terminal `blocked` | +| `planned` or `ready` | Cancellation is accepted before dispatch | terminal `cancelled` | +| `ready` | One dispatch is committed with a fresh attempt and row token | `dispatched` | +| `dispatched` | One exact keyed result passes reconciliation | terminal `succeeded` | +| `dispatched` | A definitive execution or verification error is accepted | terminal `failed` | +| `dispatched` | A trusted stop acknowledgement is accepted first | terminal `cancelled` | +| `dispatched` | Completion and stop become unobservable | terminal `lost` | +| Any nonterminal state | Evidence makes safe attribution impossible | terminal `inconsistent` | + +No other transition is valid. `cancellation_requested` is an invocation flag, not a task +state; it changes a task only through the cancellation rows above. Terminal task states are +absorbing. Dependencies and parent scopes are deterministic reducers over terminal child +evidence rather than independently mutable state. + +## Terminal precedence and reconciliation scope + +Terminal states are absorbing. A later completion cannot reopen a task that is already +`cancelled`, `lost`, `blocked`, or `inconsistent`, and a terminal attempt token cannot be +reused. When several observations compete, the ledger applies this precedence: + +1. An integrity contradiction that prevents safe attribution is `inconsistent`. +2. A previously accepted trusted terminal record remains authoritative. +3. A dispatched execution without a trusted terminal or stop record is `lost`. +4. Proven cancellation is `cancelled`. +5. A known execution or verification failure is `failed`. +6. An unscheduled task with a terminal unsatisfied prerequisite is `blocked`. + +The ledger localizes a reconciliation fault only when it can still prove the complete +expected-to-observed bijection for every unaffected task. After a trusted batch run record +arrives, a missing expected token is local `inconsistent(missing)`; it is not `lost`. When no +trusted run record exists for a dispatched task, the task is `lost`. An unknown token, +duplicate token in one reconciliation set, plan mismatch, or contradictory initial record +destroys the batch or invocation bijection and closes the invocation as `inconsistent`, +withholding all groups. Implementations must not guess a narrower scope to preserve output. + +The ledger serializes terminal acceptance. A byte-identical replay received after its +terminal record was accepted is an idempotent stale observation; any other late record for +that closed attempt is rejected as stale and cannot change state. A duplicate, foreign, +stale-at-admission, or contradictory record present before the first terminal acceptance is +an accounting inconsistency. Thus cancellation acknowledged before a late success remains +`cancelled`, while competing unsequenced evidence is `inconsistent`. + +Existing `FailedRecord` values are evidence about dropped backend rows. The phase-4 adapter +must reconcile each value to one expected task by opaque token before it maps it to a task +failure. It must preserve the published `failed_records` compatibility shape at the facade; +unattributable or contradictory failures are invocation inconsistency, not anonymous row +loss. Existing content-derived public record IDs retain their published behavior only at +that facade. They must not be copied into graph outcomes or receipts, used for correlation, +or treated as proof of datum identity; exact public `step`, ID, ordering, and shape behavior +requires compatibility tests. + +## Dependency propagation and release + +Scheduling and release use different predicates. A task becomes ready when its fixed +earlier-stage task succeeded and every direct datum prerequisite is locally qualified: all +of that prerequisite's required tasks and datum-level release predicate passed, before +atomic-group propagation. This rule prevents known-bad prerequisites from causing more +effects. A dependent that already finished may later become release-ineligible when an +atomic peer of its prerequisite fails; its successful task evidence remains unchanged. + +After every task is terminal, release qualification computes the least fixed point of these +monotone rules: + +1. Mark every datum that is not `locally_qualified` ineligible. +2. Mark every member of an atomic group that contains an ineligible datum ineligible for + release. Preserve each member's underlying datum outcome. +3. Mark every transitive dependent of an ineligible datum ineligible for release. +4. Repeat group and dependency propagation until no eligibility changes. + +This fixed point is necessary because an atomic peer may feed a dependent in another group, +and that dependent's group may contain further peers. Failure affects an unrelated group +only through an explicit dependency path. Precisely accounted independent groups may be +released together in the final result even when another group is withheld. Any +invocation-global inconsistency withholds all groups. + +## Cancellation, lost execution, and retry + +A cancellation request is an event, not proof that execution stopped. Before dispatch, it +closes the task as `cancelled` without a backend call. After dispatch, a task is `cancelled` +only when the execution boundary provides a trusted stop acknowledgement. If a caller +abandons a waiter, a transport breaks, or a worker disappears without that evidence, the +task or invocation is `lost`. Likely completion is not evidence of success. + +The terminal-record acceptance point is the cancellation linearization point. A trusted +terminal record accepted first wins. If proven cancellation closes the task first, a late +result is stale and cannot resurrect output. Cancellation after a task or invocation is +terminal has no semantic effect. Phase 4 must not claim that cancelling the public caller's +waiter stops synchronous DataDesigner or model work. + +An invocation cancellation request accepted before the final publication point sets a +release embargo. The ledger still closes every admitted child, but the invocation terminates +as `cancelled` and exposes no group output, even if all effectful tasks had already +succeeded. Publication accepted first wins; a later cancellation request has no effect. + +Phase 4 performs no automatic task retry. One dispatch is one attempt in the ledger; +provider- or client-internal behavior below `NddAdapter.run_workflow()` is opaque and creates +no Anonymizer retry guarantee. A downstream owner may start a new Anonymizer invocation, +which receives fresh opaque identities and reprocesses the declared input. The new call does +not reopen the lost attempt, reuse its output, deduplicate an effect, or imply durable +idempotency. Protected-byte retention, destination postcondition checks, and delivery retry +remain downstream responsibilities. + +## Fail-closed release algorithm + +The release barrier runs once, after exhaustive terminal accounting: + +1. Verify that the compiled plan identity and all expected invocation, stage, task, datum, + dependency, and group identities have exactly one compatible terminal record. +2. Reject or classify every missing, extra, duplicated, stale, or contradictory record + according to the terminal-acceptance rules above. +3. Derive datum qualification without trusting workframe order or contents as identity. +4. Compute the dependency-and-group ineligibility fixed point. +5. Run the strategy-specific datum and group release predicates. +6. Construct complete outputs only for released groups. +7. Remove all invocation-private tokens and verify that no withheld output, raw fallback, + or private diagnostic enters the returned compatibility result. +8. Publish one immutable terminal result. If result construction or publication + verification fails, expose no graph output. + +No iterator, callback, trace view, exception, partial DataFrame, or diagnostic may expose a +member output before its group passes this barrier. Atomic release here is a Python result +qualification rule, not a transaction over Platform artifacts, Intake persistence, +providers, telemetry, or delivery. + +## Test strategy + +### Independent reference model + +Implement a small pure model before the executor. Its inputs are a compiled declaration and +a timestamp-free sequence containing only exogenous observations: dispatch acceptance, +terminal evidence, cancellation request, stop acknowledgement, and transport loss. The +model derives readiness, reconciliation, propagation, release, and the only legal task, +datum, dependency, stage, group, and invocation outcomes. Runtime scheduling or release +decisions are not model inputs. Review this oracle independently from the executor. + +The model must enforce these equations: + +```text +task ready = no earlier-stage task exists or it succeeded + and every direct datum dependency is satisfied + +datum locally qualified = every required task has exactly one accepted success + and the datum release predicate passes + +group released = every member is locally qualified and release eligible + and the group release predicate passes + +invocation completed = every admitted unit is terminal + and every dispatch has a trusted-evidence terminal classification + or is lost + and reconciliation is globally consistent +``` + +### Example and transition tests + +Unit tests must cover every permitted transition and reject every transition out of a +terminal state. The minimum graph cases are empty and malformed input, singleton +compatibility, wide and deep DAGs, a diamond DAG, disconnected components, declaration +order different from topological order, and maximum accepted limits. + +An empty datum set is `malformed_graph`, matching the phases 1–3 compiler; it is not a +vacuous completed invocation. Admission tests include one family for every validation tier, +exact-limit and limit-plus-one cases for every count and byte bound, adjacent-tier +multiply-invalid inputs, declaration permutations, and capability permutations. They must +prove the documented rejection precedence. + +Admission tests must cover dangling endpoints, duplicate and self dependencies, cycles of +different lengths, empty groups, duplicate members, duplicate groups, coverage gaps, +partial overlap, strict nesting, and every still-unsupported context, coherence, relation, +strategy, and task cardinality. Effect spies start at graph compilation and require zero +invocation/ledger/token creation, execution-context opening, phase-4 resource acquisition, +provider or credential use, content-bearing workframe or telemetry, backend or NDD call, +reconstruction, and publication. Content-free compiler intermediates and pre-existing public +facade construction or request-start telemetry are outside this spy window. + +Plan and ledger tests mutate source declarations after compilation, attempt nested plan +tampering, replay a compiled plan where disallowed, reuse a closed ledger, concurrently open +the same one-shot invocation twice, accept a second terminal record, and publish twice. The +immutable snapshot must remain detached, and every second one-shot action must fail closed. + +Execution tests must inject known failure, cancellation before and after dispatch, stop +acknowledgement, transport loss, worker death, partial backend results, missing rows, +duplicate and foreign tokens, stale attempt results, contradictory records, hydration +failure, release-predicate failure, and result-construction failure. They must prove exact +terminal conservation, dependency blocking, fixed-point withholding, and no raw fallback. +Independent-group isolation applies only to localizable evidence in a globally consistent +invocation; a global attribution fault must withhold every group. + +An identity cross-product uses equal text, equal structured values, duplicate and null-like +public indices, distinct datum IDs, multiple synthetic stages, and fresh invocations. It +independently permutes rows, terminal records, and `FailedRecord` evidence and covers a valid +token bound to the wrong stage or datum, swapped valid tokens, token reuse or collision, +plan mismatch, one attributable failure, and duplicate, unknown, stale, or +success-plus-failure evidence. + +Cancellation tests fix the linearization expectations: accepted terminal evidence before +stop acknowledgement wins; accepted stop acknowledgement before a late result is +`cancelled`; a request after closure changes nothing; a post-dispatch request without +acknowledgement, waiter abandonment, or transport loss is `lost`. Pre-dispatch cancellation +or blocking causes zero dispatches; every task that reached `dispatched` causes exactly one, +including tasks that later fail, cancel, or become lost. A downstream re-invocation uses +fresh identities. + +A minimal alternating group/dependency graph must require more than one fixed-point round. +One schedule lets a dependent finish before an atomic peer of its prerequisite fails. The +dependent remains locally successful but is withheld, and a disconnected group is +unchanged. + +At least one process-kill smoke test uses a test-only crashable backend at the existing +execution seam and compares worker death with the pure model's `lost` result and empty +release set. It must not add a production process/session API. This supplies phase-4 +failure-boundary evidence only; it is not a materially different semantic runtime or the +lifecycle proof reserved for phase 11. + +### Generative and schedule tests + +The exhaustive conformance envelope is finite: zero through four declared datums, one +through three synthetic semantic stages for non-empty graphs, one through the datum count +atomic groups with every flat partition enumerated, one dispatch per logical task, one +primary terminal observation per dispatch, at most one late +or contradictory observation per dispatch, and at most one cancellation request, stop +acknowledgement, and transport-loss event per invocation. Reconciliation corruption is one +of the closed missing, duplicate, unknown, foreign, stale, swapped, plan-mismatch, or +contradictory classes. Schedules that differ only by commuting events for independent tasks +share one canonical representative. The harness must emit a checked manifest containing +the generator version, exact graph count, exact canonical event-trace count, and a digest; +review freezes that manifest before executor comparison. Use seeded property tests beyond +this envelope and retain minimized failing graphs and traces. + +The required properties are: + +- **Conservation:** every admitted unit has exactly one terminal outcome. +- **Output conservation:** released output keys equal the group member set exactly once, + follow the compatibility presentation order, and belong to no other group or invocation; + withheld groups contain no output. +- **Reference equivalence:** the implementation equals the pure model. +- **Permutation invariance:** declaration, ready-frontier, workframe-row, and response order + preserve ID-keyed/isomorphic semantics; compatibility presentation order may follow the + admitted input order. +- **Identity invariance:** an opaque identity renaming yields an isomorphic result. +- **Content non-identity:** equal text and structurally equal values remain distinct datums. +- **Monotone withholding:** replacing any required success with a non-success never enlarges + the release set. +- **Independent isolation:** a localizable failure in a globally consistent disconnected + component cannot change another component; global attribution faults are excluded. +- **Batch invariance:** different ready-frontier batch sizes produce the same result. +- **Attempt isolation:** stale, foreign, or prior-invocation records cannot satisfy a task. +- **Boundedness and confidentiality:** accounting respects limits and emits no content or + content-derived diagnostic. + +Mutation tests must demonstrate that the suite detects positional joins, omitted terminal +records, premature group release, skipped propagation, accepted duplicate results, stale +attempt adoption, raw-input fallback, and late-result resurrection. + +### Compatibility and boundary tests + +Compatibility tests must exercise duplicate text; reordered, filtered, concatenated, and +index-reset workframes; unique, duplicate, non-monotonic, string, and null-like public +DataFrame indices; mixed successful and failed rows; and private-column collision. They must +verify original public presentation order, user columns, result columns and attributes, +`trace_dataframe`, `failed_records`, CLI behavior, and the sanitized public error surface. +Invocation tokens must not appear in public DataFrames, receipts, exceptions, tracebacks, +logs, or persisted fixtures. Preview selection must occur before graph compilation, and a +graph executor must never silently omit an admitted datum to reproduce preview sampling. + +Privacy tests define a closed allowlist of public fields and bounded values. They inject +independent canaries for content, source identity, graph identity, row tokens, and their +known digests; capture `repr` and `str`, exception cause/context and traceback, logs, +telemetry, `trace_dataframe`, receipts, and serialization sinks; and reject every field not +on the allowlist. Canary substring scans supplement this structural check but do not stand +alone as proof that arbitrary content-derived values are absent. The grandfathered public +`FailedRecord` fields are checked against their compatibility contract separately. + +Boundary tests must prove that all DataDesigner execution still passes through +`NddAdapter.run_workflow()` and that compilation performs no effects. Source-shaped ATIF, +OTLP, and chat-completion fixtures may test adapter projection pressure only after their +owners review the relevant field and atomic mappings. The tests must not encode complete +request withholding as current Intake behavior, claim byte-exact source fidelity, or import +source types into Anonymizer core. + +## Review and implementation gates + +The operator authorized implementation after reviewers completed review of: + +1. the closed outcome algebra and precedence; +2. the DAG-only dependency policy and fixed-point propagation; +3. the flat exact atomic partition and rejection of nesting and overlap; +4. the cancellation linearization point, `lost` semantics, and no-retry decision; +5. the pure compile-before-effects barrier and global-integrity scope; +6. the independent reference model and test matrix; and +7. the preserved public, DataFrame, NDD adapter, privacy, and downstream ownership + boundaries. + +The 2026-08-20 review and operator checkpoint satisfied the seven pre-implementation +decisions above. +The Phase 4 completion gate requires the reference-model, transition, fault-injection, bounded schedule, +property, mutation, compatibility, privacy-canary, and full repository suites pass and an +independent reviewer accepts the evidence. Before later Platform adoption or stable +promotion, the NeMo Platform plugin owner must turn the current call-site compatibility +inference into executed public-facade smoke-test evidence. That cross-repository test is not +a prerequisite for starting the private phase 4 implementation. + +The branch-local implementation passed these gates on 2026-08-25. The frozen `phase4-stream-v4` +reference corpus covers 8,278 admitted graphs and 397,542 canonical traces; its manifest digest is +`e778147bf77909ddb94117fe7e6c230de57e46a722fad49c563b36f0b5660efa`. The full repository run +reported 1,508 passed and 11 skipped tests with one pre-existing deprecation warning. Formatting, +type checking, documentation, privacy-canary, compatibility, process-loss, concurrency, and +mutation checks passed. The independent remediation verifier accepted all nine council +remediations with no remaining material findings. The maintained authenticated review-only +Arc then accepted the focused remediation with zero findings and passed every configured +host validation. + +Passing Phase 4 does not authorize the Phase 5–10 branch checkpoints automatically, a public +graph or session surface, production Intake or OpenShell support, a wire protocol, transactional persistence, +an accepted privacy boundary, a `zero PII` claim, or stable promotion. It only satisfies the +phase-4 prerequisite after its evidence is reviewed. + +## Evidence and unresolved gates + +The workload evidence motivates the design but does not approve source mappings. ATIF +hierarchy, OTLP partial acceptance, and structured chat request/response fields demonstrate +why datum identity, dependencies, and atomic release cannot be reduced to DataFrame rows. +Intake owners still need to approve closed field roles, source commit units, retry identity, +and partial reconstruction. The customer or consuming-product owner still needs to select +the PII trust boundary and acceptance objective. Opaque provenance across adapter, process, +and artifact boundaries also remains unresolved. These are future adapter-adoption and +promotion gates, not prerequisites for implementing the authorized private Phase 4 design. + +The authoritative evidence and constraints for this phase are the parent +[technical proposal](graph-native-anonymizer-sdk-technical-proposal.md), the separate +[Intake workload evidence](intake-workload-validation-evidence.md), the published facade and +`NddAdapter.run_workflow()` contracts cited there, and the branch-local phases 1–3 graph, +runtime, release, and verification tests at `702f43a988cf3673d16f40be5c59bc784737e1a3`. diff --git a/docs/development/phase-5-target-context-workframe-design.md b/docs/development/phase-5-target-context-workframe-design.md new file mode 100644 index 00000000..3834a269 --- /dev/null +++ b/docs/development/phase-5-target-context-workframe-design.md @@ -0,0 +1,697 @@ + + + +# Phase 5 design — target and bounded-context workframes + +Status: reviewed phase-specific design and test strategy; private branch-local implementation +and hardening landed on 2026-08-26 in `bb79cda` through `53ef74f`. This document is +subordinate to the complete development and research plan in the +[graph-native SDK RFC](graph-native-anonymizer-sdk-rfc.md) and the +[phase 4 terminal-accounting design](phase-4-hierarchical-terminal-accounting-design.md). +RFC acceptance remains the project decision on the plan. The branch-local implementation +does not authorize a public graph or session API, production Intake or OpenShell integration, +or a privacy-boundary decision. + +Review status: independent architecture and test-strategy council review completed on +2026-08-20 with zero unresolved Critical or Warning findings for the prior draft. The +2026-08-21 revision moves product authorization and field policy outside Anonymizer, defines +preflight and private work-ID terminology, and adds the content-free observation and cleanup +contracts while retaining typed context limits and backend compatibility. Focused +architecture and test-strategy re-review completed with zero unresolved Critical or Warning +findings. The design review is complete, and the branch now contains the Phase 5 +implementation, frozen independent reference model, and focused context admission, execution, +reconciliation, cleanup, privacy, and compatibility tests. The 2026-08-31 PR checks pass; +this evidence remains private to the branch and does not establish product or publication +support. + +## Decision + +Phase 5 should add one private compiled projection for every output-bearing target datum. +The projection keeps target text and explicitly declared context datums in separate, +bounded frames and binds both to one phase 4 logical target task with private, +invocation-local work IDs. + +Phase 5 qualifies framing, minimization, lowering, reconciliation, and +cleanup. It does not allow context to create entity mentions, change detection decisions, +join entity clusters, grant replacement coherence, or become output. Phase 6 must separately +review and qualify any context-informed detection or resolution semantics. + +Context scope, dependency, coherence scope, and atomic group remain independent. A declared +relationship, dependency, common group, source adjacency, equal text, or shared context does +not grant context-read authority or replacement sharing. + +## Preconditions and preserved boundaries + +Phase 5 assumes phase 4 has implemented and qualified: + +- pure graph compilation before graph-invocation effects; +- an immutable compiled plan and one-shot ledger; +- exhaustive task, target datum, dependency, atomic-group, stage, and invocation outcomes; +- work-ID-keyed backend reconciliation, cancellation, lost-execution, and no-retry rules; + and +- fixed-point dependency and atomic-group withholding. + +The implementation must preserve these boundaries: + +- `_DatumId` remains immutable and graph-scoped. Text, source identity, DataFrame index, + row order, context position, and content-derived hashes are not datum identity. +- Current public constructors, configuration, `run()`, `preview()`, `evaluate()`, + `validate_config()`, result columns and attributes, `trace_dataframe`, `failed_records`, + CLI behavior, and errors remain compatible. +- DataFrames are temporary workframes. `NddAdapter.run_workflow()` remains the sole boundary + for executing DataDesigner workflows. +- The graph remains authoritative before lowering and after hydration. A row, batch, or + context fragment is not a semantic output unit. +- Anonymizer owns source-neutral context grammar, validation, projection, correlation, + accounting, and release qualification. Source adapters retain codecs, source identity, + field policy, projection proposals, reconstruction, persistence, retries, deduplication, + cleanup, retention, and delivery. The integrating product authorizes source access and + field use before its adapter constructs the graph. Provider access and credentials remain + outside Anonymizer's graph semantics. + +The existing public DataFrame path continues to compile an empty-context profile. Phase 5 +must not add an empty context section to legacy prompts, change public model inputs, or alter +row-local behavior merely because the private framing machinery exists. + +## Closed semantic model + +Phase 5 extends the private graph grammar with two datum purposes: + +```text +target datum — output-bearing text processed by phase 4 tasks +context-only datum — read-only text that may inform a declared target task +``` + +A target datum may also be referenced as read-only context for another target. A datum has +one immutable purpose, but a target reference used as context does not change the referenced +datum's own task, dependency, atomic-group, or output semantics. + +A target reused as context contributes the immutable source text captured in the admitted +graph and compiled projection. It never contributes transformed output or live task state. +Its own success, failure, cancellation, loss, group eligibility, or release cannot change an +already compiled context binding. A context reference creates no scheduling edge in either +direction. Read-only context-reference cycles are therefore permitted when every binding is +declared and bounded; they are not dependency cycles. Temporal, recursive, hierarchical, +or transitive context expansion remains unsupported. + +Every target has exactly one `_ContextScope`. A scope contains one target and an ordered +tuple of zero or more context datum IDs. The order inside the tuple is explicit prompt or +presentation order and is therefore semantic. Context-scope declaration order is not +semantic. A compiler-issued opaque `_ContextScopeId` identifies the compiled scope; member +text, tuple position, source IDs, and graph declaration order do not. + +The first profile permits: + +- one scope for every target datum; +- an empty context tuple; +- one context datum reused by several targets; +- one target used as context for another target; and +- ordered context without inferring chronology, hierarchy, or dependency. + +It rejects: + +- a missing or duplicate scope for a target; +- an unknown target or context datum; +- a context-only datum used as a scope target or placed in an atomic group; +- an unreferenced context-only datum; +- target self-context; +- a duplicate context member in one scope; +- implicit context derived from links, dependencies, coherence, atomic membership, source + order, or equal content; and +- any unsupported relation, nesting, wildcard expansion, or unbounded context rule. + +Atomic groups remain a flat exact partition of target datums only. Dependencies continue to +connect target datums only. Context-only datums are immutable task inputs: they do not own +logical protection tasks, dependencies, atomic-group membership, output candidates, or +datum-release outcomes. + +To preserve exhaustive accounting, the compiled plan expands each target task into a fixed +set of private context-binding records. Every expected binding closes as `available` or +`invalid` before dispatch. These records are child input evidence for the owning task, not +new graph datums or group-scoped tasks. An invalid binding deterministically fails the owning +undispatched task; it is never treated as absent context. + +Compilation creates one immutable binding identity for every `(owning target-task identity, +context-scope ID, ordinal, context datum ID)` tuple before any runtime work ID exists. +Lowering must bind each identity exactly once to a fresh `context_binding_id`. +Reconciliation starts from the compiled binding set, not from work IDs or rows observed +after lowering, and proves the full bijection from compiled binding identity to lowered work +ID to consumed evidence. + +Phase 5 uses three private correlation names: `target_work_id` identifies one lowered target +row, `context_binding_id` identifies one lowered use of context by one target, and the phase 4 +`attempt_id` identifies the dispatch attempt. They are random, invocation-local work IDs, +not credentials, permissions, public graph identifiers, or caller-supplied trace IDs. Text, +source identifiers, DataFrame indexes, row order, and the work IDs themselves never replace +the compiled identities that remain authoritative. + +A context-only datum is one graph value. A context binding is one consumer-specific, +ordinal-bearing compiled use of a datum. Reusing one datum creates distinct bindings; it +does not copy the datum or create a dependency among consumers. + +`available` and `invalid` are private binding-evidence states, not phase 4 task outcomes. A +declaration, limit, or execution-contract defect is rejected during preparation. Failure to +construct a known compiled binding before dispatch yields local task `failed`. Evidence that makes +binding attribution unsafe yields phase 4 `inconsistent` and follows its localization or +global-embargo rules. Lack of a trusted terminal record after dispatch yields `lost`. + +## Pure admission and context execution contract + +Compilation remains the only graph admission boundary. It consumes the proposed graph, +phase 4 plan inputs, context limits, strategy profile, and declared backend capabilities. It +returns one immutable plan or one bounded content-free rejection before phase-5-owned +ledger records, work IDs, workframes, provider access, or backend dispatch. The compiler +remains pure. The public wrapper must expose a versioned content-free preflight observation +through the existing opt-in measurement surface. Observation cannot change the compiler +input, decision, rejection precedence, or prepared value. + +Phase 5 extends the phase 4 rejection order: + +1. Reject a malformed outer graph or unsupported private schema version. +2. Reject global datum, identifier, and byte-limit violations. +3. Reject malformed, duplicate, or purpose-invalid datum declarations. +4. Apply the accepted phase 4 dependency and atomic-partition checks. +5. Reject malformed context scopes, missing or duplicate target coverage, unknown + references, orphan context-only datums, self-context, and duplicate members. +6. Reject per-scope context count or byte excess, total context-reference excess, and total + lowered-frame expansion excess. +7. Reject recognized but unsupported context relation, ordering, nesting, wildcard, or + cardinality semantics. +8. Reject a missing or incompatible context-workframe capability, strategy, or provider + retention posture. + +The same invalid graph must select the same safe code under declaration-order and capability +permutations. A structural error must not be presented as a backend-compatibility decision. + +The first context execution contract is a private, immutable, versioned value bound to the +compiled plan. It declares: + +- the permitted private profile and schema version; +- maximum context members and UTF-8 bytes per target; +- maximum total context references and expanded workframe bytes; +- whether target datums may be reused as context; +- the permitted context direction and ordering grammar; and +- the closed execution-boundary artifact classes and versioned closure attestations required + before release; and +- the provider-retention posture, which must be `retention_disabled` for the first profile. + +The contract is not an access credential and has no product identity, expiry, +renewal, or revocation semantics. The integrating product authorizes source access and field +use before graph construction. Public `prepare()` is the preflight boundary: it validates +the exact declared projection against the frozen contract without model or provider work. +Immediately before invocation open, `protect()` verifies that the selected backend still +provides the required profile, limits, and retention posture. A missing or incompatible +backend raises the typed pre-invocation rejection and never widens the plan, removes context, +or falls back to independent rows. Provider or credential binding already performed by the +legacy public facade remains outside this narrower no-additional-effects guarantee. + +## Compiled projection and separate frames + +The immutable compiled plan contains one projection manifest per target task: + +```text +target task identity + -> target datum identity + -> context scope identity + -> ordered context datum identities + -> compiled byte and count ceilings + -> permitted private consumer profile +``` + +The manifest contains graph identities but is never passed directly to DataDesigner. At +runtime, lowering creates fresh private work IDs and two logical projections: + +| Projection | Required fields | Forbidden fields | +| --- | --- | --- | +| Target frame | `target_work_id`, phase 4 task and `attempt_id`, target text, reviewed target-local fields | source IDs, graph IDs, context text, reconstruction state, public index | +| Context frame | `context_binding_id`, owning `target_work_id`, context ordinal, context text | source IDs, graph IDs, target output columns, unrelated fields, reconstruction state | + +Implementation may encode the context projection as a bounded structured internal column or +as a separate temporary DataFrame. If a backend needs one tabular call, a private adapter may +construct a third ephemeral request payload from the two validated logical frames. That +payload is not the target frame, must preserve both work-ID namespaces and every context +ordinal, and is discarded before hydration and release. Concatenating target and context +into one prose string, using a context row as an output row, or deriving correlation from +position is not conformant. + +All internal column names must use `COL_*` constants. Shared prompt references must use +`_jinja()`. A workflow may consume context only after a later phase defines its exact prompt +and semantic role. The no-context path must reuse the existing prompt and workflow shape +byte-for-byte where current compatibility tests require it. + +## Lowering, reconciliation, and hydration + +A target task becomes dispatchable only after every expected context binding is available, +the projection is within its compiled limits, and all phase 4 readiness rules pass. Lowering +must not fetch source data, consult reconstruction state, infer missing fields, or expand the +scope after compilation. + +The executor may batch ready target tasks. Each batch remains transport-only. Reconciliation +must prove: + +1. compiled target-task identities, lowered `target_work_id` values, and terminal target + records form exact bijections; +2. compiled binding identities, lowered `context_binding_id` values, and consumed input + evidence form exact bijections; +3. each context ordinal, datum, scope, and owner equals its compiled binding identity; and +4. no foreign, stale, duplicated, missing, cross-target, or extra binding influenced the + task. + +A trusted response missing one target work ID may be localized under the accepted phase 4 +rules. A context-binding defect is local only when the complete unaffected target and +context bijections remain provable. An unknown work ID, cross-target binding, plan mismatch, +or attribution contradiction that destroys the batch bijection closes the invocation as +`inconsistent` and triggers the phase 4 global embargo. + +Hydration produces results only for target tasks. It never creates a mention anchored to +context, emits context text, or promotes context-only datums into target outcomes. Phase 5 +returns a private context-framed graph capability for phase 6; it does not change protected +text by itself. + +## Phase 4 integration and release + +Phase 5 does not add an output-producing semantic stage. It extends the input projection and +reconciliation obligations of existing per-target phase 4 tasks. Synthetic ledger tests may +exercise context-binding children, but a backend batch or context row is never a task. + +Known projection construction or binding failure before dispatch closes only the owning +task as `failed`. Cancellation before dispatch closes it as `cancelled` with no backend call. +After dispatch, phase 4 cancellation, trusted-stop, lost-execution, terminal precedence, and +no-retry rules apply unchanged. + +A context-local fault affects every target task whose compiled projection contains that +binding. Phase 4 then propagates target ineligibility through explicit dependencies and +atomic groups. Reuse of the same context datum does not create an implicit dependency among +its consumers. A global attribution fault withholds all groups. + +The phase 4 release barrier remains the only output publication point. Released atomic +groups contain complete target outputs only. Withheld groups, failed projections, context +frames, callbacks, errors, diagnostics, and traces expose no context text and never substitute +raw target input. + +The binding transition and fault mapping is closed: + +| Point | Observation | Binding evidence | Phase 4 effect | +| --- | --- | --- | --- | +| Preparation | malformed, over-limit, or backend-incompatible declaration | none | reject; no invocation | +| Before dispatch | exact compiled binding constructed | `available` | task may become ready | +| Before dispatch | known compiled binding cannot be constructed | `invalid` | owning task `failed`; no dispatch | +| At dispatch | compiled binding-to-work-ID bijection committed | `available` | one task attempt | +| Reconciliation | one exact consumed binding record | `available` | no additional terminal effect | +| Reconciliation | localizable missing or malformed binding evidence | `invalid` | owning task `inconsistent` | +| Reconciliation | foreign, cross-target, contradictory, or plan-mismatch evidence | `invalid` | invocation `inconsistent`; global embargo | +| After dispatch | no trusted task run record | unchanged | owning task or invocation `lost` under phase 4 | + +Cancellation before binding construction wins with no frame or dispatch. A known construction +failure accepted first remains `failed`; a later cancellation request cannot rewrite it. +After dispatch, phase 4 terminal acceptance and cancellation precedence remains authoritative. + +## Cancellation, cleanup, and privacy + +Cancellation is an event, not proof that backend work stopped. Before dispatch it causes no +workframe or backend effect. After dispatch, a target task is `cancelled` only with trusted +stop evidence; otherwise it is `lost`. A late result cannot reopen a terminal task. Phase 5 +adds no automatic retry. + +Before release, an owned bounded lifecycle seam must report that every context binding is +terminal, every Anonymizer-owned context or joined workframe is closed, every work-ID map +rejects further access or mutation, and every backend-owned ephemeral artifact class named +by the frozen contract has one trusted closure attestation. Tests inspect that seam, the +bounded artifact paths owned by the invocation, and the versioned execution-boundary +attestations. Cleanup runs after task and datum terminal evidence is accepted but before +group release and public target-outcome materialization. It never rewrites an absorbing task +or datum outcome. A confirmed Anonymizer cleanup failure or trusted backend closure failure +closes the invocation as `failed` with reason `cleanup_failed`. Missing, foreign, or +contradictory closure evidence closes it as `inconsistent` with reason +`cleanup_unconfirmed`. Both apply a global release embargo; every public target receives the +corresponding non-success outcome and no group exposes output. This proves logical lifecycle +closure and required execution-boundary attestation only; it does not prove physical memory +erasure, provider deletion, or absence from an unowned provider trace. Host teardown after +immutable result acceptance is separate and cannot retroactively change the result. + +Context text, target text, source identity, prompts, graph IDs, context-scope IDs, and +content-derived hashes are forbidden in Anonymizer-owned logs, metrics, exceptions, public +receipts, diagnostics, cleanup errors, and bounded traces. Private active workframes may +contain only the compiled projection. Public compatibility traces retain only +their existing contract and gain no context fields. Provider-side retention and tracing are +integration and provider-governance controls that cross a separately owned privacy boundary. +The first profile requires a backend-attested `retention_disabled` posture; this is not proof +of provider behavior. Provider tracing is not part of this capability and Phase 5 makes no +claim about arbitrary external traces. A later retention-enabled profile would require a +separately reviewed customer or consuming-product privacy contract outside Anonymizer. + +## Observability and profiling + +Phase 5 exposes versioned, content-free observations around preflight, workframe +construction, dispatch, backend execution, reconciliation, cleanup, and release. When the +existing opt-in measurement surface is active, the observation schema records monotonic +duration, bounded or bucketed target/context counts and sizes, selected semantic and +implementation profile versions, route, terminal outcome, reason code, reconciliation +status, cleanup status, and allowlisted numeric or bucketed provider-usage fields when +available. Instrumentation around +DataDesigner work remains outside and immediately around `NddAdapter.run_workflow()`; it +does not create another execution boundary. + +Anonymizer may join a caller's distributed trace as a child operation, but a caller trace ID +is never datum identity, a work ID, or a metric label. Target/context text, prompts, entities, +replacements, source IDs, graph IDs, private work IDs, endpoints, credentials, and unbounded +or content-derived dimensions are forbidden in observations. Instrumentation is +non-authoritative: it cannot change graph semantics, supply missing terminal evidence, or +turn measurement failure into protection success. + +## Phase 6 handoff + +Phase 6 may consume the private framed projection only after it defines a closed use for +context. Any detected mention remains anchored to target datum offsets. Context may inform a +reviewed target decision, but it cannot supply a replacement span, become a target by +inference, or establish alias, dependency, coherence, or release semantics merely by being +visible. + +The handoff supplies: + +- immutable target datum identity and text; +- ordered, declared, bounded context frames; +- exact target/context binding evidence; +- the phase 4 task and attempt identities that account for execution; and +- content-free participation counts and reason codes. + +It does not supply entity mentions, clusters, replacement roles, replacement slots, +synthetic values, source reconstruction state, or durable provenance. + +## Reference model and test oracle + +Build a pure reference model before the production framing runtime. It is independent of +pandas, DataDesigner, the production compiler, and the phase 4 ledger. Its inputs are: + +- target and context-only datums, purposes, context scopes, dependencies, and atomic groups; +- every concrete count/byte ceiling, profile, immutable context execution contract, the + capability snapshot accepted at preflight, and the capability snapshot observed + immediately before invocation open; +- the fixed phase 4 readiness and terminal rules; and +- a timestamp-free sequence of binding construction, binding-work-ID commitment, consumed + binding evidence, dispatch, keyed task terminal, cancellation, trusted stop, transport + loss, cleanup, publication, and post-acceptance teardown observations. + +The model derives admission, projection manifests, binding records, readiness, correlation, +terminal target outcomes, fixed-point withholding, cleanup evidence, invocation outcome, +public target outcomes, and the only legal release set. Production framing, localization, +or release decisions are not model inputs. + +Cleanup evidence is closed: `verified` means every Anonymizer-owned frame, artifact, and +work-ID map reports closed and every required backend artifact has one trusted compatible +closure attestation; `failed` is one definitive Anonymizer cleanup failure or trusted backend +closure failure; and `unconfirmed` is missing, duplicated, foreign, incompatible, or +contradictory closure evidence. Cleanup does not rewrite terminal task or datum evidence. It +derives invocation-level +`failed(cleanup_failed)` or `inconsistent(cleanup_unconfirmed)` and the corresponding public +target outcomes before publication. + +The central oracle is: + +```text +projection valid(target) iff + exactly one admitted scope names the target + and the exact frozen projection satisfies the context execution contract + and the preflight capability snapshot satisfies the required contract + and every ordered context binding is declared, unique, known, and within limits + +invocation opens iff + the immediately-before-open capability snapshot still satisfies the complete frozen + profile, limit, schema, and retention requirements + +task dispatchable(target) iff + projection valid(target) + and every context binding is available + and the phase 4 task-readiness predicate passes + +group released iff + phase 4 accounting is exhaustive and globally consistent + and every member target remains release eligible + and cleanup evidence is verified + and no cancellation or publication embargo applies +``` + +The finite exhaustive envelope contains: + +- one through four target datums and zero through three context-only datums; +- zero through three context members per target, including target-as-context cases; +- the phase 4 DAGs and flat atomic partitions over target datums; +- exact, over-limit, and backend-incompatible scope projections; +- ordered capability pairs whose preflight snapshot is `compatible` and whose + immediately-before-open snapshot is `compatible`, `missing`, `incompatible`, `weakened`, + or `retention_enabled`; only a runtime snapshot that still satisfies the complete frozen + contract may open an invocation; +- a finite ceiling domain `{0, exact, exact + 1}` for datum bytes, ID bytes, context members, + context bytes, total references, and expanded-frame bytes, using symbolic payload classes + `{empty, one-byte, multibyte, exact-limit, one-over-limit}`; +- one dispatch per ready logical target task; +- one primary binding construction/consumption observation and at most one missing, + duplicate, wrong-ordinal, foreign, cross-target, or contradictory observation per binding; +- one primary terminal observation and at most one missing, duplicate, foreign, stale, + cross-target, plan-mismatch, or contradictory observation per task; +- cleanup evidence classes `verified`, definitive `failed`, missing, duplicate, foreign, + incompatible, and contradictory, with at most one primary cleanup observation and one + competing observation per invocation; +- at most one cancellation, trusted-stop, transport-loss, publication, and post-acceptance + teardown event per invocation; and +- context-reference topology classes with no cycle, rejected self-context, every two-target + cycle, every three-target cycle within the datum bound, and each permitted cycle both + disjoint from and overlapping a phase 4 dependency edge; and +- the deterministic trace bound `E_max = 4B + 3T + 7`, where `B` is the admitted binding + count and `T` is the admitted target-task count. The four binding slots cover construction, + work-ID commitment, consumption, and one corruption; the three task slots cover dispatch, + terminal acceptance, and one corruption; the seven invocation slots cover cancellation, + trusted stop, transport loss, one primary cleanup observation, one competing cleanup + observation, publication, and post-acceptance teardown. + +Schedules that differ only by commuting events for independent target tasks share one +canonical representative. Context order within one scope never commutes because it is +semantic. Freeze the model and generator versions, exact graph count, exact canonical trace +count by target/context cardinality, context-order class, and context-cycle class, the full +finite ceiling domain, `T`, `B`, `E_max`, the actual event count, and a SHA-256 manifest digest +before comparing the implementation. The generator rejects any trace that exceeds its +computed bound; it never truncates a legal trace at a fixed number. + +## Admission and framing tests + +Admission tests cover every rejection tier, declaration permutations, adjacent-tier +multiply-invalid inputs, exact-limit and one-over-limit cases, empty context, missing and +duplicate target scopes, orphan context-only datums, unknown and duplicate members, +self-context, unsupported target-as-context use, invalid purpose, unsupported relation, +missing or incompatible backend capability, and unsupported strategy profile. + +Effect spies start before compilation and require zero phase-5-owned invocation, ledger, +work ID, workframe, provider, credential, backend, NDD, reconstruction, and publication +effects on rejection. When opt-in measurement is active, the wrapper records exactly one +versioned preflight start/terminal pair with duration and the closed rejection code; it may +not contain graph content or create invocation identity. Pre-existing public-facade +construction and request-start telemetry remain outside this guarantee. + +Context-contract tests distinguish structural, limit, profile, and backend-compatibility +failure and prove the documented rejection precedence. They enumerate ordered capability +snapshots at preflight and immediately before invocation open. A runtime snapshot that still +satisfies every frozen profile, limit, schema, and retention requirement opens normally; +missing, unknown, incompatible, weakened, or retention-enabled runtime posture raises the +typed pre-invocation rejection with zero invocation, workframe, provider-call, or NDD +effects. Mutating the graph, limits, or prepared value cannot widen the plan from live +backend state. + +Framing tests prove: + +- target and context schemas remain separate under every transport encoding; +- declared context order is preserved exactly; +- equal or repeated text remains distinct by opaque identity; +- target-as-context creates a read-only binding, not a second output; +- reused context creates independent bindings for each consumer; +- no source ID, graph ID, public index, reconstruction state, or unrelated field enters a + workframe; and +- empty-context lowering preserves the existing private/public workflow shape. + +The last assertion uses frozen prompt text, column configuration, provider-call count, and +workflow-schema snapshots for the current no-context `run()` and `preview()` profiles. A +structural digest changes only through an explicitly reviewed compatibility update. + +## Reconciliation, fault, and schedule tests + +Independently permute target rows, context rows, batches, and terminal records. Cover missing, +duplicate, foreign, stale, cross-target, extra, and swapped work IDs; correct work IDs with +wrong ordinals; a valid work ID bound to the wrong task; one localizable missing binding; and a +global attribution contradiction. Duplicate and null-like public DataFrame indices, equal +target/context text, and identical source-shaped fixture fields must not affect attribution. + +Paired oracle fixtures use the same admitted graph and differ by one observation. In the +local fixture, one binding defect is attributable to one owning task and changes only that +target's explicit dependency and atomic-group closure. In the global fixture, a contradictory +binding destroys ownership attribution and embargos every group. The production executor +must match both release sets and reason classes exactly. + +Metamorphic cases reset, duplicate, reorder, and replace DataFrame indexes; inject colliding +source-shaped IDs and row labels; and permute source-field order. Work-ID-keyed outcomes remain +isomorphic, no source field enters either logical frame, and no mutated value can satisfy a +compiled binding. + +Cancellation tests retain the phase 4 linearization point: pre-dispatch cancellation causes +zero frames and dispatches; trusted stop accepted before a result yields `cancelled`; a +post-dispatch request without stop evidence yields `lost`; accepted terminal evidence wins +over later cancellation; late evidence cannot resurrect output. + +Inject projection construction failure, backend failure, trusted partial results, missing +run records, worker death through a test-only backend, cleanup-before-release failure, result +construction failure, and post-acceptance teardown failure. The process-kill test supplies +only phase-5 failure-boundary evidence and does not create a production process API or satisfy +the second-runtime gate. + +Cleanup linearization fixtures first accept all task and datum terminal evidence, then vary +the cleanup evidence before publication. They cover owned frame and map closure plus every +declared backend artifact's successful, failed, missing, duplicate, foreign, incompatible, +and contradictory attestation. `verified` preserves private terminal evidence and permits +normal release qualification. Definitive owned or trusted backend failure produces +invocation `failed(cleanup_failed)` and public `Failed` outcomes without rewriting task or +datum states. Missing, duplicate, foreign, incompatible, or contradictory cleanup evidence +produces invocation `inconsistent(cleanup_unconfirmed)` and public `Inconsistent` outcomes. +Every non-verified case applies a global embargo and exposes no output. + +Compiled-plan tests mutate every reachable source declaration, context tuple, adapter +projection, limit object, and available backend capability after preparation. The lowered +frames and binding identities remain those of the immutable compiled snapshot. Nested tampering through +a test seam fails before workframe construction. NDD request inspection proves that the +compiled manifest itself and graph/source IDs never enter the backend payload. + +Lifecycle tests inspect the bounded private-state seam and ephemeral artifact directory on +success, rejection, preparation failure, cancellation, lost execution, inconsistency, +publication failure, and post-acceptance teardown. Pre-release paths have no live frames or +mutable work-ID maps at publication; teardown faults are reported separately. Confirmed +cleanup failure and unconfirmed cleanup receive distinct closed reasons. These tests make no +secure-erasure claim. + +Observation-contract tests require one versioned start/terminal pair for each entered +lifecycle boundary and verify monotonic durations, closed routes and outcomes, bounded +dimensions, and stable profile versions. Batch, row, datum-ID, work-ID, and caller trace-ID +permutations must not change semantic outcomes or create high-cardinality metric labels. +Fault injection covers unavailable, throwing, delayed, duplicate, and reentrant measurement +sinks. No observer behavior may fabricate terminal evidence, alter event or cleanup +linearization, leak content, change the release set, or create unbounded labels. + +## Required properties and mutation tests + +Required properties are: + +- **Target conservation:** every admitted target has exactly one terminal phase 4 outcome. +- **Binding conservation:** every compiled context reference has exactly one terminal private + binding record for its owning task. +- **Projection exactness:** workframe content equals the compiled manifest and contains no + other datum or field. +- **Frame separation:** target and context cannot exchange roles under row or batch + permutation. +- **Identity invariance:** opaque datum and work-ID renaming yields an isomorphic result. +- **Content non-identity:** equal text never merges datums or satisfies a missing binding. +- **Declaration invariance:** scope-declaration and unrelated datum order do not change + semantics; explicit context order is preserved. +- **Contract monotonicity:** reducing limits or backend capability cannot enlarge the + dispatch or release set. +- **Snapshot isolation:** referenced target success, failure, cancellation, transformation, + withholding, and context-reference cycles cannot change another task's compiled original + context snapshot or readiness absent an explicit phase 4 dependency. +- **Isolation:** a localizable context fault changes only its consumers and their explicit + dependency/group closure; global attribution faults are excluded. +- **Batch invariance:** ready-frontier batch size does not change outcomes. +- **No context visibility:** context never appears in output, withheld results, or public + surfaces. +- **Boundedness:** retained frames, work-ID maps, observations, and diagnostics stay within + declared limits. + +Mutation tests must catch target/context concatenation, positional joins, missing binding +acceptance, fallback to no context, implicit source-order context, text-based deduplication, +target-as-context output, cross-target work-ID adoption, ignored ordinals, limit checks after +lowering, premature output, context leakage, incomplete cleanup, and raw-target fallback. +They also catch live-plan rereads, transformed-output context, target-status-derived context +dependencies, work-ID issuance from observed rows, and acceptance of a weakened or +incompatible backend capability. + +## Compatibility and privacy tests + +Compatibility tests exercise `run()`, `preview()`, `evaluate()`, display, validation, and CLI +paths with unique, duplicate, non-monotonic, string, and null-like indices; duplicate text; +row filtering, concatenation, and reset; mixed success/failure; `trace_dataframe`; and exact +`FailedRecord` shape and ordering. Public calls continue to use empty context and must not +gain prompt sections, columns, diagnostics, or behavior changes. + +Boundary spies prove every DataDesigner execution still passes through +`NddAdapter.run_workflow()`. Source-shaped ATIF, OTLP, and chat fixtures may exercise +projection pressure only after their owners approve field roles; source types and +reconstruction state never enter Anonymizer core. + +Privacy tests use separate high-entropy canaries for target text, every context datum, +source identity, graph identity, private work IDs, prompts, and known digests. The owned-sink +inventory covers target, context, and joined request workframes; work-ID maps; the bounded +ephemeral artifact directory; `repr` and `str`; exceptions and their cause/context; tracebacks; +Anonymizer-owned logs and metrics; public results and compatibility traces; receipts; +diagnostics; serialization; and cleanup errors. A structural allowlist inspects every owned +sink. The expected visibility matrix is: + +| Surface and lifecycle | Permitted canary visibility | +| --- | --- | +| Active target frame | Target canary only in the exact compiled target-text field | +| Active context frame | Context canary only in the exact compiled context-text field | +| Active joined request payload | Only the exact compiled target/context projection in its reviewed fields | +| Work-ID maps and accounting state | Private work-ID canaries only; no target or context content | +| Any owned surface after verified cleanup | No target, context, prompt, entity, replacement, source-ID, graph-ID, or work-ID canary | +| Observations, logs, exceptions, results, receipts, diagnostics, traces, and cleanup errors | No protected canary or known digest at any lifecycle point | + +Substring and known-digest scans supplement these field- and lifecycle-specific assertions; +they do not reject content in the exact active private fields that are required to execute the +compiled projection. Capability tests verify the backend's attested `retention_disabled` +posture; they do not claim to prove provider behavior. An absent, unknown, incompatible, or +retention-enabled posture rejects before invocation open. A future retention-enabled profile +remains unavailable until a separate customer-owned, versioned privacy-boundary contract is +reviewed outside Anonymizer. + +## Ownership and promotion gates + +| Decision | Required authority | +| --- | --- | +| Context grammar, framing, correlation, and failure semantics | Project and Anonymizer semantic owner | +| Context count/byte ceilings and backend capability contract | Anonymizer semantic and execution owners | +| Observation schema, privacy allowlist, and measurement non-interference | Anonymizer semantic and measurement owners | +| Backend artifact classes and trusted closure-attestation schema | Anonymizer semantic and execution owners | +| Source field roles and source-to-context projection | Source-adapter and adopter owners | +| Source access, field authorization, and earliest boundary where unprotected target/context may cross | Customer or consuming-product owner | +| Any public graph, context, receipt, artifact, or endpoint | Public-API and Platform owners | + +The implemented private Phase 5 profile is governed by these prerequisites, which remain +regression and promotion gates: + +1. Phase 4 is implemented and its evidence qualifies; +2. reviewers accept the closed purpose and context-scope grammar; +3. reviewers accept semantic context ordering and target-as-context behavior; +4. reviewers accept the bounded context contract and backend compatibility checks; +5. reviewers accept separate framing, binding reconciliation, and failure localization; +6. reviewers accept the reference model, finite envelope, mutations, privacy canaries, and + observation-contract tests; and +7. reviewers accept the unchanged public, DataFrame, NDD, measurement, and downstream + ownership boundaries. + +Completion of Phase 5 does not authorize context-informed entity decisions, the Phase 7 +branch checkpoint, a public graph/session API, source mappings, production Intake or OpenShell +support, durable state, a privacy boundary, a `zero PII` claim, or stable promotion. + +## Evidence and unresolved gates + +The current branch implements bounded multi-datum framing, private context correlation, +publication-critical cleanup, and the reviewed context execution contract. The frozen Phase 5 +reference model and focused tests cover admission, workframes, execution, reconciliation, +lifecycle, privacy, and public compatibility. They do not authorize source field selection, +prove provider retention behavior, or establish unrestricted context-aware model semantics. + +Intake hierarchy and structured fields motivate bounded context but do not approve which +fields may be exposed, the earliest protection boundary, or a source-to-context mapping. +Those remain adopter and customer decisions. Phase 5 must fail admission without the +required context execution contract and compatible backend, and it must not encode Intake +policy in Anonymizer core. + +The authoritative inputs are the parent +[technical proposal](graph-native-anonymizer-sdk-technical-proposal.md), the authorized +[phase 4 design](phase-4-hierarchical-terminal-accounting-design.md), the separate +[Intake evidence](intake-workload-validation-evidence.md), and the branch-local graph, +runtime, adapter, and private release code at +`88b59e2a4366be09aa7af802fa0a8f81afa8440d`. diff --git a/docs/development/phase-6-anchored-mention-resolution-design.md b/docs/development/phase-6-anchored-mention-resolution-design.md new file mode 100644 index 00000000..e3a5c237 --- /dev/null +++ b/docs/development/phase-6-anchored-mention-resolution-design.md @@ -0,0 +1,679 @@ + + + +# Phase 6 design — anchored mentions, resolution, and local verification + +Status: reviewed phase-specific design and test strategy; private branch-local implementation +and hardening landed on 2026-08-27 in `5bf61c6` through `29bddad`. This document is +subordinate to the complete development and research plan in the +[graph-native SDK RFC](graph-native-anonymizer-sdk-rfc.md), the +[phase 4 terminal-accounting design](phase-4-hierarchical-terminal-accounting-design.md), +and the [phase 5 target/context design](phase-5-target-context-workframe-design.md). RFC +acceptance remains the project decision on the plan. The branch-local implementation does +not authorize a public graph or session API, production Intake or OpenShell integration, or +stable promotion. + +Review status: independent architecture and test-strategy council review completed on +2026-08-20 with zero unresolved Critical or Warning findings. Focused re-review after the +Phase 5 ownership-boundary revision also completed with zero unresolved Critical or Warning +findings. The design review is complete, and the branch now contains the Phase 6 +implementation, frozen independent reference model, and focused mention, resolution, role, +Redact, backend, lifecycle, privacy, and compatibility tests. The 2026-08-31 PR checks pass; +this evidence remains private to the branch and does not establish Substitute, product, or +publication support. + +## Decision + +Phase 6 should convert untrusted detection evidence into immutable mentions anchored only to +authoritative target datum offsets, resolve those mentions into deterministic clusters from +explicit typed same-subject evidence, classify their replacement roles through a versioned +closed policy, and qualify one private Redact profile through exact mention-keyed patch and +atomic-group verification. + +Text equality, label equality, source identity, DataFrame position, context membership, and +model confidence never establish mention or cluster identity. Every mention starts in a +singleton cluster. A merge occurs only when accepted same-subject evidence names exact +mention tokens; the deterministic resolver computes the same clusters from the same evidence +regardless of row, response, or declaration order. + +The first local profile is Redact only. Public Redact, Annotate, Hash, and Substitute retain +their current DataFrame behavior. Adding a private Annotate, Hash, Substitute, or Rewrite +profile requires its own reviewed release predicate and capability. + +## Preconditions and non-goals + +Phase 6 assumes: + +- phase 4 supplies exhaustive task, datum, dependency, group, stage, and invocation + accounting plus cancellation, loss, reconciliation, and release; +- phase 5 supplies immutable target identity and separately framed declared context; +- the graph compiler has accepted the target, dependency, atomic-group, context, and + capability declarations before phase-6-owned effects; and +- every DataDesigner workflow executes through `NddAdapter.run_workflow()`. + +Phase 6 does not: + +- claim detection is exhaustive or output contains no PII; +- anchor a mention to context-only text; +- infer aliases from equal or normalized content; +- create replacement slots, replacement maps, synthetic values, or a planning ledger; +- qualify public or private Substitute, grouped Rewrite, evaluation, or repair; +- move codecs, source identity, field selection, reconstruction, persistence, retries, + deduplication, cleanup, retention, or delivery into Anonymizer; +- select an Intake field policy, source commit unit, or privacy boundary; or +- make the private graph, mentions, clusters, evidence, or roles serializable or public. + +## Phase boundaries and typed graph values + +Phase 6 makes the proposal's detection and resolution phases concrete: + +```text +context-framed ProtectionGraph + -> DetectedGraph # accepted target-anchored mentions + -> ResolvedGraph # deterministic clusters, evidence, and role results + -> TransformedGraph # Redact patches applied once by mention identity + -> VerifiedGraph # datum and atomic-group predicates ready for phase 4 release +``` + +Each graph phase is an immutable, closed, private value. The executor consumes only the +immediately preceding phase and must not reread the original authoring graph or unvalidated +backend payload after hydration. + +The private model contains: + +```text +MentionId +AnchoredMention( + id, + target_datum_id, + start, + end, + exact_source_slice, + detector_label, + provenance_kind, +) + +ClusterId +SameSubjectEvidence(left_mention_id, right_mention_id, kind, version) +DistinctSubjectEvidence(left_mention_id, right_mention_id, kind, version) +EntityCluster(id, ordered_mention_ids, accepted_evidence) + +ReplacementRoleResult = classified(role, policy_version) | unsupported(reason_code) +ResolvedMention(mention, cluster_id, role_result) +``` + +All IDs are compiler- or executor-issued opaque graph-scoped values. They are never source +IDs, legacy entity IDs, model-returned IDs, content hashes, labels, offsets encoded as a +string, or DataFrame values. Exact source slices are private evidence, not identity. + +## Target-anchored mention contract + +Offsets use Python string character indexes and half-open intervals `[start, end)`. A +candidate mention is accepted only when: + +1. its owning target task and datum are known and current; +2. `start` and `end` are integers with `0 <= start < end <= len(target_text)`; +3. `target_text[start:end]` exactly equals the returned source slice; +4. its detector label and closed provenance kind are non-empty and supported; +5. its target-attribution token is current and unambiguous; and +6. it does not overlap another accepted mention in the same target datum. + +Context frames may inform a reviewed candidate, validation, augmentation, or resolution task, +but every accepted mention must still point to an exact target slice. A value found only in +context is not a target mention. A model-returned value without offsets is insufficient for +the private graph profile. + +Exact duplicate evidence for one `(datum, start, end, label, provenance)` tuple may collapse +after candidate finalization when its complete authoritative tuple and lineage are +byte-equivalent. Different final labels, slices, provenance, attribution, or terminal +decisions for one span are contradictory. Partial overlap, strict containment, duplicate +non-identical final evidence, source-slice mismatch, and an offset into context close the +affected target task as `inconsistent` or `failed` according to the phase 4 attribution +rule; no heuristic tie-break is permitted. + +The current public detector may continue its existing value-based augmentation, occurrence +expansion, overlap resolution, and legacy entity IDs. The private graph profile must use a +separate keyed schema with explicit offsets. It must not import `group_entities_by_value()` +or `expand_entity_occurrences()` as graph identity or clustering rules. + +## Detection, validation, and context use + +The first phase 6 profile keeps detector and target boundaries explicit: + +- GLiNER or another span detector receives target text only and returns target offsets. +- Validation may receive the target candidate plus the phase 5 compiled context frame. +- Augmentation may receive target and compiled context, but it must return exact target + offsets and a source slice; value-only suggestions fail graph hydration. +- A resolution task may receive accepted mention tokens, bounded target excerpts, and + separately compiled context. It returns only closed pairwise evidence keyed by mention + tokens. + +Candidate refinement has one closed lineage. Detector and augmenter records are provisional, +target-task-keyed candidate evidence. After exact offset and source-slice validation, the +executor issues a fresh candidate token. A validator returns exactly one of `keep`, +`reclass(label)`, or `drop` for that token. `keep` creates one final candidate with the +provisional label, `reclass` creates one with the accepted validated label, and `drop` +creates none. Missing, duplicate, foreign, stale, or contradictory terminal decisions are +not defaults. Only finalized candidates enter duplicate collapse, overlap validation, and +mention ID issuance. Competing final candidates without one valid lineage are inconsistent; +legitimate reclassification is not contradictory evidence. + +Each effectful step is a compiler-owned semantic stage or task accounted by the phase 4 +ledger. DataDesigner columns may declare the model operations, but a DataDesigner column, +row, or `FailedRecord.step` is not a semantic task. The compiler freezes which steps exist +for the selected profile. + +No-context execution must preserve the public compatibility workflow. The private graph +profile may use stricter schemas and fail-closed behavior without changing the public trace +or legacy detection path. + +## Resolver task cardinality and readiness + +The first profile compiles exactly one phase 4 resolver logical task for each target datum. +Its eligible endpoint set contains the finalized mentions owned by that target plus the +finalized mentions of target datums explicitly named in its compiled phase 5 context +scope. Context-only datums may inform the resolver request, but they never supply a mention +endpoint. Every returned edge must have at least one endpoint owned by the resolver task's +target; its other endpoint must belong to the eligible set. + +The compiler enumerates the resolver's exact input bindings and mention-finalization +predecessors. A resolver becomes ready only after mention-finalization tasks for its owner +and every referenced target datum have terminated successfully. All mention finalization is +therefore a stage before resolution. A read-only phase 5 context cycle creates symmetric +resolver predecessors after that stage and cannot create a scheduling cycle. These are +phase 6 task predecessors, not phase 4 datum dependencies, and do not change release +propagation. + +A failed, cancelled, lost, inconsistent, or blocked mention-finalization predecessor blocks +the resolver. Phase 6 neither dispatches it nor substitutes singleton clusters for requested +but incomplete resolution. After every dispatched resolver task terminates, a deterministic +reducer combines the evidence. Byte-identical duplicate evidence for one ordered endpoint +pair and kind may collapse. Conflicting `same_subject` and `distinct_subject` evidence for +one pair, or any contradiction that cannot be localized to the involved targets, follows the +fail-closed attribution rules below. There is no invocation-global or atomic-group resolver +task; phase 4 accounts for exactly one resolver task per owning target. + +## Deterministic clustering from explicit evidence + +Every mention begins in a singleton cluster. Phase 6 accepts a versioned closed evidence +grammar: + +```text +same_subject — the two mentions refer to one semantic subject +distinct_subject — the two mentions must not be in one cluster +``` + +Evidence in the first profile comes only from a separately accounted invocation-private +resolver task that receives mention tokens after finalization. The evidence origin is a +closed private provenance kind. Source adapters cannot name or observe mention tokens and do +not author alias evidence. A later declaration path would require a separate private +post-detection binding design; content or source-ID joins are never an alternative. + +The resolver: + +1. validates every evidence endpoint against current mention IDs; +2. rejects self-edges, duplicates, foreign or stale tokens, and unsupported evidence + versions; +3. sorts accepted `same_subject` edges by opaque mention declaration position only as a + deterministic implementation order; +4. computes connected components with deterministic union-find; +5. rejects the evidence set if both endpoints of any `distinct_subject` edge belong to the + same component after all `same_subject` unions; a distinct edge whose endpoints remain in + different components is valid separation evidence; and +6. issues a fresh opaque cluster ID for each resulting component. + +Mention declaration position affects only deterministic presentation. Renaming opaque IDs or +permuting evidence and workframe rows yields an isomorphic cluster graph. Equal text and +labels remain separate without an accepted edge. An unresolved or absent edge means separate +clusters, not guessed sameness. + +Phase 6 does not own or validate coherence-scope membership. It emits cluster membership in +mention and datum identities only. Phase 7 validates those clusters against its compiled +flat coherence partition and rejects any cross-scope cluster at its documented admission +tier. Context, dependencies, atomic groups, and phase 6 clusters remain independent. + +## Replacement-role classification + +Detector labels and replacement roles are separate axes. A versioned closed policy maps an +accepted detector label and allowed private profile to either: + +```text +classified(replacement_role, policy_version) +unsupported(reason_code) +``` + +An unsupported role does not invalidate a mention for Redact: Redact needs only the +authoritative span and a local patch. It does block any later phase 7 Substitute task that +requires a type-appropriate slot. Phase 7 may rely only on `classified` roles from the exact +policy version named by its frozen semantic contract. + +For the Redact-only phase 6 profile, the Anonymizer semantic owner freezes only a versioned +label/provenance admission policy and the structural `classified | unsupported` result +grammar. Phase 6 carries that result without inventing a generic role, copying an arbitrary +custom label into the role grammar, or treating `unsupported` as a silent default. + +Phase 7 separately freezes and selects the broader replacement-role and relational-constraint +vocabulary, distinct-slot matrix, host ceilings, and cleanup-observability contract described +in its branch decision record. It validates compatibility with the Phase 6 result version +at admission. Phase 6 does not freeze that phase 7 contract early. + +A cluster represents one semantic subject; a replacement role describes one mention's +replacement function. Phase 6 creates no slot. One cluster may therefore contain mentions +with several classified roles, which phase 7 may later map to distinct type-appropriate +slots. + +## Private Redact transformation profile + +The first qualified transform is deterministic local Redact. After mention finalization, a +pure phase refinement creates an immutable patch manifest with exactly one expected entry per +accepted mention and no invocation token or replacement payload. After the phase 4 invocation +opens, the executor binds every manifest entry exactly once to a fresh mention-private patch +token and materializes the closed Redact operation. Rejected compilation or failed mention +refinement creates neither patch tokens nor patch workframes. + +Each runtime patch contains: + +- the owning target-task token; +- the exact authoritative `[start, end)` interval; +- a closed Redact replacement value or operation selected by the reviewed profile; and +- no source, graph, cluster, or public identifier. + +The transform validates the complete patch set before application. It requires one patch per +mention and no missing, duplicate, foreign, stale, cross-target, or extra patch. Patches are +applied once in ascending source offsets by copying untouched source intervals and inserting +the closed replacement. Application never searches the evolving output and never falls back +to `(value, label)` or value-only lookup. + +The public offset replacement primitive may be reused only below a private adapter that +disables value fallback and proves exact mention-token coverage. Its current legacy +value/label map and unambiguous value fallback remain public compatibility behavior, not the +graph contract. + +Private Annotate is not qualified because its intended output retains the source value and +needs a different release claim. Private Hash is not qualified because content-derived +output, normalization, collision, and leakage policy require separate review. Substitute +remains phase 7. + +## Datum and atomic-group verification + +A target datum is locally qualified for the private Redact profile only when: + +1. its detection and resolution tasks closed successfully; +2. every accepted mention has one valid authoritative span and exactly one patch; +3. the patch set is non-overlapping and applied exactly once without skips; +4. the returned text exactly equals the output reconstructed from authoritative untouched + source intervals and the accepted patch set; +5. no replacement payload contains its protected source slice; and +6. the datum-level strategy predicate passes. + +A target with zero accepted final mentions requires an empty patch manifest, zero patch +tokens and applications, and output exactly equal to its input. It still closes through the +normal verified no-work datum and atomic-group predicates. Unchanged input is never used as +a fallback for a failed mention-bearing target. + +Exact reconstruction is the primary proof. A global substring search is not: the same +source text may legitimately occur at an unprotected location. Tests may retain substring +checks as a conservative secondary canary, but they must not replace mention conservation. + +The group predicate verifies that every member target has one exact locally qualified +result, every expected mention and patch belongs to exactly one member, and no context-only +datum, partial result, token, or raw fallback enters the output. Phase 4 then applies +dependency and atomic-group fixed-point withholding and remains the only release authority. + +A localizable target verification failure withholds its atomic group and explicit dependent +closure. Missing or contradictory evidence that destroys global attribution closes the +invocation as `inconsistent` and withholds all groups. A successful cluster or patch never +overrides a phase 4 cancellation, loss, or embargo. + +## Failure, cancellation, retry, and cleanup + +Phase 6 uses phase 4 terminal outcomes and precedence. A known detector, resolver, +classification, transform, or predicate error maps to `failed`. Trusted missing keyed output +may be localized as `inconsistent(missing)`. Foreign, duplicate, swapped, stale-at-admission, +or contradictory evidence that prevents attribution is `inconsistent`. Dispatch without a +trusted terminal or stop record is `lost`. + +The phase 6 task and fault mapping is closed: + +| Semantic task or evidence | Local terminal result | Release effect | +| --- | --- | --- | +| detector/augmenter attempt returns one attributable known failure | task `failed` | owning target and phase 4 closure withheld | +| validator raises or returns one attributable known failure | task `failed` | owning target and phase 4 closure withheld | +| validator returns `drop` for one current candidate | task success; no final candidate | no failure by itself | +| validator returns one current `keep` or `reclass` | task success; one final candidate | continue mention refinement | +| validator omits, duplicates, or contradicts one expected decision | task or invocation `inconsistent` | localize when ownership remains provable; otherwise global embargo | +| finalization receives contradictory candidate lineage | task or invocation `inconsistent` | localize when ownership remains provable; otherwise global embargo | +| finalization finds one attributable invalid span, overlap, or source-slice mismatch | task `failed` | owning target and phase 4 closure withheld | +| resolver returns one valid complete evidence set | task success | publish immutable clusters privately | +| known role-policy or transform failure | task `failed` | owning target and phase 4 closure withheld | +| trusted batch omits one expected current token with all other bijections proven | task `inconsistent(missing)` | localize to owning target | +| foreign, duplicate, swapped, plan-mismatch, or contradictory attribution | invocation `inconsistent` | global release embargo | +| dispatched attempt has no trusted terminal or stop evidence | task or invocation `lost` | affected output withheld | +| datum or group predicate rejects exact accounted evidence | datum `failed(release_predicate_failed)` | group withheld | + +Every semantic stage declares its fixed predecessor tasks in the compiled phase 4 plan: +candidate generation precedes validation/finalization, finalization precedes resolution and +role classification, and mention refinement precedes patch transformation and verification. +Missing or blocked predecessors cause the accepted phase 4 `blocked` outcome without +dispatch. Competing terminal evidence follows phase 4 precedence and never rewrites an +accepted terminal record. + +Cancellation before dispatch causes no provider or transform call. After dispatch, +cancellation requires trusted stop evidence; otherwise the task is `lost`. Accepted terminal +evidence wins over later cancellation, and accepted cancellation makes later completion +stale. Phase 6 performs no automatic retry. + +Before phase 4 release, cleanup closes all mention, evidence, cluster, role, patch, excerpt, +and token stores to mutation and verifies that no partial graph phase is observable. A +publication-critical cleanup failure closes the invocation as `inconsistent` and withholds +all output. Post-acceptance host teardown cannot rewrite an already accepted result. Python +reference release is not secure erasure. + +## Privacy and diagnostic boundary + +Mention source slices, target/context text, excerpts, prompts, resolver evidence, labels, +roles, clusters, patches, graph IDs, and content-derived hashes are private. Active bounded +workframes may contain only the projection compiled for that task. Withheld and nonterminal +results expose none of them. + +Logs, metrics, exceptions, tracebacks, receipts, diagnostic views, cleanup errors, and +serialized artifacts may expose only allowlisted content-free reason codes and bounded +counts. Public `trace_dataframe` and `FailedRecord` retain their current compatibility shape; +phase 6 must not inject mention, cluster, context, or correlation identity into them. + +The private rejection grammar is closed and versioned. Its classes are `unknown_target`, +`invalid_offset`, `source_slice_mismatch`, `unsupported_provenance`, `missing_decision`, +`duplicate_decision`, `overlap`, `foreign_token`, `stale_token`, `contradictory_candidate`, +`invalid_evidence`, `evidence_contradiction`, `unsupported_role`, `invalid_patch`, and +`release_predicate_failed`. More specific subcodes may be added only within a reviewed +version. Multiply-invalid candidate evidence follows target attribution, token integrity, +offset bounds, source-slice equality, lineage, duplicate, overlap, evidence, role, then patch +precedence. Reason codes and counts contain no content, label, offset, or identifier. + +## Phase 7 handoff contract + +For phase 7 admission, phase 6 supplies an immutable complete input containing: + +- target-anchored mentions with exact source offsets and closed provenance; +- compiler-issued mention and cluster IDs; +- deterministic cluster membership and accepted evidence version; +- one replacement-role result per mention; +- terminal phase 4 task and target outcomes; and +- content-free bounded reason codes for non-success. + +Phase 7 compiles its own flat coherence partition, validates every phase 6 cluster against +that partition, and rejects a cross-scope cluster. It must also reject an unsupported role, +incomplete mention set, unaccepted evidence version, nonterminal predecessor, or missing +group-verification input. It must not repair phase 6 data, regroup by text, or fall back to +row-local planning. + +Phase 7 still owns replacement slots, its scope-planning task and ledger, bundle validation, +collision and relational policy, assignments, and Substitute transformation. Phase 6 does +not precompute or expose those values. + +## Pure reference model + +Build a pure model independent of pandas, DataDesigner, the production resolver, the patch +implementation, and the phase 4 ledger. Its inputs are: + +- target texts, compiled phase 5 context frames, dependencies, and atomic groups; +- closed detection, evidence, role-policy, Redact, capability, and limit versions; +- target-keyed detector, validator, augmenter, and resolver observations; +- mention-keyed patch and transformed-output observations; and +- timestamp-free dispatch, `FailedRecord`, exception, cancellation, trusted-stop, loss, + finalization, and teardown observations. + +The model derives accepted mentions, clusters, role results, readiness, reconciliation, +patches, exact transformed output, datum qualification, group verification, terminal +outcomes, and the only legal release set. Production overlap selection, clustering, +classification, patch application, verification, or release decisions are never model +inputs. + +The central oracle is: + +```text +accepted mention iff + one current finalized candidate survived exactly one keep or reclass decision + and its complete lineage is valid + and it has an exact in-range non-overlapping target source span + +same cluster iff + mentions are connected by accepted same-subject evidence + and no accepted distinct-subject evidence contradicts the component + +qualified target iff + every required phase-6 task succeeded + and every accepted mention has one exact applied patch + and returned output equals authoritative patch reconstruction + +released group iff + every member target remains phase-4 release eligible + and the phase-6 group predicate passes + and no invocation embargo applies +``` + +## Finite conformance envelope + +The exhaustive envelope contains: + +- one through four target datums with zero through three compiled context frames each; +- a finite symbolic text domain with alphabet classes `{ASCII, multibyte BMP, astral, + combining, whitespace}` and length classes `{0, 1, exact span, exact byte limit, + one-over-limit}`; concrete fixtures freeze one representative per class and bound target, + context, excerpt, label, and source-slice bytes; +- zero through six candidate mentions, with zero through four accepted non-overlapping + mentions per target; +- zero through six same-subject or distinct-subject evidence edges; +- every evidence graph over up to four accepted mentions, including contradictory + components; +- zero through three entity clusters and one role result per accepted mention; +- the phase 4 DAGs and flat atomic partitions over target datums; +- one dispatch per required detector, validation, augmentation, or resolver task; +- exactly one resolver task per target, with the compiled eligible endpoint set and every + required mention-finalization predecessor represented; +- one patch and transform observation per accepted mention/target; +- one primary terminal observation and at most one missing, duplicate, foreign, stale, + swapped, plan-mismatch, or contradictory observation per attempt; and +- a computed event bound that includes every required task dispatch/terminal pair, candidate + decision, evidence result, patch, transformation, group verification, cancellation/loss, + finalization, and teardown observation for the admitted graph. + +Canonicalization orders commuting events for independent targets and evidence edges while +retaining every race around dispatch, terminal acceptance, cancellation, transformation, +group verification, finalization, and release. Freeze the model/generator versions, exact +symbolic domain cardinalities, computed maximum event count, exact graph count, exact +canonical trace count, and SHA-256 manifest digest before executor comparison. The checked +generator publishes its machine-readable event alphabet and independence relation; tests +prove commuting schedules collapse while dispatch/terminal, cancellation/terminal, +verification/release, finalization/release, and teardown/acceptance races remain distinct. +Larger seeded state-machine tests supplement the finite envelope. + +## Mention, resolution, and role tests + +Mention tests cover exact boundaries; empty and whole-string spans; adjacent spans; repeated +equal text; equal labels at different offsets; astral Unicode, combining characters, emoji, +newlines, and mixed scripts under Python character indexing; negative, reversed, zero-width, +and out-of-range offsets; source-slice mismatch; context-only offsets; exact duplicate and +non-identical duplicate evidence; containment and partial overlap; and maximum count/byte +limits. + +Independently permute candidate records and workframe rows. The same accepted evidence must +produce isomorphic mentions. A model-returned ID, public record ID, text, label, DataFrame +index, and response position must never satisfy mention correlation. + +Resolution tests enumerate singleton, chain, star, diamond, and disconnected evidence +graphs; duplicate and reversed edges; unknown, stale, foreign, and cross-scope endpoints; +self-edges; a direct distinct edge between singleton components; a distinct edge whose +endpoints become transitively same; a valid distinct edge across separate final components; +equal text in distinct clusters; different text in one explicitly evidenced cluster; and +evidence-order permutations. Cluster IDs may change under opaque renaming, but membership +must be isomorphic. Cross-scope validation is exercised in phase 7 admission tests, not as a +phase 6 resolver rule. + +Resolver-readiness tests cover empty and non-empty eligible endpoint sets; one target using +another target as context; permitted two-way and three-way context cycles; a context cycle +overlapping a phase 4 dependency edge; every mention-finalization predecessor outcome; an +edge with neither endpoint owned by the resolver target; a context-only endpoint; and +duplicate resolver evidence across owners. They prove that all finalization precedes +resolution, context cycles cannot deadlock, non-success predecessors block without singleton +fallback, and one local resolver failure changes only its attributable phase 4 closure. + +Role tests cover every frozen label mapping, every role, unsupported custom labels, policy +version mismatch, missing mapping, declaration permutations, and the difference between +Redact eligibility and phase 7 Substitute readiness. No unknown label becomes a generic role. + +The role-policy manifest is a required versioned input artifact with a content-free version +and digest plus positive and negative fixtures for every admitted mapping. Before that +artifact is frozen, tests exercise only the structural `classified | unsupported` grammar, +unknown-version rejection, and fail-closed absence behavior; they do not invent the eventual +phase 7 vocabulary. + +## Transformation and verification tests + +Generate valid and invalid patch sets across zero, one, adjacent, repeated-text, and maximum +mention cases. Cover missing, duplicate, extra, foreign, stale, swapped, cross-target, +overlapping, out-of-range, wrong-source, wrong-replacement, and wrong-order patch evidence. +The implementation must reconstruct expected output from the original target and accepted +patches; response order and evolving-output search are irrelevant. + +Tests prove: + +- every accepted mention is applied exactly once; +- adjacent and repeated equal source slices remain distinct; +- replacements do not cascade into later source matching; +- value/label and value-only fallback are impossible in the graph profile; +- a source value repeated at an unprotected location does not cause false success or become + an implicit patch; +- empty-mention targets retain unchanged text only through a verified no-work path; +- a datum failure withholds the exact atomic/dependency closure; and +- no withheld group exposes protected or raw fallback text. + +Group tests cover one scope across several atomic groups, several clusters in one group, +independent disconnected groups, a target used as another target's context, fixed-point +dependency propagation, one localizable patch failure, and one global attribution fault. + +## Schedule, fault, and lifecycle tests + +Use deterministic barriers around candidate receipt, mention acceptance, context-informed +validation, evidence receipt, cluster publication, role classification, patch construction, +transformation, group verification, release, and cleanup. + +Required races include cancellation on both sides of every dispatch and terminal acceptance; +late candidates or evidence after cancellation/loss; duplicate resolver completion; patch +application racing with a contradictory record; one target failure while an independent +group succeeds; invocation cancellation after verification but before release; +publication-critical cleanup failure; and teardown failure after immutable result acceptance. + +At least one process-kill test uses a test-only crashable backend at the existing execution +seam. It compares worker death with the pure model's `lost` outcome and empty affected release +set. It does not add a production process API or satisfy the materially different runtime +gate. + +## Required properties and mutation tests + +Required properties are: + +- **Mention conservation:** every canonical finalized candidate equivalence class yields + exactly one mention. Every dropped or rejected candidate lineage has one bounded reason; + byte-equivalent duplicate records in one class do not create extra mentions. +- **Anchor integrity:** every mention slice equals its authoritative target substring. +- **Non-overlap:** accepted mention intervals in one target are disjoint. +- **Cluster partition:** every accepted mention belongs to exactly one cluster. +- **Evidence determinism:** the same accepted evidence yields isomorphic clusters under + declaration and response permutation. +- **Content non-identity:** equal or normalized text and labels do not merge clusters. +- **Role completeness:** every mention has exactly one classified or unsupported result. +- **Patch conservation:** every qualified mention has exactly one applied patch and no other + patch exists. +- **Exact reconstruction:** released target text equals the pure source-plus-patches result. +- **Non-cascading application:** inserted text is never reconsidered as a source span. +- **Identity invariance:** opaque ID renaming yields an isomorphic result. +- **Monotone withholding:** replacing accepted evidence with non-success cannot enlarge the + release set. +- **Independent isolation:** a localizable fault cannot change an unrelated component; + global attribution faults are excluded. +- **Attempt isolation:** stale or foreign evidence cannot satisfy a current task. +- **No partial visibility:** nonterminal, failed, or withheld phases expose no private values. +- **Boundedness and confidentiality:** all retained state and diagnostics respect ceilings + and allowlists. + +Mutation tests must catch value-only mention admission, skipped slice validation, heuristic +overlap selection, text/label clustering, transitive distinct-edge contradiction ignored, +cluster IDs derived from content, unknown role defaulting, positional patch joins, value +fallback, evolving-output replacement, skipped or duplicate patch acceptance, row-local +release before group verification, raw fallback, late-result resurrection, and incomplete +cleanup. + +## Compatibility, boundary, and privacy tests + +Compatibility tests exercise public `run()`, `preview()`, `evaluate()`, display, validation, +and CLI behavior; duplicate and non-monotonic indices; duplicate text; filtered, reordered, +concatenated, and reset frames; result columns and attributes; `trace_dataframe`; exact +`FailedRecord` ID, step, order, and shape; existing public detection expansion and overlap +behavior; local Redact/Annotate/Hash behavior; legacy Substitute fallback; and no-entity +provider bypass. + +Paired tests show that the unchanged public path may retain a legacy value-only or +value-expanded behavior while the private graph profile rejects the same unverifiable input. +Spies prove all DataDesigner execution passes through `NddAdapter.run_workflow()` and no +source-format type enters Anonymizer core. + +Privacy tests inject separate high-entropy canaries for target/context text, mention slices, +labels, evidence, roles, graph/source IDs, tokens, prompts, patches, protected output, and +known digests. Structural allowlists inspect private active state, withheld results, public +results, traces, logs, metrics, exceptions and cause/context, tracebacks, receipts, +serialization, and cleanup errors. Substring and digest scans supplement the structural +inspection. + +## Ownership and promotion gates + +| Decision | Required authority | +| --- | --- | +| Mention grammar, evidence algebra, clustering, roles, and Redact predicates | Project and Anonymizer semantic owner | +| Context semantics, model/provider capability, limits, and cleanup observability | Anonymizer semantic and execution owners | +| Source projection, field policy, reconstruction, retry, and destination checks | Source-adapter and adopter owners | +| Source access, field authorization, privacy boundary, and residual risk | Customer or consuming-product owner | +| Any public mention, cluster, role, graph, receipt, or endpoint | Public-API and Platform owners | + +The implemented private Phase 6 profile is governed by these prerequisites, which remain +regression and promotion gates: + +1. Phases 4 and 5 are implemented and their evidence qualifies; +2. reviewers accept the exact target-anchor and no-heuristic-overlap rules; +3. reviewers accept singleton-by-default clustering and the closed evidence algebra; +4. the Anonymizer semantic owner freezes the Redact-only phase-6 label/provenance admission + policy and structural role-result version needed by the selected private profile; +5. reviewers accept the Redact-only patch and group predicates; +6. reviewers accept context use, limits, provider capability, and cleanup behavior; +7. reviewers accept the reference model, exhaustive envelope, mutations, and privacy + canaries; and +8. reviewers accept unchanged public DataFrame behavior, the NDD boundary, and downstream + ownership. + +Completion of Phase 6 does not authorize the Phase 7 branch checkpoint automatically, Substitute or +Rewrite graph execution, public graph/session/mention APIs, production Intake or OpenShell +support, durable state, an accepted privacy boundary, a `zero PII` claim, or stable promotion. + +## Evidence and unresolved gates + +Current branch code provides graph mention IDs, exact keyed augmentation spans, typed alias +evidence, deterministic evidence-based clusters, closed role results, mention-keyed Redact +patches, exact reconstruction, and group verification. The frozen Phase 6 reference model and +focused tests cover admission, resolution, role policy, backend evidence, lifecycle, privacy, +and public compatibility. They do not qualify graph Substitute or Rewrite. + +The branch freezes the Redact-only label/provenance admission policy and structural +`phase6-role-result/v1` contract. Its intentionally empty Redact role mapping remains +fail-closed rather than inventing Phase 7 roles. Phase 7 separately owns the broader role and +relational contract. Intake field roles, context exposure, source +mappings, atomic commit units, and privacy objectives remain adopter or customer decisions +and do not move into either policy. + +The authoritative inputs are the parent +[technical proposal](graph-native-anonymizer-sdk-technical-proposal.md), the reviewed +[phase 4 design](phase-4-hierarchical-terminal-accounting-design.md), the reviewed +[phase 5 design](phase-5-target-context-workframe-design.md), the reviewed future +[phase 7 design](phase-7-stable-substitute-design.md), the separate +[Intake evidence](intake-workload-validation-evidence.md), and the branch-local detection, +replacement, graph runtime, adapter, and private release code at +`88b59e2a4366be09aa7af802fa0a8f81afa8440d`. diff --git a/docs/development/phase-7-stable-substitute-design.md b/docs/development/phase-7-stable-substitute-design.md new file mode 100644 index 00000000..79c81e74 --- /dev/null +++ b/docs/development/phase-7-stable-substitute-design.md @@ -0,0 +1,732 @@ + + + +# Phase 7 design — stable Substitute planning + +Status: reviewed phase-specific design and test strategy; branch implementation authorization +is pending and cannot be requested before its prerequisites pass. This +document is subordinate to the complete development and research plan in the +[graph-native SDK RFC](graph-native-anonymizer-sdk-rfc.md). The checkpoint is not a separate +project acceptance decision and does not authorize a public graph or session API, production +Intake or OpenShell integration, durable replacement state, or stable promotion. + +Review status: independent architecture and test-strategy council review completed on +2026-08-20 with zero unresolved Critical or Warning findings. The decision record below +preserves that reviewed design; it does not authorize implementation. The phase is not +implemented or tested. Work remains sequenced after the +Phase 4–6 gates, and its owned pre-implementation contract must be frozen before Phase 7 +implementation begins and receives a separate operator checkpoint. + +## Decision + +Phase 7 should plan Substitute assignments once per declared coherence scope and store the +complete assignment bundle in a bounded invocation-private ledger. Every mention resolves +through an immutable replacement slot; no row generates or owns an independent replacement +map. + +The first qualified profile is invocation-bounded. It guarantees consistent +assignments only among datums admitted to one coherence scope in one invocation. It does +not guarantee the same replacement across invocations, processes, workers, restarts, or +delivery attempts. A longer-lived private session or durable state backend requires a +separate capability and state-authority review. + +Stable Substitute is a semantic guarantee about assignment reuse inside the admitted +scope. It is not a claim of durable idempotency, transactional persistence, exhaustive +detection, or absence of all PII. + +## Branch decision record + +The first phase 7 profile is invocation-bounded, and it is the only stability profile in +phase 7. Session-bounded consistency, durable consistency, cross-worker consistency, and a +governed state backend are deferred to a separately authorized design. They are not phase 7 +implementation options. + +The graph profile performs no automatic candidate regeneration. An invalid, failed, +cancelled, lost, inconsistent, partial, or unverifiable planning attempt fails closed under +the outcome and release rules below. Adding bounded regeneration would require a new review +of attempt identity, limits, cancellation, and stale-result behavior. + +Before phase 7 implementation begins, the Anonymizer semantic and execution owners must +jointly publish and freeze one versioned implementation contract containing: + +- the closed replacement-role and relational-constraint vocabulary, owned by the + Anonymizer semantic owner; +- the matrix of slot pairs that must use distinct synthetic values, owned by the + Anonymizer semantic owner; +- accepted count, byte, concurrency, and lifetime ceilings, owned by the execution owner; + and +- the cleanup-observability contract for every terminal path, owned by the execution owner. + +The contract is an input to compilation, candidate validation, the independent reference +model, and the frozen conformance manifest. Unknown versions, roles, constraints, or missing +host limits fail admission. This deliverable may be prepared while earlier phases proceed, +but it cannot be frozen against an implementation until phases 4–6 establish the accounting, +workframe, mention, cluster, and slot inputs it depends on. + +## Contract and current behavior to preserve + +Current public Substitute behavior is DataFrame-oriented. `LlmReplaceWorkflow` generates a +replacement map per row through `NddAdapter.run_workflow()`, filters unrequested entries, +and repairs a synthetic value that equals another protected original in the same row. +`ReplacementWorkflow` then applies the map with the existing offset replacement primitive. +The public trace compatibility surface retains replacement maps while the result DataFrame +does not expose them. + +Phase 7 must not recast this behavior as graph-wide coherence. It adds a private graph +profile while preserving public constructors, configuration, `run()`, `preview()`, +`evaluate()`, result columns and attributes, `trace_dataframe`, `failed_records`, CLI +behavior, and errors. DataFrames remain temporary workframes, datum identity remains graph +identity, and `NddAdapter.run_workflow()` remains the sole DataDesigner execution boundary. + +Until a public change is separately reviewed, the compatibility facade retains row-local +Substitute semantics on the legacy path, including its current same-row collision repair. +Phase 7 does not lower public Substitute through the stricter graph profile. A future +compatibility migration must define and review a distinct semantic profile before changing +that boundary. Cluster, slot, scope, reservation, and planning identities never appear in +public DataFrames or artifacts. + +Phase 7 qualifies Substitute only. Current rewrite mode may use a replacement map in its +prompt, but graph-wide grouped rewrite, evaluation, and repair remain phase 8 work. + +## Preconditions and non-goals + +Phase 7 assumes the earlier gates have supplied: + +- phase 4 exhaustive task, datum, dependency, atomic-group, and invocation accounting; +- phase 5 separately framed target and bounded context workframes; and +- phase 6 datum-anchored mentions, deterministic entity clusters, replacement-role + classification, and group verification for local strategies. + +This phase does not: + +- infer entity clusters from repeated text, labels, DataFrame position, or source IDs; +- infer coherence scopes from context, dependencies, atomic groups, or source hierarchy; +- make the graph or ledger public or serializable; +- provide durable, restart, or cross-worker replacement consistency; +- move codecs, source reconstruction, persistence, retries, deduplication, retention, + cleanup, or delivery into Anonymizer; +- retry replacement planning automatically; +- qualify grouped rewrite or independent-row fallback; or +- claim secure memory erasure when Python releases ledger references. + +## Typed semantic model + +Phase 7 keeps four identities distinct: + +| Identity | Meaning | Must not be derived from | +| --- | --- | --- | +| Datum ID | Immutable graph datum | DataFrame index, row order, text, or source ID | +| Entity cluster ID | Mentions that refer to one semantic subject | Raw entity text alone | +| Replacement slot ID | One type-appropriate assignment reused by its mentions | Original or synthetic content, or a content hash | +| Coherence scope ID | The boundary inside which slot assignments are stable | Context, dependency, or atomic membership | + +One entity cluster may own multiple replacement slots. For example, one person cluster can +have name, email, and phone slots whose values must be relationally consistent. Mentions +that must share one literal synthetic value reference the same slot. Mentions that refer to +the same subject but require different types reference different slots in the same cluster. + +Slot and scope IDs are compiler-issued opaque values. Source identity remains in the +adapter's reconstruction state. Runtime workframes use invocation-private correlation +tokens and must not contain source IDs or public graph IDs. + +The phase 4 terminal-accounting ledger and the phase 7 replacement-planning ledger are +separate state machines. The first proves exhaustive task, datum, dependency, group, and +invocation closure. The second retains only provisional slot assignments for the bounded +invocation. Phase 4 remains the only authority for releasing output. + +The owned internal grammar should use immutable closed variants. Expected scope-planning +outcomes are: + +```text +planned(bundle_ref) +blocked(reason_code) +failed(reason_code) +cancelled +lost +inconsistent(reason_code) +``` + +Only `planned` contains a private reference to a complete immutable assignment bundle. +`blocked` records that an admitted prerequisite prevented planning from running. All other +terminal outcomes contain no replacement values. Pending, reserved, and running are +internal states and cannot appear in a terminal invocation result. Outcome precedence and +fixed-point propagation follow the accepted phase 4 accounting model. + +## Coherence-scope admission policy + +For the first qualified profile, coherence scopes form a flat exact partition of target +datums: + +- every target datum belongs to exactly one non-empty scope; +- every member resolves to one declared target datum; +- duplicate members and duplicate scopes are invalid; +- coverage gaps and implicit singleton completion are invalid; +- nesting and partial overlap are recognized but unsupported; and +- every entity cluster and replacement slot belongs to exactly one scope. + +Reject a cluster that crosses scopes. Reject a mention whose datum, cluster, and slot do not +agree on the same scope. Scope membership order has no semantic meaning; declaration order +is only a deterministic presentation tie-break. + +The compiler validates scope shape, cluster and slot references, declared bounds, strategy, +and required runtime capabilities before any additional graph-invocation effect: ledger or +workframe creation, graph telemetry, candidate dispatch, transformation, or publication. On +the current compatibility facade, providers and other host resources may already be bound +before the internally generated graph is compiled; phase 7 does not claim otherwise. +Unsupported semantics never degrade to row-local planning. + +Phase 7 extends the phase 4 deterministic rejection order rather than creating a second +validator order: + +1. malformed outer graph, type, or version; +2. total count and byte limits, including phase 7 scope, cluster, slot, mention, and bundle + limits; +3. malformed or duplicate phase 4 datum, dependency, and atomic-group declarations; +4. unknown or wrong-purpose phase 4 references, invalid atomic partition, and dependency + cycle; +5. empty or duplicate coherence declarations, duplicate members, coverage gaps, and partial + overlap; +6. unknown, duplicate, mismatched, or cross-scope mention, cluster, and slot declarations; +7. recognized but unsupported coherence nesting, cardinality, role, or constraint grammar; +8. unsupported Substitute profile or missing runtime capability. + +Compiler-issued scope, cluster, and slot IDs cannot collide by authored input. Tests instead +use duplicate semantic declarations or force a compiler invariant failure through a test +seam. Multiply-invalid tests cover adjacent tiers, declaration permutations, and exact-limit +versus one-over-limit cases. + +Context scope, coherence scope, dependency, and atomic group remain independent. A datum +can use related context without sharing replacements, share replacements without depending +on another datum, or share a coherence scope across several atomic groups. + +Two scopes have no assignment-sharing or cross-scope uniqueness guarantee. Equal original +text does not join them, and an accidentally equal synthetic value does not create a shared +slot. Any broader anti-linkability or global uniqueness policy would expand the privacy and +state boundary and requires separate authorization. + +## Replacement-planning bundle + +The compiler derives one closed slot manifest per coherence scope. Each manifest contains +only typed identities, role constraints, and the mention bindings needed to verify coverage. +The planner receives only an allowlisted bounded projection: original values for admitted + slot-bound mentions, closed slot roles and relational constraints, and target or context + fragments contained in the exact compiled context projection. +Coherence membership alone grants no context access. Workframes exclude source IDs, +unrelated fields, datums outside the compiled target/context projection, and reconstruction +state. Public receipts and diagnostics see only counts and allowlisted reason codes. + +Candidate workframes correlate expected slots with invocation-private opaque tokens. A +returned `original` value, label, row position, or DataDesigner record ID is content or +backend evidence, not slot identity. Missing, duplicate, foreign, or contradictory slot +tokens make the scope inconsistent. + +A candidate bundle is valid only when: + +1. every expected slot appears exactly once; +2. no unknown, missing, or duplicate slot appears; +3. every synthetic value is non-empty and differs from every original bound to that slot; +4. no synthetic value equals any protected original in the coherence scope; +5. slots declared distinct do not share a synthetic value; +6. mentions declared to share a slot have exactly one assignment; +7. type, format, wildcard, and closed relational constraints pass; and +8. the bundle remains within declared count and byte limits. + +The graph profile fails closed on an invalid candidate bundle. It does not use the current +row-local placeholder repair as graph semantics, regenerate candidates automatically, or +accept a partial bundle. Any future bounded regeneration policy would need explicit attempt +identity, limits, cancellation, stale-result handling, and separate review. + +A scope with no admitted Substitute mentions completes as `planned` with an empty bundle and +does not call a provider or DataDesigner. A qualified no-work datum may legitimately retain +its unchanged text. That is distinct from substituting raw input for a failed or withheld +result. + +## Phase 4 accounting integration + +The compiler creates one scope-planning stage task for every admitted coherence scope. This +is the phase 7 review of the grouped-task cardinality deferred by phase 4: one planning task +may read bounded inputs from several datums but produces one private scope plan, not a datum +output. It remains a compiler-owned Anonymizer task rather than a DataDesigner scheduler +task. An empty manifest succeeds as verified no-work with zero dispatched attempts. + +Every scope-planning task and dispatched attempt appears in the phase 4 one-shot terminal +ledger. It receives a task ID, attempt ID, and invocation-private correlation tokens before +dispatch. Candidate generation executes through `NddAdapter.run_workflow()`. Reconciliation +requires exactly one current terminal attempt record and the exact expected slot-token set; +missing, duplicate, foreign, stale, extra, contradictory, or unmapped failure evidence uses +the phase 4 `failed`, `cancelled`, `lost`, or `inconsistent` outcome rules. + +The terminal scope outcome is a pure reduction of the phase 4 planning-task outcome plus +candidate validation. A successful verified task yields `planned`; a prerequisite that +prevents dispatch yields `blocked`; all other task outcomes map to the same named non-success +scope outcome. An empty manifest yields `planned` through a verified-no-work task with no +backend attempt. + +Datum transformation tasks for Substitute are not ready until their scope-planning task is +terminal `planned`. A non-planned scope blocks those tasks and withholds every affected +datum and atomic group through the phase 4 fixed-point rules. No provider or backend dispatch +exists outside phase 4 conservation. + +Scope-planning readiness depends only on phase 6 cluster, role-result, mention, and compiled +context inputs for every member datum. A non-terminal phase 6 input leaves the +planning task unresolved and causes no dispatch; a terminal non-success input closes the +planning task and scope as `blocked`. Execution dependencies among member +datums do not gate the scope planner; applying datum dependency readiness to this grouped +task could deadlock a scope that contains both a prerequisite and its dependent. Normal +datum dependencies gate transformation and release after planning. + +## Ledger and linearization + +One invocation owns one bounded ledger. The ledger is created only after pure compilation +succeeds and is closed with the invocation. It is not serialized, returned, logged, or used +as downstream retry identity. + +Planning uses a scope-level all-or-none reservation: + +```text +absent + -> reserved(planning_attempt) + -> planned(immutable_bundle) + or aborted + or poisoned +``` + +`reserved`, `aborted`, and `poisoned` are private ledger states. The terminal accounting +model exposes their closed result as `planned`, `blocked`, `failed`, `cancelled`, `lost`, +or `inconsistent`. + +The planning linearization point is the atomic transition from a reservation owned by +the current planning attempt to the complete validated immutable bundle. Before that point, +no transformation task may read any candidate. After that point, all readers observe the +same bundle and no task may replace, merge, or partially update it. + +The executor schedules at most one active planner for a coherence scope. Different scopes +may plan concurrently. Compare-and-set ownership still guards every transition so duplicate +dispatch, late results, and implementation defects fail closed rather than overwrite state. + +An identical replay received before planning acceptance may be treated as an idempotent +notification only if no second effect or publication occurs. Conflicting evidence observed +before that linearization makes the scope inconsistent. After `planned` is accepted, the +terminal outcome is absorbing: later duplicate, different, stale, or foreign evidence is +rejected and cannot rewrite the scope. A result from another invocation is foreign even when +source data and declared scope are identical. + +`planned` is not an externally committed assignment. The bundle remains provisional and +invocation-private until phase 4 releases a qualified atomic-group output that uses it. +Rollback in this design means abandoning or poisoning an unpublished reservation or +candidate before `planned` is accepted. A planned bundle remains immutable and private until +phase 4 either releases qualified output or withholds and discards it during finalization. +Rollback does not undo provider execution, persistence, delivery, or any other downstream +effect. + +The closed transition table is: + +| From | Observation and guard | To | Phase 4 effect | +| --- | --- | --- | --- | +| `absent` | Admitted non-empty manifest; owner CAS succeeds | `reserved` | Planning task becomes dispatchable | +| `absent` | Verified empty manifest | `planned(empty)` | Task succeeds as verified no-work; zero dispatches | +| `absent` or `reserved` | Prerequisite prevents planning | `blocked` | Task and scope close blocked | +| `reserved` | Exactly one current result reconciles and validates | `planned(bundle)` | Task succeeds with private plan | +| `absent` or `reserved` | Local validation or attributable backend failure | `failed` | Task and scope close failed | +| `absent` or `reserved` | Cancellation before dispatch, or trusted stop before accepted result | `cancelled` | Task and scope close cancelled | +| `reserved` | Dispatch occurred without trusted stop or terminal run evidence | `lost` | Task and scope close lost | +| `reserved` | Contradictory, foreign, non-identical duplicate, or ambiguous evidence before acceptance | `poisoned` then `inconsistent` | Task and scope close inconsistent | +| `planned` | Any later result or terminal signal | `planned` | Evidence is stale and rejected; no state rewrite | +| Any terminal scope outcome | Repeated transition request | Same terminal outcome | Inert or rejected; no redispatch or release | + +Publication-critical finalization failure does not rewrite an absorbing scope outcome. It +closes the invocation as inconsistent and withholds every output under the phase 4 global +embargo. Post-acceptance host teardown failure likewise cannot rewrite scope or invocation +results already accepted for return. + +## Failure, cancellation, and lost execution + +Planning follows the phase 4 cancellation and lost-execution rules: + +- cancellation before dispatch aborts the reservation and closes the scope as `cancelled`; +- cancellation after dispatch closes as `cancelled` only with trusted evidence that the + planning execution stopped without producing an assignment; +- without trusted stop evidence, post-dispatch cancellation closes as `lost` and poisons + the scope; +- a transport break or missing trusted run record closes as `lost`; +- a late result from an aborted, lost, or superseded attempt cannot assign the scope; and +- duplicate, foreign, contradictory, or partial results close as `inconsistent`. + +Phase 7 performs no automatic planning retry. A downstream retry of delivery reuses the +exact already-protected payload when safe; it does not rerun Anonymizer. A new Anonymizer +invocation receives fresh correlation, scope, slot, and attempt identities and may produce +different assignments. + +A non-planned scope withholds every atomic group that contains a mention bound to that +scope. Dependency and group withholding then follow the phase 4 fixed-point rules. A scope +planning failure does not affect a graph component that neither uses nor depends on that +scope. + +## Transformation and fail-closed release + +Transformation reads only planned bundles. Each datum-local replacement application must +prove that: + +- every admitted Substitute mention resolves to one expected slot; +- every expected span is in range, non-overlapping, and still matches its anchored source + value; +- every targeted mention is applied exactly once; +- no unplanned value-only fallback selects an assignment; and +- the resulting datum passes the declared phase 6 and Substitute release predicates. + +Skipped, ambiguous, stale, overlapping, missing, or extra applications are not qualified +successes. They withhold the affected atomic group; raw input is never substituted for a +withheld output. + +Coherence and atomic release have different boundaries. A valid planned bundle may serve +several atomic groups. A precisely accounted transformation failure in one group does not +invalidate the assignment or automatically fail another group in the same coherence scope. +The other group can release only if its own datums, dependencies, and predicates qualify and +the invocation has no global accounting embargo. + +No replacement map becomes externally visible before exhaustive invocation reconciliation. +The public DataFrame compatibility path may continue to materialize its current trace +columns, but the private graph profile must not expose a partial scope bundle through a +result, exception, callback, log, receipt, or diagnostic view. + +## Bounds, cleanup, and privacy + +Compilation requires ceilings for scopes, clusters, slots, mentions, candidate bytes, and +total ledger bytes. The executor rejects unsupported or excessive declarations before +creating the ledger. It retains only the content needed for the active invocation and +releases ledger references on every terminal path. + +Cleanup must cover successful closure, validation failure, planner failure, cancellation, +lost execution, inconsistent accounting, transformation failure, and publication failure. +Before phase 4 releases output, phase 7 must abandon or poison every unused reservation, +close the planning ledger to mutation, and verify that no provisional bundle can become +observable. Failure in this publication-critical finalization makes the invocation +inconsistent and withholds output. + +Host-resource teardown after immutable result acceptance is a separate lifecycle step. Its +failure is reported through the owning host surface and cannot retroactively retract an +already accepted result. Tests must therefore inject pre-release finalization faults and +post-acceptance teardown faults as different events. Python reference release does not prove +physical zeroization, so the design makes no secure-erasure claim. + +Replacement values, original values, prompts, candidate bundles, and content-derived hashes +must not enter logs, metrics, exceptions, public receipts, or unbounded traces. Diagnostic +surfaces may expose only allowlisted reason codes, opaque invocation-private test identities, +and bounded counts by non-sensitive category. + +## Reference model and test oracle + +Build a pure reference model independent of pandas, DataDesigner, and the production ledger. +Its input is: + +- datums, mentions, clusters, slots, coherence scopes, dependencies, and atomic groups; +- closed slot and relational constraints; +- declared limits and capabilities; and +- a timestamp-free sequence of exogenous observations: dispatch accepted or rejected, + keyed candidate rows, `FailedRecord` evidence, backend exception, cancellation request, + trusted stop acknowledgement, transport or process loss, anchored transformation evidence, + verification-task evidence, publication-critical finalization success or failure, and + post-acceptance teardown success or failure. + +The model derives admission, reservation eligibility and transitions, candidate validation, +task and scope reconciliation, dependency propagation, terminal outcomes, finalization +consequences, datum and group eligibility, and the only legal release set. Production +reconciliation or release decisions are never model inputs. Production outcomes and release +sets must equal the reference result for every generated case. + +The central oracle is: + +```text +planned(scope) iff + its empty manifest completed as verified no-work with zero attempts + or exactly one current planning attempt produced one complete valid bundle + +qualified(datum) iff + its scope is planned + and every admitted mention resolved and applied exactly once + and every required predicate passed + +released(group) iff + phase-4 exhaustive reconciliation completed + and every member datum remains release-eligible + and no dependency, cancellation, loss, or accounting embargo applies +``` + +The finite exhaustive conformance envelope contains: + +- zero through four datums; +- zero through two coherence scopes; +- zero through three clusters, zero through four slots, and zero through six mentions; +- zero through three flat atomic groups and the phase 4 DAGs over the admitted datums; +- exactly one scope-planning task per admitted scope, with zero attempts for an empty + manifest and zero or one attempt for a non-empty manifest; a non-empty manifest has + exactly one attempt only after dispatch is accepted; +- at most one primary terminal observation and one late, duplicate, stale, foreign, or + contradictory observation per planning attempt; +- at most one cancellation request, one trusted-stop or loss observation, one transformation + observation per datum, one verification observation per group, one publication-critical + finalization observation, and one post-acceptance teardown observation; and +- at most 16 exogenous observations in one canonical trace. + +Canonicalization orders commuting events from independent scopes, tasks, and datums by +opaque declaration position while retaining every non-commuting race around dispatch, +planning acceptance, cancellation, finalization, and release. Before executor comparison, +freeze the model and generator versions, exact graph count, canonical schedule count, and a +SHA-256 digest of the manifest. Larger seeded state-machine tests supplement this envelope; +they do not replace it. + +## Admission and state-machine tests + +Admission tests cover empty scopes, gaps, duplicate membership, duplicate semantic scope +declarations, unknown datums, cross-scope clusters, unknown slots, duplicate slot +declarations, nesting, overlap, unsupported strategy, missing capabilities, and every +declared limit. Every rejection asserts zero +additional graph-owned ledger, workframe, graph telemetry, DataDesigner, transformation, +reconstruction, and publication effects. Provider or host construction that preceded +internal compilation on the current facade is outside this assertion. + +Spies also assert that rejection creates no invocation, attempt, reservation, or runtime +correlation token; performs no credential or resource lookup beyond already-bound host +construction; and emits no content-bearing telemetry. Adjacent-tier multiply-invalid cases +must return the earlier deterministic code under declaration and capability permutations. + +Separately test a non-empty scope whose datums contain no admitted Substitute mentions. It +must plan an empty bundle without provider or DataDesigner work and may release unchanged +text only after the normal qualified no-work accounting path. + +Readiness tests place a prerequisite and its dependent in one coherence scope. A +non-terminal phase 6 member leaves the planning task unresolved with zero planner +dispatches. After all required phase 6 inputs succeed, the planner dispatches without +waiting for either datum's transformation; transformations then follow the normal +dependency order. A terminal non-success phase 6 member closes the planning task and scope +as `blocked`, with zero planner dispatches. + +State-machine tests cover every allowed transition and reject: + +- read before planning completes; +- plan publication without the owning reservation; +- partial-bundle plan publication; +- second active reservation for one scope; +- overwrite or merge after planning; +- planning completion after abort, poison, cancellation, or loss; +- stale or foreign attempt results; +- duplicate or contradictory terminal signals; +- transformation before planning; and +- any terminal invocation containing an active reservation. + +Terminal states are absorbing. Replaying an accepted notification cannot duplicate an effect +or release. A late completion cannot resurrect a cancelled, lost, inconsistent, or closed +scope. + +## Candidate and collision tests + +Generate complete and malformed candidate bundles across cluster and slot shapes. Cover: + +- missing, extra, duplicate, empty, and unchanged assignments; +- synthetic/original collisions at the same slot and across the complete scope; +- duplicate synthetics for slots declared distinct; +- intentional reuse through one shared slot; +- one cluster with several type-appropriate related slots; +- type, format, wildcard, geographic, temporal, and contact consistency constraints; +- duplicate original text belonging to different clusters; +- identical labels with different slot identities; +- the same original or synthetic value in isolated scopes without shared state; +- candidate order and backend row-order permutations; and +- maximum allowed bundle size and one-over-limit rejection. + +Graph-profile tests assert that invalid bundles fail closed without placeholder repair or +automatic regeneration. Separate public compatibility tests preserve the existing row-local +collision behavior until a public change is separately reviewed. + +Before freezing the reference-model manifest, freeze a versioned closed slot-role and +relational-constraint vocabulary with positive and negative conformance fixtures. This +includes every supported type, format, wildcard, geographic, temporal, and contact rule. +An unknown role or constraint rejects at admission; an independent implementation is not +expected to infer a verdict from prompt prose. + +Planner reconciliation tests take the cross-product of current, missing, duplicate, stale, +and foreign attempt and slot tokens with success rows and `FailedRecord` evidence. One +failure attributed by trusted correlation evidence to exactly one planning task closes that +task failed; lack of slot-level attribution within that proven task invalidates its complete +scope. Unknown, duplicate, stale, success-plus-failure, or otherwise contradictory failure +evidence closes the invocation inconsistent. A planner `FailedRecord` that cannot be +attributed to exactly one expected task by an invocation-private task or attempt token also +closes the invocation inconsistent and triggers the phase 4 global release embargo, unless +trusted batch or call evidence independently proves that it belongs to exactly one planning +task. Content-derived record IDs never establish attribution. + +## Concurrency, cancellation, and fault injection + +Use a deterministic scheduler and barriers around reservation, dispatch, result receipt, +validation, plan publication, transformation, reconciliation, output release, and cleanup. +Explore the frozen finite envelope exhaustively and use seeded schedules for larger graphs. + +Required races include: + +- two planners attempting one scope; +- duplicate dispatch of one attempt; +- cancellation on both sides of dispatch and assignment; +- transport loss before and after the planner may have produced a candidate; +- late success after cancellation or loss; +- assignment concurrent with a foreign or contradictory result; +- one scope failing while independent scopes complete; +- transformation failure in one of several atomic groups sharing a scope; +- invocation cancellation after planning but before release; +- publication-critical finalization interrupted before release; and +- host teardown failure after immutable result acceptance. + +The race oracle is: + +| Accepted evidence order | Scope result | Dispatch count | Release effect | +| --- | --- | ---: | --- | +| Cancellation before dispatch | `cancelled` | 0 | No output | +| Dispatch, then cancellation without trusted stop | `lost` | 1 | Global phase 4 embargo | +| Dispatch, then trusted stop before candidate acceptance | `cancelled` | 1 | No output | +| Valid candidate accepted, then trusted stop or cancellation | `planned` remains absorbing | 1 | Invocation cancellation still withholds if accepted before output release | +| Trusted stop accepted, then candidate arrives | `cancelled`; candidate is stale | 1 | No output | +| Contradictory evidence before planning acceptance | `inconsistent` | 1 | Global phase 4 embargo | +| Byte-identical replay | Existing outcome | 1 | Inert; no duplicate effect | +| Output release accepted, then cancellation | Existing released result | 1 | No retroactive retraction | + +No race causes an automatic redispatch. Every dispatched planning attempt corresponds to +exactly one Anonymizer dispatch even if the provider or DataDesigner performs opaque work +internally. + +Inject malformed, missing, extra, duplicate, stale, and foreign workframe results. At least +one mandatory process-kill test must use a test-only crashable backend at the existing +execution seam, with no production process or session API. It asserts only `lost`, poisoned +or abandoned private state, complete cleanup handling, and an empty release set. This is +lifecycle evidence for phase 7, not the materially different semantic runtime required for +stable public promotion. + +Run sequential and concurrent fresh invocations with identical declarations and content. +They must not reuse ledger, reservation, scope, slot, attempt, correlation-token, or bundle +object identity; an old result is foreign. Each scope with a non-empty manifest dispatches +a new planner, while an empty manifest creates a fresh verified-no-work task with zero +attempts. Equal generated literals are allowed but do not prove shared state. Ledger access +must fail after every invocation terminal path. + +## Property, privacy, and compatibility tests + +Required properties include: + +- **Conservation:** every admitted scope has exactly one terminal outcome; every expected + slot appears exactly once in its planned bundle or is covered by the scope's non-planned + outcome. +- **Dispatch conservation:** every admitted scope has exactly one phase 4 planning-task + record; an empty manifest has zero attempts, and every accepted non-empty planner dispatch + has exactly one current attempt-terminal record or an explicit `lost` classification. +- **Completeness:** `planned` contains every expected slot and no other slot. +- **Stability:** every mention of one slot observes the same assignment. +- **Collision safety:** no distinct slots in one uniqueness namespace receive the same + forbidden or canonicalized synthetic value. +- **Declared reuse:** every mention of a shared slot receives one value, while slots declared + distinct never collapse through equal text or labels. +- **Application conservation:** every admitted mention is applied exactly once at its + authoritative span; released output retains no admitted original at that span. +- **Non-cascading application:** a synthetic value is never treated as a second source span + during the same transformation. +- **Isolation:** changing or failing one independent scope does not change another scope. +- **Permutation invariance:** declaration, scheduling, workframe, and candidate order do not + change semantic results. +- **Identity invariance:** opaque identity renaming yields an isomorphic outcome. +- **Content non-identity:** equal original text does not merge clusters or slots. +- **Monotonic withholding:** replacing a valid event with blocking, failure, cancellation, + loss, or inconsistency cannot increase the release set. +- **Attempt isolation:** an old or foreign attempt cannot assign the current scope. +- **No partial visibility:** a non-planned bundle and a withheld group expose no replacement + values through any observable surface. +- **Boundedness:** retained states and diagnostics stay within declared ceilings. + +Use separate high-entropy canaries for original values, synthetic values, prompts, source +identity, and known digests of each. Enforce a structural surface-and-lifecycle allowlist: + +| Surface | Original values | Synthetic values | +| --- | --- | --- | +| Active private planner input, prompt, ledger, and workframe | Only the compiled projection | Only the active candidate or plan channel | +| Released private graph output | Not at admitted anchored spans | Allowed only as protected text | +| Grandfathered public compatibility trace | Allowed only in existing trace fields | Allowed only in existing replacement-map and protected-text fields | +| Withheld or non-planned result | Forbidden | Forbidden | +| Logs, metrics, exceptions, receipts, IDs, diagnostics, and cleanup errors | Forbidden, including known digests | Forbidden, including known digests | + +Structural allowlist assertions are primary; substring and known-digest scans supplement +them. Serialized results are inspected field by field rather than rejected merely because a +qualified protected output contains its synthetic value. + +Compatibility tests preserve duplicate and non-monotonic DataFrame indexes, duplicate text, +row reordering, existing public replacement-map trace behavior, result columns and +attributes, CLI behavior, and current Substitute configuration. The matrix explicitly +covers no-entity provider bypass, filtering of unrequested entries, same-row placeholder +collision repair, custom instructions in the generator prompt, preview limiting after the +entity-row partition, non-cascading offset application, the legacy value-only fallback, +`FailedRecord` passthrough with exact ID, step, order, and shape, and the separate +`evaluate()` judge path. + +Exercise those contracts through `run()`, `preview()`, `evaluate()`, trace materialization, +display, validation, and CLI paths. In paired cases, the unchanged public facade retains +legacy filtering and repair while the private graph profile rejects the same invalid bundle +without partial output. Spies assert that every DataDesigner execution still passes through +`NddAdapter.run_workflow()` and that no source-format type enters Anonymizer core. + +Mutation tests must catch at least premature bundle visibility, slot joins by original text, +row-local regeneration inside a coherence scope, accepted partial bundles, collision checks +limited to one datum, distinct-slot aliasing, shared-slot divergence, last-writer-wins +planning, stale-attempt adoption, cancellation resurrection, value-only fallback, cascading +replacement, missing or duplicate application, released output retaining an admitted +original at its anchored span, raw-input fallback, and incomplete cleanup. + +## Ownership and promotion gates + +| Decision | Required authority | +| --- | --- | +| Scope grammar, slot semantics, collision policy, outcome algebra, and release predicates | Project and Anonymizer semantic owner | +| Invocation lifetime, ceilings, concurrency, provider authority, and cleanup observability | Host authority | +| Source-to-scope mapping, reconstruction, retry identity, and destination postconditions | Source-adapter and adopter owners | +| Trust boundary, allowed linkability, leakage criteria, and residual risk | Customer or consuming-product owner | +| Any public session, graph, ledger, receipt, artifact, or endpoint | Public-API and Platform owners | +| Any durable or cross-worker assignment backend | Separately named state-service and governance owners | + +No explanatory SDK or type-safety guidance assigns these product decisions. RFC acceptance +does not replace a named owner's later product, public-API, deployment, or promotion decision. + +## Review gates + +Phase 7 is ready for implementation only after reviewers accept: + +1. the invocation-bounded stability claim and explicit non-durability; +2. the flat exact coherence-scope partition and cross-scope cluster rejection; +3. cluster-to-slot modeling and the closed relational policy; +4. all-or-none provisional scope bundles and planning linearization; +5. fail-closed collision handling with no automatic repair or retry in the graph profile; +6. cancellation, loss, late-result, and cleanup behavior; +7. interaction between coherence scopes, dependencies, and atomic groups; +8. the pure reference model, finite schedule exploration, mutation set, and privacy canaries; +9. unchanged public DataFrame behavior and the `NddAdapter.run_workflow()` boundary; and +10. the retained downstream ownership of durable state, retries, deduplication, + reconstruction, persistence, and delivery. + +The 2026-08-20 review accepted this architecture as the design candidate for a future branch +checkpoint; it did not authorize implementation. Phase 7 remains blocked until Phases 4–6 pass and the versioned semantic and +execution contract in the decision record is reviewed and frozen; that contract supplies +the concrete closed relational policy and execution limits required by gates 3, 6, and 8. + +Any future operator checkpoint must not bypass these prerequisites. Passing Phase 7 tests later +would not authorize a public graph or session API, durable replacement state, production +Intake or OpenShell support, a `zero PII` claim, grouped rewrite, or stable public promotion. + +## Evidence and deferred decisions + +The current implementation and tests establish row-local map generation, filtering, +same-row synthetic/original collision repair, offset-based non-cascading application, +PII-free log summaries for tested paths, and public trace compatibility. They do not +establish coherence scopes, cluster-to-slot identity, graph-wide collision policy, +concurrent ledger semantics, rollback, or cross-call stability. + +The branch decision selected invocation-bounded stability as the only Phase 7 profile and retained +the no-regeneration baseline. The versioned role and constraint vocabulary, distinct-slot +matrix, ceilings, and cleanup-observability contract are owned pre-implementation inputs as +recorded above; they are not unclassified open decisions. + +Whether a future governed state backend belongs in Anonymizer, the execution host, or a separate +service remains deferred. No durable backend is selected by this design, and that later +ownership decision does not block the invocation-bounded phase 7 profile. diff --git a/docs/development/plan-a-design-spike.md b/docs/development/plan-a-design-spike.md new file mode 100644 index 00000000..cba099df --- /dev/null +++ b/docs/development/plan-a-design-spike.md @@ -0,0 +1,81 @@ + + + +# Historical private protection design-spike record + +Status: historical branch-local implementation evidence. This record predates +and is subordinate to the current +[graph-native technical proposal](graph-native-anonymizer-sdk-technical-proposal.md); +its former “Plan A” vocabulary is retained only where needed to describe the +implemented checkpoint and test lineage. + +This record covers the private, synchronous protection slice only. It does not +authorize a public SDK, Intake integration, release decision, or alternate +engine path. + +The Intake team is the named adopter. The project owner accepted the canonical +cause-free `AnonymizerWorkflowError` mapping for pipeline failures that cross +the private row-verification boundary. The customer or consumer PII boundary +remains unresolved and must be recorded by the consuming-product owner before +a production validation placement is selected. + +## Ownership verdicts + +Ownership: stay; owner=anonymizer.interface._protection._compile_protection_plan; evidence=the helper consumes AnonymizerConfig, ModelSelection, and ModelConfig and returns a distinct private plan; reason=cross-domain release-policy compilation is not a same-type config transform. + +Ownership: stay; owner=anonymizer.interface._protection._build_operation_plan; evidence=the helper consumes a compiled plan and protection records and returns an invocation-private operation plan; reason=batch admission bounds and correlation span several domain values. + +Ownership: stay; owner=anonymizer.interface._protection._ProtectionFlow._execute; evidence=the lifecycle caller coordinates the pandas runtime, invocation verifier, and terminal accounting; reason=effectful runtime coordination does not belong on a frozen domain value. + +Ownership: stay; owner=anonymizer.interface._protection._failure; evidence=the helper constructs one closed safe-failure domain value from static enum and stage inputs; reason=the private protection domain module owns its stable failure taxonomy. + +Ownership: stay; owner=anonymizer.interface._protection release-policy helpers; evidence=_has_accepted_detections and _redact_release_passed inspect verified engine entity values to derive a protection disposition and enforce the compiled Redact predicate; reason=these helpers coordinate engine schema values with protection-domain policy rather than transform one config model. + +Ownership: stay; owner=anonymizer.interface._protection._SafeRepr; evidence=all new private domain values inherit the content-free rendering mixin; reason=one domain-local rendering policy prevents record content and references from entering repr, logs, or errors. + +Ownership: stay; owner=anonymizer.engine.ndd.adapter.NddAdapter.private_execution; evidence=the policy coordinates DataDesigner invocation artifacts and ambient measurement, message-trace, and task-trace collection across all engine workflows while preserving run_workflow as the sole execution boundary; reason=filesystem and collector isolation belong at the DataDesigner adapter boundary, with _ProtectionFlow activating the policy only for private execution. + +## Test strategy + +Focused contract tests cover private value bounds, closed compilation outcomes, +pre-admission batch rejection, terminal accounting, lifecycle overlap and close, +safe rendering, and adversarial engine results. A focused integration test uses +the synthetic detector and the real local Redact/pandas runtime seam. Tests +assert returned outcomes and absence of private data, not internal call counts. + +The flow borrows the facade runtime. Closing it rejects new admission and lets +an already admitted synchronous invocation drain; it never closes the borrowed +DataDesigner or provider resources. The current runtime does not establish hard +cancellation or deterministic dependency teardown, and this spike makes no such +claim. Execution completion remains distinct from external data-handling and +release or commit authority. + +The implemented private execution path gives each DataDesigner `create()` call +an invocation-scoped temporary artifact root, loads its output before cleanup, +and disables ambient Anonymizer measurement plus DataDesigner message and task +traces for that scope. `preview()` retains its existing in-memory behavior. +Adapter and flow canaries cover durable-root and collector isolation, backend +exception and failed-record diagnostic safety, and fail-closed terminal +accounting. + +The compiled plan deep-copies execution inputs and fingerprints the complete +allowlisted Plan A semantic snapshot, including selected models, model configs, +detection settings, replacement settings, profile, versions, and limits. +Execution verifies that fingerprint before effects. Receipts bind the plan +digest and a fresh content-independent attempt identity. Failure retry safety +is `unknown` and retry ownership is `Unassigned`; this spike makes no stronger +retry claim before taxonomy review. + +## Intake-format validation + +The synthetic validation corpus covers ATIF v1.0 and v1.7, an +extension-bearing chat-completion request and response, a real OTLP protobuf +batch, and an Intake-shaped local CHAIN-to-LLM trace. Every declared target +runs through the private Plan A flow with deterministic local detection and +`Redact`. Tests verify reconstruction, structural and topology preservation, +closed field handling, and complete-item withholding after invalid OTLP spans +or non-success outcomes. + +This is adapter and execution evidence only. It does not run the Intake +service, persist records, call an external provider, use customer data, or +establish ATIF, chat-completion, or OTLP production support. diff --git a/pyproject.toml b/pyproject.toml index 4c5c4b82..841925a9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -26,6 +26,7 @@ anonymizer-chunked-validation = "anonymizer.engine.workflow_columns.detection.pl [dependency-groups] dev = [ + "opentelemetry-proto>=1.27.0,<2", "pre-commit>=4.0.0,<5", "pytest>=9.0.3,<10", "pytest-cov>=7.0,<8", diff --git a/src/anonymizer/__init__.py b/src/anonymizer/__init__.py index 4a63168a..e5492e86 100644 --- a/src/anonymizer/__init__.py +++ b/src/anonymizer/__init__.py @@ -25,6 +25,7 @@ from anonymizer.interface.errors import ( AnonymizerError, AnonymizerIOError, + AnonymizerWorkflowError, InvalidConfigError, InvalidInputError, ) @@ -52,6 +53,7 @@ def __getattr__(name: str) -> object: "AnonymizerError", "AnonymizerInput", "AnonymizerIOError", + "AnonymizerWorkflowError", "Annotate", "DEFAULT_ENTITY_LABELS", "Detect", diff --git a/src/anonymizer/engine/constants.py b/src/anonymizer/engine/constants.py index c0c47dec..c938d92b 100644 --- a/src/anonymizer/engine/constants.py +++ b/src/anonymizer/engine/constants.py @@ -12,6 +12,26 @@ # Input COL_TEXT = "__nemo_anonymizer_text_input__" +# Private target/context workframes. These columns never enter public results. +COL_TARGET_WORK_ID = "__anonymizer_private_row_correlation__" +COL_TASK_ID = "__anonymizer_private_task_identity__" +COL_ATTEMPT_ID = "__anonymizer_private_attempt_identity__" +COL_CONTEXT_BINDING_ID = "__anonymizer_context_binding_id__" +COL_CONTEXT_OWNER_WORK_ID = "__anonymizer_context_owner_work_id__" +COL_CONTEXT_ORDINAL = "__anonymizer_context_ordinal__" +COL_CONTEXT_TEXT = "__anonymizer_context_text__" + +# Private Phase 6 provider workframes. These columns never enter public results. +COL_PHASE6_CONTEXT = "__anonymizer_phase6_context__" +COL_PHASE6_CANDIDATES = "__anonymizer_phase6_candidates__" +COL_PHASE6_AUGMENTED = "__anonymizer_phase6_augmented__" +COL_PHASE6_VALIDATION = "__anonymizer_phase6_validation__" + +# Private Phase 7 candidate workframes. These columns never enter public results. +COL_PHASE7_INVOCATION_ID = "__anonymizer_phase7_invocation_identity__" +COL_PHASE7_CANDIDATE_REQUEST = "__anonymizer_phase7_candidate_request__" +COL_PHASE7_CANDIDATE_BUNDLE = "__anonymizer_phase7_candidate_bundle__" + # Step 1: GLiNER detection COL_RAW_DETECTED = "_raw_detected_entities" diff --git a/src/anonymizer/engine/detection/detection_workflow.py b/src/anonymizer/engine/detection/detection_workflow.py index a577a47b..f8e7e55f 100644 --- a/src/anonymizer/engine/detection/detection_workflow.py +++ b/src/anonymizer/engine/detection/detection_workflow.py @@ -5,7 +5,7 @@ import logging from copy import deepcopy -from dataclasses import dataclass +from dataclasses import dataclass, field from pathlib import Path from typing import cast @@ -42,7 +42,7 @@ _jinja, ) from anonymizer.engine.detection.postprocess import EntitySpan, group_entities_by_value -from anonymizer.engine.ndd.adapter import FailedRecord, NddAdapter +from anonymizer.engine.ndd.adapter import FailedRecord, NddAdapter, _FailedRowEvidence from anonymizer.engine.ndd.model_loader import resolve_model_alias, resolve_model_aliases from anonymizer.engine.prompt_utils import substitute_placeholders from anonymizer.engine.schemas import ( @@ -75,6 +75,7 @@ class EntityDetectionResult: dataframe: pd.DataFrame failed_records: list[FailedRecord] + failed_row_evidence: tuple[_FailedRowEvidence, ...] = field(default=(), repr=False) class EntityDetectionWorkflow: @@ -123,7 +124,11 @@ def detect_and_validate_entities( preview_num_records=preview_num_records, ) detected_df = detection_result.dataframe.copy() - return EntityDetectionResult(dataframe=detected_df, failed_records=detection_result.failed_records) + return EntityDetectionResult( + dataframe=detected_df, + failed_records=detection_result.failed_records, + failed_row_evidence=detection_result.failed_row_evidence, + ) def _build_detection_spec( self, @@ -347,7 +352,11 @@ def identify_latent_entities( workflow_name="latent-entity-detection", preview_num_records=preview_num_records, ) - return EntityDetectionResult(dataframe=latent_result.dataframe, failed_records=latent_result.failed_records) + return EntityDetectionResult( + dataframe=latent_result.dataframe, + failed_records=latent_result.failed_records, + failed_row_evidence=latent_result.failed_row_evidence, + ) def run( self, @@ -407,9 +416,14 @@ def run( ) final_df = latent_result.dataframe.copy() final_failures = [*detected_result.failed_records, *latent_result.failed_records] + final_failure_evidence = ( + *detected_result.failed_row_evidence, + *latent_result.failed_row_evidence, + ) else: final_df = detected_result.dataframe.copy() final_failures = detected_result.failed_records + final_failure_evidence = detected_result.failed_row_evidence # When entity_labels is explicitly provided (even if it matches DEFAULT_ENTITY_LABELS), # the augmenter is strict and out-of-scope labels are filtered. @@ -425,6 +439,7 @@ def run( result = EntityDetectionResult( dataframe=final_df, failed_records=final_failures, + failed_row_evidence=final_failure_evidence, ) measurement.update( output_row_count=len(result.dataframe), @@ -440,19 +455,35 @@ def _inject_detector_params( labels: list[str], gliner_detection_threshold: float, ) -> list[ModelConfig]: - resolved = deepcopy(model_configs) - for config in resolved: - if config.alias != selected_models.entity_detector: - continue - if config.inference_parameters.extra_body is None: - config.inference_parameters.extra_body = {} - config.inference_parameters.extra_body["labels"] = labels - config.inference_parameters.extra_body["threshold"] = gliner_detection_threshold - config.inference_parameters.extra_body["chunk_length"] = 384 - config.inference_parameters.extra_body["overlap"] = 128 - config.inference_parameters.extra_body["flat_ner"] = False - break - return resolved + return _inject_detector_params( + model_configs=model_configs, + selected_models=selected_models, + labels=labels, + gliner_detection_threshold=gliner_detection_threshold, + ) + + +def _inject_detector_params( + *, + model_configs: list[ModelConfig], + selected_models: DetectionModelSelection, + labels: list[str], + gliner_detection_threshold: float, +) -> list[ModelConfig]: + """Return detached GLiNER model configs for one detector workflow.""" + resolved = deepcopy(model_configs) + for config in resolved: + if config.alias != selected_models.entity_detector: + continue + if config.inference_parameters.extra_body is None: + config.inference_parameters.extra_body = {} + config.inference_parameters.extra_body["labels"] = labels + config.inference_parameters.extra_body["threshold"] = gliner_detection_threshold + config.inference_parameters.extra_body["chunk_length"] = 384 + config.inference_parameters.extra_body["overlap"] = 128 + config.inference_parameters.extra_body["flat_ner"] = False + break + return resolved def _resolve_detection_labels(entity_labels: list[str] | None) -> list[str]: diff --git a/src/anonymizer/engine/execution/__init__.py b/src/anonymizer/engine/execution/__init__.py new file mode 100644 index 00000000..f16d8026 --- /dev/null +++ b/src/anonymizer/engine/execution/__init__.py @@ -0,0 +1,4 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Private pandas execution coordination for the public interface.""" diff --git a/src/anonymizer/engine/execution/accounting_admission.py b/src/anonymizer/engine/execution/accounting_admission.py new file mode 100644 index 00000000..12f4dbf2 --- /dev/null +++ b/src/anonymizer/engine/execution/accounting_admission.py @@ -0,0 +1,344 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pure admission boundary for phase-4 accounting plans.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from typing import NoReturn, TypeAlias + +from anonymizer.engine.execution.accounting_plan import ( + _AccountingLimits, + _AccountingPlan, + _admit_accounting_plan, + _AtomicGroupKey, + _CompiledAtomicGroup, + _CompiledDependency, + _DatumTaskSubject, + _StageId, + _TaskKey, +) +from anonymizer.engine.execution.graph import ( + _AtomicGroup, + _CoherenceScope, + _ContextScope, + _DatumDependency, + _DatumId, + _DatumLink, + _DatumPurpose, + _ProtectionGraph, + _TextDatum, +) + + +class _AccountingAdmissionCode(str, Enum): + MALFORMED_GRAPH = "malformed_graph" + TOO_MANY_DATUMS = "too_many_datums" + DATUM_TOO_LARGE = "datum_too_large" + GRAPH_TOO_LARGE = "graph_too_large" + DUPLICATE_DATUM_ID = "duplicate_datum_id" + TOO_MANY_DEPENDENCIES = "too_many_dependencies" + TOO_MANY_ATOMIC_GROUPS = "too_many_atomic_groups" + MALFORMED_DEPENDENCY = "malformed_dependency" + DANGLING_DEPENDENCY = "dangling_dependency" + SELF_DEPENDENCY = "self_dependency" + DUPLICATE_DEPENDENCY = "duplicate_dependency" + DEPENDENCY_CYCLE = "dependency_cycle" + EMPTY_ATOMIC_GROUP = "empty_atomic_group" + DANGLING_ATOMIC_MEMBER = "dangling_atomic_member" + DUPLICATE_ATOMIC_MEMBER = "duplicate_atomic_member" + DUPLICATE_ATOMIC_GROUP = "duplicate_atomic_group" + ATOMIC_COVERAGE_GAP = "atomic_coverage_gap" + ATOMIC_GROUP_OVERLAP = "atomic_group_overlap" + UNSUPPORTED_ATOMIC_NESTING = "unsupported_atomic_nesting" + UNSUPPORTED_RELATIONSHIPS = "unsupported_relationships" + UNSUPPORTED_CONTEXT = "unsupported_context" + UNSUPPORTED_COHERENCE = "unsupported_coherence" + UNSUPPORTED_TASK_CARDINALITY = "unsupported_task_cardinality" + + +@dataclass(frozen=True, slots=True, repr=False) +class _AccountingRejected: + code: _AccountingAdmissionCode + + def __repr__(self) -> str: + return "" + + +_AccountingAdmissionResult: TypeAlias = _AccountingPlan | _AccountingRejected + + +class _AdmissionFailure(Exception): + def __init__(self, code: _AccountingAdmissionCode) -> None: + self.code = code + + +def _compile_accounting_plan( + graph: object, + *, + limits: _AccountingLimits, + stages: tuple[str, ...] = ("protect",), +) -> _AccountingAdmissionResult: + try: + return _compile(graph, limits=limits, stages=stages) + except _AdmissionFailure as failure: + return _AccountingRejected(failure.code) + + +def _compile(graph: object, *, limits: _AccountingLimits, stages: tuple[str, ...]) -> _AccountingPlan: + if not isinstance(graph, _ProtectionGraph): + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + _check_counts(graph, limits) + datums, datum_by_value = _compile_datums(graph.datums, limits) + _check_declaration_shapes(graph) + dependencies = _compile_dependencies(graph.dependencies, datum_by_value) + group_members = _compile_group_members(graph.atomic_groups, datum_by_value) + _check_reference_coverage(group_members, datum_by_value) + _check_dependency_structure(dependencies) + _check_partial_overlap(group_members) + topological_datums = _topological_order(tuple(datum.id for datum in datums), dependencies) + _check_unsupported_semantics(graph, group_members, datum_by_value, stages, limits) + compiled_stages = tuple(_StageId(stage) for stage in stages) + groups = _materialize_groups(group_members, datum_by_value) + tasks = tuple( + _TaskKey(stage, _DatumTaskSubject(datum_id)) for stage in compiled_stages for datum_id in topological_datums + ) + return _admit_accounting_plan( + datums, + compiled_stages, + tasks, + dependencies, + groups, + topological_datums, + ) + + +def _check_counts(graph: _ProtectionGraph, limits: _AccountingLimits) -> None: + datums = getattr(graph, "datums", None) + dependencies = getattr(graph, "dependencies", None) + groups = getattr(graph, "atomic_groups", None) + if not isinstance(datums, tuple) or not datums: + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + if len(datums) > limits.max_datums: + _fail(_AccountingAdmissionCode.TOO_MANY_DATUMS) + if isinstance(dependencies, tuple) and len(dependencies) > limits.max_dependencies: + _fail(_AccountingAdmissionCode.TOO_MANY_DEPENDENCIES) + if isinstance(groups, tuple) and len(groups) > limits.max_atomic_groups: + _fail(_AccountingAdmissionCode.TOO_MANY_ATOMIC_GROUPS) + + +def _compile_datums( + source: tuple[object, ...], limits: _AccountingLimits +) -> tuple[tuple[_TextDatum, ...], dict[str, _DatumId]]: + datums: list[_TextDatum] = [] + datum_by_value: dict[str, _DatumId] = {} + total_bytes = 0 + for candidate in source: + if not isinstance(candidate, _TextDatum): + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + try: + datum_id = candidate.id + value = datum_id.value + text = candidate.text + purpose = candidate.purpose + except AttributeError: + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + if ( + not isinstance(datum_id, _DatumId) + or not isinstance(value, str) + or not value + or not isinstance(text, str) + or purpose is not _DatumPurpose.TARGET + ): + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + if _utf8_size(value) > limits.max_id_bytes: + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + size = _utf8_size(text) + if size > limits.max_datum_bytes: + _fail(_AccountingAdmissionCode.DATUM_TOO_LARGE) + total_bytes += size + if value in datum_by_value: + _fail(_AccountingAdmissionCode.DUPLICATE_DATUM_ID) + detached_id = _DatumId(value) + datum_by_value[value] = detached_id + datums.append(_TextDatum(detached_id, text)) + if total_bytes > limits.max_graph_bytes: + _fail(_AccountingAdmissionCode.GRAPH_TOO_LARGE) + return tuple(datums), datum_by_value + + +def _utf8_size(value: str) -> int: + try: + return len(value.encode("utf-8")) + except UnicodeEncodeError: + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + + +def _check_declaration_shapes(graph: _ProtectionGraph) -> None: + if not isinstance(getattr(graph, "dependencies", None), tuple): + _fail(_AccountingAdmissionCode.MALFORMED_DEPENDENCY) + if not isinstance(getattr(graph, "atomic_groups", None), tuple): + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + if not isinstance(getattr(graph, "links", None), tuple) or not all( + isinstance(link, _DatumLink) for link in graph.links + ): + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + if not isinstance(getattr(graph, "context_scopes", None), tuple) or not all( + isinstance(scope, _ContextScope) for scope in graph.context_scopes + ): + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + if not isinstance(getattr(graph, "coherence_scopes", None), tuple) or not all( + isinstance(scope, _CoherenceScope) for scope in graph.coherence_scopes + ): + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + + +def _compile_dependencies( + source: tuple[object, ...], datum_by_value: dict[str, _DatumId] +) -> tuple[_CompiledDependency, ...]: + dependencies: list[_CompiledDependency] = [] + for candidate in source: + if not isinstance(candidate, _DatumDependency): + _fail(_AccountingAdmissionCode.MALFORMED_DEPENDENCY) + try: + prerequisite = candidate.prerequisite.value + dependent = candidate.dependent.value + except AttributeError: + _fail(_AccountingAdmissionCode.MALFORMED_DEPENDENCY) + if not isinstance(prerequisite, str) or not isinstance(dependent, str): + _fail(_AccountingAdmissionCode.MALFORMED_DEPENDENCY) + if prerequisite not in datum_by_value or dependent not in datum_by_value: + _fail(_AccountingAdmissionCode.DANGLING_DEPENDENCY) + dependencies.append(_CompiledDependency(datum_by_value[prerequisite], datum_by_value[dependent])) + return tuple(dependencies) + + +def _compile_group_members( + source: tuple[object, ...], datum_by_value: dict[str, _DatumId] +) -> tuple[frozenset[str], ...]: + groups: list[frozenset[str]] = [] + for candidate in source: + if not isinstance(candidate, _AtomicGroup): + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + try: + members = candidate.members + except AttributeError: + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + if not isinstance(members, tuple): + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + if not members: + _fail(_AccountingAdmissionCode.EMPTY_ATOMIC_GROUP) + values: list[str] = [] + for member in members: + value = getattr(member, "value", None) + if not isinstance(member, _DatumId) or not isinstance(value, str): + _fail(_AccountingAdmissionCode.MALFORMED_GRAPH) + if value not in datum_by_value: + _fail(_AccountingAdmissionCode.DANGLING_ATOMIC_MEMBER) + values.append(value) + if len(set(values)) != len(values): + _fail(_AccountingAdmissionCode.DUPLICATE_ATOMIC_MEMBER) + member_set = frozenset(values) + if member_set in groups: + _fail(_AccountingAdmissionCode.DUPLICATE_ATOMIC_GROUP) + groups.append(member_set) + return tuple(groups) + + +def _check_reference_coverage(groups: tuple[frozenset[str], ...], datum_by_value: dict[str, _DatumId]) -> None: + covered = set().union(*groups) if groups else set() + if covered != set(datum_by_value): + _fail(_AccountingAdmissionCode.ATOMIC_COVERAGE_GAP) + + +def _check_dependency_structure(dependencies: tuple[_CompiledDependency, ...]) -> None: + observed: set[tuple[str, str]] = set() + for dependency in dependencies: + edge = (dependency.prerequisite.value, dependency.dependent.value) + if edge[0] == edge[1]: + _fail(_AccountingAdmissionCode.SELF_DEPENDENCY) + if edge in observed: + _fail(_AccountingAdmissionCode.DUPLICATE_DEPENDENCY) + observed.add(edge) + + +def _check_partial_overlap(groups: tuple[frozenset[str], ...]) -> None: + for index, left in enumerate(groups): + for right in groups[index + 1 :]: + if left & right and not (left < right or right < left): + _fail(_AccountingAdmissionCode.ATOMIC_GROUP_OVERLAP) + + +def _topological_order( + datum_ids: tuple[_DatumId, ...], dependencies: tuple[_CompiledDependency, ...] +) -> tuple[_DatumId, ...]: + position = {datum_id.value: index for index, datum_id in enumerate(datum_ids)} + incoming = {datum_id.value: 0 for datum_id in datum_ids} + dependents: dict[str, list[str]] = {datum_id.value: [] for datum_id in datum_ids} + for dependency in dependencies: + incoming[dependency.dependent.value] += 1 + dependents[dependency.prerequisite.value].append(dependency.dependent.value) + ready = [datum_id.value for datum_id in datum_ids if incoming[datum_id.value] == 0] + ordered: list[_DatumId] = [] + while ready: + value = ready.pop(0) + ordered.append(datum_ids[position[value]]) + for dependent in dependents[value]: + incoming[dependent] -= 1 + if incoming[dependent] == 0: + ready.append(dependent) + ready.sort(key=position.__getitem__) + if len(ordered) != len(datum_ids): + _fail(_AccountingAdmissionCode.DEPENDENCY_CYCLE) + return tuple(ordered) + + +def _check_unsupported_semantics( + graph: _ProtectionGraph, + groups: tuple[frozenset[str], ...], + datum_by_value: dict[str, _DatumId], + stages: tuple[str, ...], + limits: _AccountingLimits, +) -> None: + if any(left < right or right < left for index, left in enumerate(groups) for right in groups[index + 1 :]): + _fail(_AccountingAdmissionCode.UNSUPPORTED_ATOMIC_NESTING) + if graph.links: + _fail(_AccountingAdmissionCode.UNSUPPORTED_RELATIONSHIPS) + expected = set(datum_by_value) + context_targets = {getattr(scope.target, "value", None) for scope in graph.context_scopes} + if ( + len(graph.context_scopes) != len(expected) + or context_targets != expected + or any(scope.context for scope in graph.context_scopes) + ): + _fail(_AccountingAdmissionCode.UNSUPPORTED_CONTEXT) + coherence = tuple(frozenset(member.value for member in scope.members) for scope in graph.coherence_scopes) + if len(coherence) != len(expected) or set(coherence) != {frozenset((value,)) for value in expected}: + _fail(_AccountingAdmissionCode.UNSUPPORTED_COHERENCE) + if ( + not isinstance(stages, tuple) + or not stages + or len(stages) > limits.max_stages + or any(not isinstance(stage, str) or not stage for stage in stages) + or len(set(stages)) != len(stages) + ): + _fail(_AccountingAdmissionCode.UNSUPPORTED_TASK_CARDINALITY) + + +def _materialize_groups( + groups: tuple[frozenset[str], ...], datum_by_value: dict[str, _DatumId] +) -> tuple[_CompiledAtomicGroup, ...]: + canonical = sorted(groups, key=lambda members: tuple(sorted(members))) + return tuple( + _CompiledAtomicGroup( + _AtomicGroupKey(), + tuple(datum_id for value, datum_id in datum_by_value.items() if value in members), + ) + for members in canonical + ) + + +def _fail(code: _AccountingAdmissionCode) -> NoReturn: + raise _AdmissionFailure(code) diff --git a/src/anonymizer/engine/execution/accounting_evidence.py b/src/anonymizer/engine/execution/accounting_evidence.py new file mode 100644 index 00000000..e0236247 --- /dev/null +++ b/src/anonymizer/engine/execution/accounting_evidence.py @@ -0,0 +1,64 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Opaque invocation identities and keyed terminal evidence.""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Generic, TypeAlias, TypeVar, final + +from anonymizer.engine.execution.accounting_plan import _TaskKey + +T = TypeVar("T") + + +class _PrivateEvidenceValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private accounting evidence is not serializable") + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _InvocationId(_PrivateEvidenceValue): + value: str + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _AttemptId(_PrivateEvidenceValue): + value: str + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _RowToken(_PrivateEvidenceValue): + value: str + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _Dispatch(_PrivateEvidenceValue): + invocation_id: _InvocationId + task: _TaskKey + attempt_id: _AttemptId + row_token: _RowToken + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _SuccessRecord(_PrivateEvidenceValue, Generic[T]): + dispatch: _Dispatch + candidate: T + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _FailureRecord(_PrivateEvidenceValue): + dispatch: _Dispatch + + +_TerminalRecord: TypeAlias = _SuccessRecord[T] | _FailureRecord diff --git a/src/anonymizer/engine/execution/accounting_ledger.py b/src/anonymizer/engine/execution/accounting_ledger.py new file mode 100644 index 00000000..9a7b978e --- /dev/null +++ b/src/anonymizer/engine/execution/accounting_ledger.py @@ -0,0 +1,929 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""One-shot phase-4 invocation ledger and terminal evidence acceptance.""" + +from __future__ import annotations + +import operator +import secrets +from collections.abc import Callable, Iterable +from dataclasses import dataclass +from enum import Enum +from functools import reduce, wraps +from threading import RLock +from typing import Concatenate, Generic, ParamSpec, TypeAlias, TypeVar, assert_never, cast, final + +from anonymizer.engine.execution.accounting_evidence import ( + _AttemptId, + _Dispatch, + _FailureRecord, + _InvocationId, + _RowToken, + _SuccessRecord, + _TerminalRecord, +) +from anonymizer.engine.execution.accounting_outcomes import ( + _AccountingResult, + _CauseCode, + _CauseSet, + _DatumBlocked, + _DatumCancelled, + _DatumFailed, + _DatumInconsistent, + _DatumLost, + _DatumOutcome, + _DatumQualified, + _DependencySatisfied, + _DependencyUnsatisfied, + _GroupOutcome, + _GroupReleased, + _GroupWithheld, + _InvocationCancelled, + _InvocationCompleted, + _InvocationFailed, + _InvocationInconsistent, + _InvocationLost, + _InvocationOutcome, + _StageBlocked, + _StageCancelled, + _StageFailed, + _StageInconsistent, + _StageLost, + _StageOutcome, + _StageSucceeded, + _TaskBlocked, + _TaskCancelled, + _TaskFailed, + _TaskInconsistent, + _TaskLost, + _TaskOutcome, + _TaskSucceeded, + _TerminalCause, +) +from anonymizer.engine.execution.accounting_plan import ( + _AccountingPlan, + _AtomicGroupKey, + _DatumTaskSubject, + _ScopeTaskSubject, + _StageId, + _TaskKey, +) +from anonymizer.engine.execution.accounting_release import _qualify_release +from anonymizer.engine.execution.graph import _DatumId + +T = TypeVar("T") +P = ParamSpec("P") +R = TypeVar("R") + + +def _serialized( + method: Callable[Concatenate[_AccountingLedger[T], P], R], +) -> Callable[Concatenate[_AccountingLedger[T], P], R]: + @wraps(method) + def wrapped(ledger: _AccountingLedger[T], /, *args: P.args, **kwargs: P.kwargs) -> R: + with ledger._lock: + return method(ledger, *args, **kwargs) + + return wrapped + + +class _LedgerStateError(RuntimeError): + def __init__(self) -> None: + super().__init__("private accounting ledger state violation") + + def __repr__(self) -> str: + return "" + + +class _EvidenceAcceptance(str, Enum): + ACCEPTED = "accepted" + IDEMPOTENT_STALE = "idempotent_stale" + REJECTED_STALE = "rejected_stale" + + +class _ResultConstructionFailure(Exception): + pass + + +@final +@dataclass(frozen=True, slots=True) +class _Planned: + task: _TaskKey + + +@final +@dataclass(frozen=True, slots=True) +class _Ready: + task: _TaskKey + + +@final +@dataclass(frozen=True, slots=True) +class _Dispatched: + dispatch: _Dispatch + + +_TaskState: TypeAlias = _Planned | _Ready | _Dispatched | _TaskOutcome[T] + + +def _default_identity() -> str: + return secrets.token_hex(16) + + +class _AccountingLedger(Generic[T]): + """Identity-bearing one-shot shell around pure accounting reducers.""" + + def __init__( + self, + plan: _AccountingPlan, + *, + identity_factory: Callable[[], str] = _default_identity, + datum_release_predicate: Callable[[_DatumId, T], bool] = lambda _datum_id, _candidate: True, + ) -> None: + self._plan = plan + self._lock = RLock() + self._identity_factory = identity_factory + self._used_identities: set[str] = set() + self._datum_release_predicate = datum_release_predicate + self._datum_qualification: dict[_DatumId, bool] = {} + self._invocation_id: _InvocationId | None = None + self._states: dict[_TaskKey, _TaskState[T]] = {task: _Planned(task) for task in plan.tasks} + self._accepted_records: dict[_AttemptId, _TerminalRecord[T]] = {} + self._opened = False + self._closed = False + self._mutation_sealed = False + self._cancellation_requested = False + self._global_inconsistent = False + self._invocation_lost = False + self._cleanup_failed = False + self._cleanup_unconfirmed = False + + @_serialized + def open(self) -> None: + if self._opened or self._closed: + raise _LedgerStateError + self._invocation_id = _InvocationId(self._next_identity()) + self._opened = True + + @_serialized + def import_terminal_outcomes(self, outcomes: tuple[object, ...]) -> None: + """Import an exact earlier-phase terminal prefix into an expanded plan. + + The compiler may append a later-phase scope task to an already closed + plan. Re-reducing that plan must retain the original terminal records, + rather than manufacturing new prerequisite failures for them. + """ + self._require_active() + if not isinstance(outcomes, tuple) or not all( + isinstance( + outcome, + (_TaskSucceeded, _TaskFailed, _TaskCancelled, _TaskLost, _TaskBlocked, _TaskInconsistent), + ) + for outcome in outcomes + ): + raise _LedgerStateError + typed_outcomes = tuple(cast(_TaskOutcome[T], outcome) for outcome in outcomes) + if len({outcome.task for outcome in typed_outcomes}) != len(typed_outcomes) or any( + outcome.task not in self._states or not isinstance(self._states[outcome.task], _Planned) + for outcome in typed_outcomes + ): + raise _LedgerStateError + for outcome in typed_outcomes: + self._states[outcome.task] = outcome + + @_serialized + def ready_tasks(self) -> tuple[_TaskKey, ...]: + self._require_active() + self._advance_planned() + return tuple(task for task in self._plan.tasks if isinstance(self._states[task], _Ready)) + + @_serialized + def dispatch(self, task: _TaskKey, *, row_token_value: str | None = None) -> _Dispatch: + self._require_active() + self._advance_planned() + if not isinstance(self._states.get(task), _Ready) or self._invocation_id is None: + raise _LedgerStateError + row_token = self._next_identity() if row_token_value is None else self._claim_identity(row_token_value) + dispatch = _Dispatch( + self._invocation_id, + task, + _AttemptId(self._next_identity()), + _RowToken(row_token), + ) + self._states[task] = _Dispatched(dispatch) + return dispatch + + @_serialized + def dispatch_batch( + self, + tasks: tuple[_TaskKey, ...], + *, + row_token_values: tuple[str, ...], + ) -> tuple[_Dispatch, ...]: + """Atomically commit one context frontier after workframe construction.""" + self._require_active() + self._advance_planned() + if ( + self._invocation_id is None + or len(tasks) != len(row_token_values) + or len(set(tasks)) != len(tasks) + or not all(isinstance(self._states.get(task), _Ready) for task in tasks) + ): + raise _LedgerStateError + row_tokens = tuple(self._claim_identity(value) for value in row_token_values) + dispatches = tuple( + _Dispatch(self._invocation_id, task, _AttemptId(self._next_identity()), _RowToken(row_token)) + for task, row_token in zip(tasks, row_tokens, strict=True) + ) + for dispatch in dispatches: + self._states[dispatch.task] = _Dispatched(dispatch) + return dispatches + + @_serialized + def accept_success(self, dispatch: _Dispatch, candidate: T) -> _EvidenceAcceptance: + return self._accept(_SuccessRecord(dispatch, candidate)) + + @_serialized + def accept_failure(self, dispatch: _Dispatch) -> _EvidenceAcceptance: + return self._accept(_FailureRecord(dispatch)) + + @_serialized + def reconcile( + self, + dispatches: tuple[_Dispatch, ...], + records: tuple[_TerminalRecord[T], ...], + *, + trusted_run_record: bool, + ) -> None: + self._require_active() + if not trusted_run_record: + self._invocation_lost = True + for dispatch in dispatches: + self._close_dispatch(dispatch, _TaskLost(dispatch.task, _causes(_CauseCode.TRANSPORT_LOST))) + return + expected = {dispatch.attempt_id: dispatch for dispatch in dispatches} + observed_attempts = tuple(record.dispatch.attempt_id for record in records) + if len(expected) != len(dispatches) or len(set(observed_attempts)) != len(observed_attempts): + self._close_globally_inconsistent(_CauseCode.DUPLICATE) + return + fault = next( + ( + code + for record in records + if ( + code := self._reconciliation_fault( + record.dispatch, + expected.get(record.dispatch.attempt_id), + dispatches, + ) + ) + is not None + ), + None, + ) + if fault is not None: + self._close_globally_inconsistent(fault) + return + for record in records: + self._accept(record) + for attempt_id in expected.keys() - set(observed_attempts): + dispatch = expected[attempt_id] + self._close_dispatch(dispatch, _TaskInconsistent(dispatch.task, _causes(_CauseCode.MISSING))) + + @_serialized + def request_cancellation(self) -> None: + if self._closed: + return + self._require_opened() + self._cancellation_requested = True + cause = _causes(_CauseCode.CANCELLATION) + self._states = { + task: _TaskCancelled(task, cause) if isinstance(state, (_Planned, _Ready)) else state + for task, state in self._states.items() + } + + @_serialized + def acknowledge_stop(self, dispatch: _Dispatch) -> _EvidenceAcceptance: + self._require_opened() + if self._closed: + return _EvidenceAcceptance.REJECTED_STALE + state = self._states.get(dispatch.task) + if isinstance(state, _Dispatched) and state.dispatch == dispatch: + self._states[dispatch.task] = _TaskCancelled( + dispatch.task, + _causes(_CauseCode.CANCELLATION, _CauseCode.STOP_ACKNOWLEDGED), + ) + return _EvidenceAcceptance.ACCEPTED + return _EvidenceAcceptance.REJECTED_STALE + + @_serialized + def mark_transport_lost(self, dispatch: _Dispatch) -> _EvidenceAcceptance: + self._require_opened() + if self._closed: + return _EvidenceAcceptance.REJECTED_STALE + state = self._states.get(dispatch.task) + if isinstance(state, _Dispatched) and state.dispatch == dispatch: + self._invocation_lost = True + self._states[dispatch.task] = _TaskLost(dispatch.task, _causes(_CauseCode.TRANSPORT_LOST)) + return _EvidenceAcceptance.ACCEPTED + return _EvidenceAcceptance.REJECTED_STALE + + @_serialized + def mark_inconsistent(self, code: _CauseCode) -> None: + self._require_active() + if code not in { + _CauseCode.DUPLICATE, + _CauseCode.UNKNOWN, + _CauseCode.FOREIGN, + _CauseCode.STALE, + _CauseCode.SWAPPED, + _CauseCode.CONTRADICTORY, + _CauseCode.PLAN_MISMATCH, + }: + raise _LedgerStateError + self._close_globally_inconsistent(code) + + @_serialized + def mark_task_inconsistent(self, task: _TaskKey, code: _CauseCode) -> None: + """Close one attributable task without widening context-derived dependencies.""" + self._require_active() + if code not in {_CauseCode.MISSING, _CauseCode.DUPLICATE, _CauseCode.CONTRADICTORY}: + raise _LedgerStateError + state = self._states.get(task) + if state is None: + self._close_globally_inconsistent(_CauseCode.PLAN_MISMATCH) + elif not _is_terminal(state): + self._states[task] = _TaskInconsistent(task, _causes(code)) + + @_serialized + def mark_task_failed(self, task: _TaskKey) -> None: + """Close one known pre-dispatch construction failure locally.""" + self._require_active() + state = self._states.get(task) + if state is None: + self._close_globally_inconsistent(_CauseCode.PLAN_MISMATCH) + elif isinstance(state, (_Planned, _Ready)): + self._states[task] = _TaskFailed(task, _causes(_CauseCode.KNOWN_FAILURE)) + + @_serialized + def mark_task_succeeded(self, task: _TaskKey, candidate: T) -> None: + """Close verified no-work without manufacturing a dispatch attempt.""" + self._require_active() + state = self._states.get(task) + if state is None: + self._close_globally_inconsistent(_CauseCode.PLAN_MISMATCH) + elif isinstance(state, (_Planned, _Ready)): + self._states[task] = _TaskSucceeded(task, candidate) + + @_serialized + def mark_task_blocked(self, task: _TaskKey) -> None: + """Close a known non-dispatch prerequisite gate without an attempt.""" + self._require_active() + state = self._states.get(task) + if state is None: + self._close_globally_inconsistent(_CauseCode.PLAN_MISMATCH) + elif isinstance(state, (_Planned, _Ready)): + self._states[task] = _TaskBlocked(task, _causes(_CauseCode.PREREQUISITE)) + + @_serialized + def mark_cleanup_failed(self) -> None: + self._require_active() + self._cleanup_failed = True + + @_serialized + def mark_cleanup_unconfirmed(self) -> None: + self._require_active() + self._cleanup_unconfirmed = True + + @_serialized + def seal_mutation(self) -> None: + """Freeze external lifecycle transitions before cleanup attestation.""" + self._require_active() + self._mutation_sealed = True + + @_serialized + def record_cleanup_unconfirmed_after_seal(self) -> None: + """Record failed publication-critical cleanup without reopening tasks.""" + self._require_opened() + if self._closed or not self._mutation_sealed: + raise _LedgerStateError + self._cleanup_unconfirmed = True + + @_serialized + def finish( + self, + *, + datum_release_predicate: Callable[[_DatumId, T], bool] | None = None, + group_release_predicate: Callable[[tuple[tuple[_DatumId, T], ...]], bool] = lambda _outputs: True, + ) -> _AccountingResult[T]: + self._require_opened() + if self._closed: + raise _LedgerClosedError + if datum_release_predicate is not None: + if self._datum_qualification: + raise _LedgerStateError + self._datum_release_predicate = datum_release_predicate + self._advance_planned() + self._close_unfinished() + self._advance_planned() + tasks = tuple(self._terminal_state(task) for task in self._plan.tasks) + try: + result = _reduce_result( + self._plan, + tasks, + datum_release_predicate=self._qualifies, + group_release_predicate=group_release_predicate, + cancellation_requested=self._cancellation_requested, + global_inconsistent=self._global_inconsistent, + invocation_lost=self._invocation_lost, + cleanup_failed=self._cleanup_failed, + cleanup_unconfirmed=self._cleanup_unconfirmed, + ) + except Exception: + result = _construction_failed_result(self._plan, tasks) + self._closed = True + return result + + def _accept(self, record: _TerminalRecord[T]) -> _EvidenceAcceptance: + self._require_opened() + if self._closed or self._mutation_sealed: + return _EvidenceAcceptance.REJECTED_STALE + dispatch = record.dispatch + state = self._states.get(dispatch.task) + accepted = self._accepted_records.get(dispatch.attempt_id) + if accepted is not None: + try: + identical = accepted == record + except Exception: + identical = False + return _EvidenceAcceptance.IDEMPOTENT_STALE if identical else _EvidenceAcceptance.REJECTED_STALE + if not isinstance(state, _Dispatched) or state.dispatch != dispatch: + if state is None or not _is_terminal(state): + expected = state.dispatch if isinstance(state, _Dispatched) else None + fault = self._reconciliation_fault(dispatch, expected, (expected,) if expected is not None else ()) + self._close_globally_inconsistent(fault or _CauseCode.CONTRADICTORY) + return _EvidenceAcceptance.REJECTED_STALE + self._accepted_records[dispatch.attempt_id] = record + match record: + case _SuccessRecord(candidate=candidate): + self._states[dispatch.task] = _TaskSucceeded(dispatch.task, candidate) + case _FailureRecord(): + self._states[dispatch.task] = _TaskFailed(dispatch.task, _causes(_CauseCode.KNOWN_FAILURE)) + case unreachable: + assert_never(unreachable) + return _EvidenceAcceptance.ACCEPTED + + def _reconciliation_fault( + self, + observed: _Dispatch, + expected: _Dispatch | None, + batch: tuple[_Dispatch, ...], + ) -> _CauseCode | None: + if observed.task not in self._states: + return _CauseCode.PLAN_MISMATCH + if observed.invocation_id != self._invocation_id: + return _CauseCode.FOREIGN + if expected is None: + if any(observed == state.dispatch for state in self._states.values() if isinstance(state, _Dispatched)): + return _CauseCode.CONTRADICTORY + if any(observed.task == dispatch.task and observed.row_token == dispatch.row_token for dispatch in batch): + return _CauseCode.STALE + return _CauseCode.UNKNOWN + accepted = self._accepted_records.get(expected.attempt_id) + state = self._states.get(expected.task) + if accepted is not None or (state is not None and _is_terminal(state)): + return None + if observed == expected and isinstance(state, _Dispatched): + return None + if any( + observed.task == dispatch.task + and observed.row_token == dispatch.row_token + and dispatch.attempt_id != expected.attempt_id + for dispatch in batch + ): + return _CauseCode.SWAPPED + if observed.attempt_id != expected.attempt_id: + return ( + _CauseCode.STALE + if observed.task == expected.task and observed.row_token == expected.row_token + else _CauseCode.UNKNOWN + ) + if observed.row_token != expected.row_token: + return _CauseCode.FOREIGN + return _CauseCode.SWAPPED if observed.task != expected.task else _CauseCode.CONTRADICTORY + + def _close_dispatch(self, dispatch: _Dispatch, outcome: _TaskOutcome[T]) -> None: + state = self._states.get(dispatch.task) + if isinstance(state, _Dispatched) and state.dispatch == dispatch: + self._states[dispatch.task] = outcome + else: + self._close_globally_inconsistent(_CauseCode.STALE) + + def _close_globally_inconsistent(self, code: _CauseCode) -> None: + self._global_inconsistent = True + causes = _causes(code) + self._states = { + task: state if _is_terminal(state) else _TaskInconsistent(task, causes) + for task, state in self._states.items() + } + + def _advance_planned(self) -> None: + changed = True + while changed: + changed = False + for task in self._plan.tasks: + state = self._states[task] + if not isinstance(state, _Planned): + continue + guard = self._readiness(task) + if guard == "ready": + self._states[task] = _Ready(task) + changed = True + elif guard == "blocked": + self._states[task] = _TaskBlocked(task, _causes(_CauseCode.PREREQUISITE)) + changed = True + + def _readiness(self, task: _TaskKey) -> str: + if isinstance(task.subject, _DatumTaskSubject): + stage_index = self._plan.stages.index(task.stage) + if stage_index: + previous = _TaskKey(self._plan.stages[stage_index - 1], task.subject) + previous_state = self._states[previous] + if isinstance(previous_state, _TaskSucceeded): + pass + elif _is_terminal(previous_state): + return "blocked" + else: + return "waiting" + explicit_states = tuple( + self._states[predecessor.prerequisite] + for predecessor in self._plan.task_predecessors + if predecessor.dependent == task + ) + if any(_is_terminal(state) and not isinstance(state, _TaskSucceeded) for state in explicit_states): + return "blocked" + if any(not isinstance(state, _TaskSucceeded) for state in explicit_states): + return "waiting" + if isinstance(task.subject, _ScopeTaskSubject): + return "ready" + prerequisites = tuple( + dependency.prerequisite + for dependency in self._plan.dependencies + if dependency.dependent == task.subject.datum_id + ) + prerequisite_states = tuple(self._datum_execution_state(datum_id) for datum_id in prerequisites) + if any(state == "unsatisfied" for state in prerequisite_states): + return "blocked" + return "ready" if all(state == "satisfied" for state in prerequisite_states) else "waiting" + + def _datum_execution_state(self, datum_id: _DatumId) -> str: + subject = _DatumTaskSubject(datum_id) + states = tuple(self._states[_TaskKey(stage, subject)] for stage in self._plan.stages) + if all(isinstance(state, _TaskSucceeded) for state in states): + final_state = self._states[_TaskKey(self._plan.stages[-1], subject)] + if not isinstance(final_state, _TaskSucceeded): + raise _LedgerStateError + try: + return "satisfied" if self._qualifies(datum_id, final_state.candidate) else "unsatisfied" + except _ResultConstructionFailure: + return "unsatisfied" + if any(_is_terminal(state) and not isinstance(state, _TaskSucceeded) for state in states): + return "unsatisfied" + return "waiting" + + def _qualifies(self, datum_id: _DatumId, candidate: T) -> bool: + if datum_id in self._datum_qualification: + return self._datum_qualification[datum_id] + try: + qualified = self._datum_release_predicate(datum_id, candidate) + except Exception as cause: + del cause + raise _ResultConstructionFailure from None + if type(qualified) is not bool: + raise _ResultConstructionFailure + self._datum_qualification[datum_id] = qualified + return qualified + + def _close_unfinished(self) -> None: + for task, state in tuple(self._states.items()): + if isinstance(state, _Dispatched): + self._invocation_lost = True + causes = _causes( + *( + (_CauseCode.CANCELLATION, _CauseCode.TRANSPORT_LOST) + if self._cancellation_requested + else (_CauseCode.TRANSPORT_LOST,) + ) + ) + self._states[task] = _TaskLost(task, causes) + elif isinstance(state, (_Planned, _Ready)): + self._states[task] = _TaskBlocked(task, _causes(_CauseCode.PREREQUISITE)) + + def _terminal_state(self, task: _TaskKey) -> _TaskOutcome[T]: + state = self._states[task] + match state: + case ( + _TaskSucceeded() | _TaskFailed() | _TaskCancelled() | _TaskLost() | _TaskBlocked() | _TaskInconsistent() + ): + return state + case _Planned() | _Ready() | _Dispatched(): + raise _LedgerStateError + case unreachable: + assert_never(unreachable) + + def _next_identity(self) -> str: + value = self._identity_factory() + return self._claim_identity(value) + + def _claim_identity(self, value: object) -> str: + if not isinstance(value, str) or not value or value in self._used_identities: + raise _LedgerStateError + self._used_identities.add(value) + return value + + def _require_opened(self) -> None: + if not self._opened: + raise _LedgerStateError + + def _require_active(self) -> None: + self._require_opened() + if self._closed or self._mutation_sealed: + raise _LedgerClosedError + + +class _LedgerClosedError(_LedgerStateError): + pass + + +def _is_terminal(state: _TaskState[T]) -> bool: + return isinstance(state, (_TaskSucceeded, _TaskFailed, _TaskCancelled, _TaskLost, _TaskBlocked, _TaskInconsistent)) + + +def _causes(*codes: _CauseCode) -> _CauseSet: + return _CauseSet(tuple(_TerminalCause(code) for code in codes)) + + +def _cause_union(outcomes: Iterable[object]) -> _CauseSet: + return reduce(operator.or_, map(_causes_of, outcomes), _CauseSet()) + + +def _causes_of(outcome: object) -> _CauseSet: + match outcome: + case ( + _TaskFailed(causes=causes) + | _TaskCancelled(causes=causes) + | _TaskLost(causes=causes) + | _TaskBlocked(causes=causes) + | _TaskInconsistent(causes=causes) + | _DatumFailed(causes=causes) + | _DatumCancelled(causes=causes) + | _DatumLost(causes=causes) + | _DatumBlocked(causes=causes) + | _DatumInconsistent(causes=causes) + | _DependencyUnsatisfied(causes=causes) + | _StageFailed(causes=causes) + | _StageCancelled(causes=causes) + | _StageLost(causes=causes) + | _StageBlocked(causes=causes) + | _StageInconsistent(causes=causes) + | _GroupWithheld(causes=causes) + ): + return causes + case _TaskSucceeded() | _DatumQualified() | _DependencySatisfied() | _StageSucceeded() | _GroupReleased(): + return _CauseSet() + case _: + raise _LedgerStateError + + +def _reduce_result( + plan: _AccountingPlan, + tasks: tuple[_TaskOutcome[T], ...], + *, + datum_release_predicate: Callable[[_DatumId, T], bool], + group_release_predicate: Callable[[tuple[tuple[_DatumId, T], ...]], bool], + cancellation_requested: bool, + global_inconsistent: bool, + invocation_lost: bool, + cleanup_failed: bool, + cleanup_unconfirmed: bool, +) -> _AccountingResult[T]: + datums = tuple(_reduce_datum(plan, datum.id, tasks, datum_release_predicate) for datum in plan.datums) + datum_by_id = {outcome.datum_id: outcome for outcome in datums} + dependencies = tuple( + _DependencySatisfied(dependency) + if isinstance(datum_by_id[dependency.prerequisite], _DatumQualified) + else _DependencyUnsatisfied( + dependency, + _cause_union((datum_by_id[dependency.prerequisite],)) | _causes(_CauseCode.PREREQUISITE), + ) + for dependency in plan.dependencies + ) + stages = tuple(_reduce_stage(stage, tasks) for stage in plan.stages) + groups = _reduce_groups( + plan, + datum_by_id, + group_release_predicate, + cancellation_requested=cancellation_requested, + global_inconsistent=global_inconsistent, + invocation_lost=invocation_lost, + cleanup_failed=cleanup_failed, + cleanup_unconfirmed=cleanup_unconfirmed, + ) + all_causes = _cause_union((*tasks, *datums, *dependencies, *stages, *groups)) + invocation = _reduce_invocation( + groups, + all_causes, + cancellation_requested=cancellation_requested, + global_inconsistent=global_inconsistent, + invocation_lost=invocation_lost, + cleanup_failed=cleanup_failed, + cleanup_unconfirmed=cleanup_unconfirmed, + ) + return _AccountingResult(tasks, datums, dependencies, stages, groups, invocation) + + +def _reduce_groups( + plan: _AccountingPlan, + datum_by_id: dict[_DatumId, _DatumOutcome[T]], + group_release_predicate: Callable[[tuple[tuple[_DatumId, T], ...]], bool], + *, + cancellation_requested: bool, + global_inconsistent: bool, + invocation_lost: bool, + cleanup_failed: bool, + cleanup_unconfirmed: bool, +) -> tuple[_GroupOutcome[T], ...]: + qualified = frozenset(outcome.datum_id for outcome in datum_by_id.values() if isinstance(outcome, _DatumQualified)) + embargoed = ( + global_inconsistent or invocation_lost or cancellation_requested or cleanup_failed or cleanup_unconfirmed + ) + predicate_failed_groups: frozenset[_AtomicGroupKey] = frozenset() + if embargoed: + qualified = frozenset() + else: + predicate_failed_groups = _failed_group_predicates( + plan, + datum_by_id, + qualified, + group_release_predicate, + ) + qualified -= frozenset( + member for group in plan.atomic_groups if group.key in predicate_failed_groups for member in group.members + ) + decision = _qualify_release(plan, qualified) + groups = tuple( + _reduce_group(plan, group.key, datum_by_id, decision.released_groups, predicate_failed_groups) + for group in plan.atomic_groups + ) + cleanup_code = ( + _CauseCode.CLEANUP_UNCONFIRMED if cleanup_unconfirmed else _CauseCode.CLEANUP_FAILED if cleanup_failed else None + ) + if cleanup_code is not None: + groups = tuple(_GroupWithheld(group.key, _causes(cleanup_code)) for group in plan.atomic_groups) + return groups + + +def _reduce_invocation( + groups: tuple[_GroupOutcome[T], ...], + all_causes: _CauseSet, + *, + cancellation_requested: bool, + global_inconsistent: bool, + invocation_lost: bool, + cleanup_failed: bool, + cleanup_unconfirmed: bool, +) -> _InvocationOutcome[T]: + if cleanup_unconfirmed: + return _InvocationInconsistent(all_causes | _causes(_CauseCode.CLEANUP_UNCONFIRMED)) + if cleanup_failed: + return _InvocationFailed(all_causes | _causes(_CauseCode.CLEANUP_FAILED)) + if global_inconsistent: + return _InvocationInconsistent(all_causes | _causes(_CauseCode.CONTRADICTORY)) + if invocation_lost: + return _InvocationLost(all_causes | _causes(_CauseCode.TRANSPORT_LOST)) + if cancellation_requested: + return _InvocationCancelled(all_causes | _causes(_CauseCode.CANCELLATION)) + return _InvocationCompleted(groups) + + +def _construction_failed_result( + plan: _AccountingPlan, + tasks: tuple[_TaskOutcome[T], ...], +) -> _AccountingResult[T]: + datums = tuple(_reduce_datum(plan, datum.id, tasks, lambda _datum_id, _candidate: True) for datum in plan.datums) + datum_by_id = {outcome.datum_id: outcome for outcome in datums} + dependencies = tuple( + _DependencySatisfied(dependency) + if isinstance(datum_by_id[dependency.prerequisite], _DatumQualified) + else _DependencyUnsatisfied( + dependency, + _cause_union((datum_by_id[dependency.prerequisite],)) | _causes(_CauseCode.PREREQUISITE), + ) + for dependency in plan.dependencies + ) + stages = tuple(_reduce_stage(stage, tasks) for stage in plan.stages) + causes = _causes(_CauseCode.RESULT_CONSTRUCTION_FAILED) + groups = tuple(_GroupWithheld(group.key, causes) for group in plan.atomic_groups) + return _AccountingResult(tasks, datums, dependencies, stages, groups, _InvocationFailed(causes)) + + +def _reduce_datum( + plan: _AccountingPlan, + datum_id: _DatumId, + tasks: tuple[_TaskOutcome[T], ...], + release_predicate: Callable[[_DatumId, T], bool], +) -> _DatumOutcome[T]: + child_tasks = tuple( + outcome + for outcome in tasks + if isinstance(outcome.task.subject, _DatumTaskSubject) and outcome.task.subject.datum_id == datum_id + ) + if all(isinstance(outcome, _TaskSucceeded) for outcome in child_tasks): + final_task = next( + outcome + for outcome in child_tasks + if outcome.task.stage == plan.stages[-1] and isinstance(outcome, _TaskSucceeded) + ) + return ( + _DatumQualified(datum_id, final_task.candidate) + if release_predicate(datum_id, final_task.candidate) + else _DatumFailed(datum_id, _causes(_CauseCode.RELEASE_PREDICATE_FAILED)) + ) + causes = _cause_union(child_tasks) + if any(isinstance(outcome, _TaskInconsistent) for outcome in child_tasks): + return _DatumInconsistent(datum_id, causes) + if any(isinstance(outcome, _TaskLost) for outcome in child_tasks): + return _DatumLost(datum_id, causes) + if any(isinstance(outcome, _TaskCancelled) for outcome in child_tasks): + return _DatumCancelled(datum_id, causes) + if any(isinstance(outcome, _TaskFailed) for outcome in child_tasks): + return _DatumFailed(datum_id, causes) + if any(isinstance(outcome, _TaskBlocked) for outcome in child_tasks): + return _DatumBlocked(datum_id, causes) + raise _LedgerStateError + + +def _reduce_stage(stage: _StageId, tasks: tuple[_TaskOutcome[T], ...]) -> _StageOutcome: + children = tuple(outcome for outcome in tasks if outcome.task.stage == stage) + if all(isinstance(outcome, _TaskSucceeded) for outcome in children): + return _StageSucceeded(stage) + causes = _cause_union(children) + if any(isinstance(outcome, _TaskInconsistent) for outcome in children): + return _StageInconsistent(stage, causes) + if any(isinstance(outcome, _TaskLost) for outcome in children): + return _StageLost(stage, causes) + if any(isinstance(outcome, _TaskCancelled) for outcome in children): + return _StageCancelled(stage, causes) + if any(isinstance(outcome, _TaskFailed) for outcome in children): + return _StageFailed(stage, causes) + if any(isinstance(outcome, _TaskBlocked) for outcome in children): + return _StageBlocked(stage, causes) + raise _LedgerStateError + + +def _reduce_group( + plan: _AccountingPlan, + group_key: _AtomicGroupKey, + datum_by_id: dict[_DatumId, _DatumOutcome[T]], + released_groups: frozenset[_AtomicGroupKey], + predicate_failed_groups: frozenset[_AtomicGroupKey], +) -> _GroupOutcome[T]: + group = next(group for group in plan.atomic_groups if group.key == group_key) + member_outcomes = tuple(datum_by_id[datum.id] for datum in plan.datums if datum.id in group.members) + if group.key in released_groups: + outputs = tuple( + (outcome.datum_id, outcome.candidate) for outcome in member_outcomes if isinstance(outcome, _DatumQualified) + ) + if len(outputs) != len(group.members): + return _GroupWithheld(group.key, _causes(_CauseCode.RELEASE_PREDICATE_FAILED)) + return _GroupReleased(group.key, outputs) + if group.key in predicate_failed_groups: + return _GroupWithheld(group.key, _causes(_CauseCode.RELEASE_PREDICATE_FAILED)) + return _GroupWithheld(group.key, _cause_union(member_outcomes) | _causes(_CauseCode.PREREQUISITE)) + + +def _failed_group_predicates( + plan: _AccountingPlan, + datum_by_id: dict[_DatumId, _DatumOutcome[T]], + qualified: frozenset[_DatumId], + release_predicate: Callable[[tuple[tuple[_DatumId, T], ...]], bool], +) -> frozenset[_AtomicGroupKey]: + """Evaluate complete groups before dependency propagation can release dependents.""" + failed: set[_AtomicGroupKey] = set() + for group in plan.atomic_groups: + if not frozenset(group.members).issubset(qualified): + continue + outputs = tuple( + (datum_id, outcome.candidate) + for datum_id in group.members + if isinstance((outcome := datum_by_id[datum_id]), _DatumQualified) + ) + if len(outputs) != len(group.members): + raise _ResultConstructionFailure + passed = release_predicate(outputs) + if type(passed) is not bool: + raise _ResultConstructionFailure + if not passed: + failed.add(group.key) + return frozenset(failed) diff --git a/src/anonymizer/engine/execution/accounting_outcomes.py b/src/anonymizer/engine/execution/accounting_outcomes.py new file mode 100644 index 00000000..1672abbc --- /dev/null +++ b/src/anonymizer/engine/execution/accounting_outcomes.py @@ -0,0 +1,302 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Closed terminal outcome algebra for phase-4 accounting.""" + +from __future__ import annotations + +from collections.abc import Iterator +from dataclasses import dataclass +from enum import Enum +from typing import Generic, TypeAlias, TypeVar, final + +from anonymizer.engine.execution.accounting_plan import ( + _AtomicGroupKey, + _CompiledDependency, + _StageId, + _TaskKey, +) +from anonymizer.engine.execution.graph import _DatumId + +T = TypeVar("T") + + +class _PrivateTerminalValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private terminal accounting values are not serializable") + + +class _CauseCode(str, Enum): + KNOWN_FAILURE = "known_failure" + VERIFICATION_FAILED = "verification_failed" + RELEASE_PREDICATE_FAILED = "release_predicate_failed" + CANCELLATION = "cancellation" + STOP_ACKNOWLEDGED = "stop_acknowledged" + TRANSPORT_LOST = "transport_lost" + MISSING = "missing" + DUPLICATE = "duplicate" + UNKNOWN = "unknown" + FOREIGN = "foreign" + STALE = "stale" + SWAPPED = "swapped" + CONTRADICTORY = "contradictory" + PLAN_MISMATCH = "plan_mismatch" + PREREQUISITE = "prerequisite" + RESULT_CONSTRUCTION_FAILED = "result_construction_failed" + CLEANUP_FAILED = "cleanup_failed" + CLEANUP_UNCONFIRMED = "cleanup_unconfirmed" + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _TerminalCause(_PrivateTerminalValue): + code: _CauseCode + + +_CAUSE_PRECEDENCE = {code: ordinal for ordinal, code in enumerate(_CauseCode)} + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _CauseSet(_PrivateTerminalValue): + items: tuple[_TerminalCause, ...] = () + + def __post_init__(self) -> None: + canonical = tuple( + _TerminalCause(code) + for code in sorted({cause.code for cause in self.items}, key=_CAUSE_PRECEDENCE.__getitem__) + ) + object.__setattr__(self, "items", canonical) + + def __or__(self, other: _CauseSet) -> _CauseSet: + if not isinstance(other, _CauseSet): + return NotImplemented + return _CauseSet((*self.items, *other.items)) + + def __iter__(self) -> Iterator[_TerminalCause]: + return iter(self.items) + + def __bool__(self) -> bool: + return bool(self.items) + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _TaskSucceeded(_PrivateTerminalValue, Generic[T]): + task: _TaskKey + candidate: T + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _TaskFailed(_PrivateTerminalValue): + task: _TaskKey + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _TaskCancelled(_PrivateTerminalValue): + task: _TaskKey + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _TaskLost(_PrivateTerminalValue): + task: _TaskKey + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _TaskBlocked(_PrivateTerminalValue): + task: _TaskKey + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _TaskInconsistent(_PrivateTerminalValue): + task: _TaskKey + causes: _CauseSet + + +_TaskOutcome: TypeAlias = ( + _TaskSucceeded[T] | _TaskFailed | _TaskCancelled | _TaskLost | _TaskBlocked | _TaskInconsistent +) + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _DatumQualified(_PrivateTerminalValue, Generic[T]): + datum_id: _DatumId + candidate: T + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _DatumFailed(_PrivateTerminalValue): + datum_id: _DatumId + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _DatumCancelled(_PrivateTerminalValue): + datum_id: _DatumId + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _DatumLost(_PrivateTerminalValue): + datum_id: _DatumId + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _DatumBlocked(_PrivateTerminalValue): + datum_id: _DatumId + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _DatumInconsistent(_PrivateTerminalValue): + datum_id: _DatumId + causes: _CauseSet + + +_DatumOutcome: TypeAlias = ( + _DatumQualified[T] | _DatumFailed | _DatumCancelled | _DatumLost | _DatumBlocked | _DatumInconsistent +) + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _DependencySatisfied(_PrivateTerminalValue): + dependency: _CompiledDependency + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _DependencyUnsatisfied(_PrivateTerminalValue): + dependency: _CompiledDependency + causes: _CauseSet + + +_DependencyOutcome: TypeAlias = _DependencySatisfied | _DependencyUnsatisfied + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _StageSucceeded(_PrivateTerminalValue): + stage: _StageId + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _StageFailed(_PrivateTerminalValue): + stage: _StageId + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _StageCancelled(_PrivateTerminalValue): + stage: _StageId + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _StageLost(_PrivateTerminalValue): + stage: _StageId + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _StageBlocked(_PrivateTerminalValue): + stage: _StageId + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _StageInconsistent(_PrivateTerminalValue): + stage: _StageId + causes: _CauseSet + + +_StageOutcome: TypeAlias = ( + _StageSucceeded | _StageFailed | _StageCancelled | _StageLost | _StageBlocked | _StageInconsistent +) + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _GroupReleased(_PrivateTerminalValue, Generic[T]): + group: _AtomicGroupKey + outputs: tuple[tuple[_DatumId, T], ...] + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _GroupWithheld(_PrivateTerminalValue): + group: _AtomicGroupKey + causes: _CauseSet + + +_GroupOutcome: TypeAlias = _GroupReleased[T] | _GroupWithheld + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _InvocationCompleted(_PrivateTerminalValue, Generic[T]): + groups: tuple[_GroupOutcome[T], ...] + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _InvocationFailed(_PrivateTerminalValue): + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _InvocationCancelled(_PrivateTerminalValue): + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _InvocationLost(_PrivateTerminalValue): + causes: _CauseSet + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _InvocationInconsistent(_PrivateTerminalValue): + causes: _CauseSet + + +_InvocationOutcome: TypeAlias = ( + _InvocationCompleted[T] | _InvocationFailed | _InvocationCancelled | _InvocationLost | _InvocationInconsistent +) + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _AccountingResult(_PrivateTerminalValue, Generic[T]): + tasks: tuple[_TaskOutcome[T], ...] + datums: tuple[_DatumOutcome[T], ...] + dependencies: tuple[_DependencyOutcome, ...] + stages: tuple[_StageOutcome, ...] + groups: tuple[_GroupOutcome[T], ...] + invocation: _InvocationOutcome[T] diff --git a/src/anonymizer/engine/execution/accounting_plan.py b/src/anonymizer/engine/execution/accounting_plan.py new file mode 100644 index 00000000..e721c076 --- /dev/null +++ b/src/anonymizer/engine/execution/accounting_plan.py @@ -0,0 +1,287 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Immutable proof produced by phase-4 graph admission.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import TypeAlias, final + +from anonymizer.engine.execution.graph import _DatumId, _TextDatum + + +class _PrivateAccountingPlanValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private accounting plan values are not serializable") + + +@dataclass(frozen=True, slots=True, repr=False) +class _StageId(_PrivateAccountingPlanValue): + value: str + + +@final +@dataclass(frozen=True, slots=True, repr=False) +class _DatumTaskSubject(_PrivateAccountingPlanValue): + datum_id: _DatumId + + def __post_init__(self) -> None: + if not isinstance(self.datum_id, _DatumId): + raise TypeError("private datum task subject is malformed") + + +@final +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _ScopeTaskSubject(_PrivateAccountingPlanValue): + """Compiler-issued opaque capability for one scope-owned task.""" + + +_TaskSubject: TypeAlias = _DatumTaskSubject | _ScopeTaskSubject + + +@dataclass(frozen=True, slots=True, repr=False) +class _TaskKey(_PrivateAccountingPlanValue): + stage: _StageId + subject: _TaskSubject + + def __post_init__(self) -> None: + if not isinstance(self.stage, _StageId) or not isinstance(self.subject, (_DatumTaskSubject, _ScopeTaskSubject)): + raise TypeError("private task subject is malformed") + + +@dataclass(frozen=True, slots=True, repr=False) +class _TaskPredecessor(_PrivateAccountingPlanValue): + prerequisite: _TaskKey + dependent: _TaskKey + + +@dataclass(frozen=True, slots=True, repr=False) +class _CompiledDependency(_PrivateAccountingPlanValue): + prerequisite: _DatumId + dependent: _DatumId + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _AtomicGroupKey(_PrivateAccountingPlanValue): + """Compiler-created, graph-scoped capability for one atomic group.""" + + +@dataclass(frozen=True, slots=True, repr=False) +class _CompiledAtomicGroup(_PrivateAccountingPlanValue): + key: _AtomicGroupKey + members: tuple[_DatumId, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _AccountingPlanProof(_PrivateAccountingPlanValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +_ADMISSION_SEAL = object() + + +@dataclass(frozen=True, slots=True, repr=False) +class _AccountingPlan(_PrivateAccountingPlanValue): + datums: tuple[_TextDatum, ...] + stages: tuple[_StageId, ...] + tasks: tuple[_TaskKey, ...] + dependencies: tuple[_CompiledDependency, ...] + atomic_groups: tuple[_CompiledAtomicGroup, ...] + topological_datums: tuple[_DatumId, ...] + task_predecessors: tuple[_TaskPredecessor, ...] = () + _proof: _AccountingPlanProof | None = field(default=None, compare=False) + + def with_task_predecessors(self, predecessors: tuple[_TaskPredecessor, ...]) -> _AccountingPlan: + """Return a resealed plan with explicit compiler-owned task readiness.""" + if not _is_admitted_accounting_plan(self): + raise TypeError("private accounting plan is not admitted") + validated = _validate_task_predecessors(self, predecessors) + return _admit_accounting_plan( + self.datums, + self.stages, + self.tasks, + self.dependencies, + self.atomic_groups, + self.topological_datums, + validated, + ) + + def with_scope_tasks( + self, + stage: _StageId, + subjects: tuple[_ScopeTaskSubject, ...], + ) -> _AccountingPlan: + """Return a resealed plan with one task for each compiler-issued scope.""" + if not _is_admitted_accounting_plan(self): + raise TypeError("private accounting plan is not admitted") + existing_scope_subjects = frozenset( + task.subject for task in self.tasks if isinstance(task.subject, _ScopeTaskSubject) + ) + if ( + not isinstance(stage, _StageId) + or not isinstance(stage.value, str) + or not stage.value + or stage in self.stages + or not isinstance(subjects, tuple) + or not all(isinstance(subject, _ScopeTaskSubject) for subject in subjects) + or len(set(subjects)) != len(subjects) + or any(subject in existing_scope_subjects for subject in subjects) + ): + raise TypeError("private scope task subjects are malformed") + if not subjects: + return self + tasks = (*self.tasks, *(_TaskKey(stage, subject) for subject in subjects)) + return _admit_accounting_plan( + self.datums, + self.stages, + tasks, + self.dependencies, + self.atomic_groups, + self.topological_datums, + self.task_predecessors, + ) + + def with_datum_stage(self, stage: _StageId) -> _AccountingPlan: + """Append one compiler-owned task stage for every admitted datum.""" + if ( + not _is_admitted_accounting_plan(self) + or not isinstance(stage, _StageId) + or not isinstance(stage.value, str) + or not stage.value + or stage in self.stages + or any(task.stage == stage for task in self.tasks) + ): + raise TypeError("private datum stage is malformed") + tasks = ( + *self.tasks, + *(_TaskKey(stage, _DatumTaskSubject(datum.id)) for datum in self.datums), + ) + return _admit_accounting_plan( + self.datums, + (*self.stages, stage), + tasks, + self.dependencies, + self.atomic_groups, + self.topological_datums, + self.task_predecessors, + ) + + +def _admit_accounting_plan( + datums: tuple[_TextDatum, ...], + stages: tuple[_StageId, ...], + tasks: tuple[_TaskKey, ...], + dependencies: tuple[_CompiledDependency, ...], + atomic_groups: tuple[_CompiledAtomicGroup, ...], + topological_datums: tuple[_DatumId, ...], + task_predecessors: tuple[_TaskPredecessor, ...] = (), +) -> _AccountingPlan: + values = (datums, stages, tasks, dependencies, atomic_groups, topological_datums, task_predecessors) + plan = _AccountingPlan(*values) + snapshot = _plan_snapshot(plan) + if snapshot is None: + raise TypeError("private accounting plan admission failed") + return _AccountingPlan(*values, _AccountingPlanProof(_ADMISSION_SEAL, snapshot)) + + +def _is_admitted_accounting_plan(value: object) -> bool: + if not isinstance(value, _AccountingPlan) or value._proof is None: + return False + return value._proof.seal is _ADMISSION_SEAL and value._proof.snapshot == _plan_snapshot(value) + + +def _plan_snapshot(plan: _AccountingPlan) -> tuple[object, ...] | None: + """Detach admission proof from every nested mutable Python object.""" + try: + return ( + tuple((datum.id.value, datum.text, datum.purpose.value) for datum in plan.datums), + tuple(stage.value for stage in plan.stages), + tuple(_task_snapshot(task) for task in plan.tasks), + tuple((dependency.prerequisite.value, dependency.dependent.value) for dependency in plan.dependencies), + tuple((group.key, tuple(member.value for member in group.members)) for group in plan.atomic_groups), + tuple(datum_id.value for datum_id in plan.topological_datums), + tuple( + ( + *_task_snapshot(predecessor.prerequisite), + *_task_snapshot(predecessor.dependent), + ) + for predecessor in plan.task_predecessors + ), + ) + except (AttributeError, TypeError): + return None + + +def _validate_task_predecessors( + plan: _AccountingPlan, + predecessors: object, +) -> tuple[_TaskPredecessor, ...]: + if not isinstance(predecessors, tuple) or not all( + isinstance(predecessor, _TaskPredecessor) for predecessor in predecessors + ): + raise TypeError("private task predecessors are malformed") + task_set = frozenset(plan.tasks) + edges = tuple((predecessor.prerequisite, predecessor.dependent) for predecessor in predecessors) + implicit = frozenset( + ( + _TaskKey(plan.stages[stage_index - 1], task.subject), + task, + ) + for task in plan.tasks + if isinstance(task.subject, _DatumTaskSubject) and (stage_index := plan.stages.index(task.stage)) > 0 + ) + if ( + any(prerequisite not in task_set or dependent not in task_set for prerequisite, dependent in edges) + or any(prerequisite == dependent for prerequisite, dependent in edges) + or len(set(edges)) != len(edges) + or any(edge in implicit for edge in edges) + ): + raise TypeError("private task predecessors are malformed") + _validate_task_predecessor_dag(plan.tasks, (*implicit, *edges)) + return tuple(predecessors) + + +def _task_snapshot(task: _TaskKey) -> tuple[object, object]: + match task.subject: + case _DatumTaskSubject(datum_id=datum_id): + return task.stage.value, datum_id.value + case _ScopeTaskSubject(): + return task.stage.value, task.subject + + +def _validate_task_predecessor_dag( + tasks: tuple[_TaskKey, ...], + edges: tuple[tuple[_TaskKey, _TaskKey], ...], +) -> None: + incoming = {task: 0 for task in tasks} + dependents: dict[_TaskKey, list[_TaskKey]] = {task: [] for task in tasks} + for prerequisite, dependent in edges: + incoming[dependent] += 1 + dependents[prerequisite].append(dependent) + ready = [task for task in tasks if incoming[task] == 0] + visited = 0 + while ready: + task = ready.pop(0) + visited += 1 + for dependent in dependents[task]: + incoming[dependent] -= 1 + if incoming[dependent] == 0: + ready.append(dependent) + if visited != len(tasks): + raise TypeError("private task predecessors contain a cycle") + + +@dataclass(frozen=True, slots=True) +class _AccountingLimits: + max_datums: int + max_datum_bytes: int + max_graph_bytes: int + max_id_bytes: int = 256 + max_dependencies: int = 1_024 + max_atomic_groups: int = 1_024 + max_stages: int = 3 diff --git a/src/anonymizer/engine/execution/accounting_release.py b/src/anonymizer/engine/execution/accounting_release.py new file mode 100644 index 00000000..0640f4e7 --- /dev/null +++ b/src/anonymizer/engine/execution/accounting_release.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pure dependency and atomic-group release qualification.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from anonymizer.engine.execution.accounting_plan import _AccountingPlan, _AtomicGroupKey +from anonymizer.engine.execution.graph import _DatumId + + +@dataclass(frozen=True, slots=True, repr=False) +class _ReleaseDecision: + release_eligible: frozenset[_DatumId] + released_groups: frozenset[_AtomicGroupKey] + withheld_groups: frozenset[_AtomicGroupKey] + + def __repr__(self) -> str: + return "" + + +def _qualify_release(plan: _AccountingPlan, locally_qualified: frozenset[_DatumId]) -> _ReleaseDecision: + """Return the least fixed point of group and dependency withholding.""" + eligible = locally_qualified & frozenset(datum.id for datum in plan.datums) + while True: + propagated = _propagate_once(plan, eligible) + if propagated == eligible: + break + eligible = propagated + released = frozenset(group.key for group in plan.atomic_groups if frozenset(group.members).issubset(eligible)) + all_groups = frozenset(group.key for group in plan.atomic_groups) + return _ReleaseDecision(eligible, released, all_groups - released) + + +def _propagate_once(plan: _AccountingPlan, eligible: frozenset[_DatumId]) -> frozenset[_DatumId]: + group_withheld = frozenset( + member + for group in plan.atomic_groups + if not frozenset(group.members).issubset(eligible) + for member in group.members + ) + after_groups = eligible - group_withheld + dependency_withheld = frozenset( + dependency.dependent for dependency in plan.dependencies if dependency.prerequisite not in after_groups + ) + return after_groups - dependency_withheld diff --git a/src/anonymizer/engine/execution/context_admission.py b/src/anonymizer/engine/execution/context_admission.py new file mode 100644 index 00000000..6efe0419 --- /dev/null +++ b/src/anonymizer/engine/execution/context_admission.py @@ -0,0 +1,459 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pure admission boundary for private target and bounded-context plans.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum +from typing import TypeAlias, TypeGuard + +from anonymizer.engine.execution.accounting_admission import ( + _AccountingAdmissionCode, + _AccountingRejected, + _compile_accounting_plan, +) +from anonymizer.engine.execution.accounting_plan import ( + _AccountingLimits, + _AccountingPlan, + _DatumTaskSubject, + _is_admitted_accounting_plan, + _TaskKey, +) +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _capability_satisfies, + _ContextBackendCapability, + _ContextExecutionContract, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, + _valid_context_limits, +) +from anonymizer.engine.execution.graph import ( + _ContextScope, + _DatumId, + _DatumPurpose, + _ProtectionGraph, + _TextDatum, +) + + +class _PrivateContextPlanValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private context plan values are not serializable") + + +class _ContextAdmissionCode(str, Enum): + MALFORMED_GRAPH = "malformed_graph" + INVALID_DATUM_PURPOSE = "invalid_datum_purpose" + MISSING_CONTEXT_SCOPE = "missing_context_scope" + DUPLICATE_CONTEXT_SCOPE = "duplicate_context_scope" + UNKNOWN_CONTEXT_TARGET = "unknown_context_target" + UNKNOWN_CONTEXT_DATUM = "unknown_context_datum" + CONTEXT_ONLY_TARGET = "context_only_target" + ORPHAN_CONTEXT_DATUM = "orphan_context_datum" + SELF_CONTEXT = "self_context" + DUPLICATE_CONTEXT_MEMBER = "duplicate_context_member" + TARGET_CONTEXT_DISABLED = "target_context_disabled" + CONTEXT_MEMBERS_EXCEEDED = "context_members_exceeded" + CONTEXT_BYTES_EXCEEDED = "context_bytes_exceeded" + TOTAL_CONTEXT_REFERENCES_EXCEEDED = "total_context_references_exceeded" + EXPANDED_FRAME_BYTES_EXCEEDED = "expanded_frame_bytes_exceeded" + UNSUPPORTED_CONTEXT_CONTRACT = "unsupported_context_contract" + BACKEND_INCOMPATIBLE = "backend_incompatible" + + +_ContextRejectionCode: TypeAlias = _AccountingAdmissionCode | _ContextAdmissionCode + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextRejected(_PrivateContextPlanValue): + code: _ContextRejectionCode + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _ContextScopeKey(_PrivateContextPlanValue): + """Compiler-issued graph-scoped identity for one admitted context scope.""" + + +@dataclass(frozen=True, slots=True, repr=False) +class _CompiledContextBinding(_PrivateContextPlanValue): + owner_task: _TaskKey + scope: _ContextScopeKey + ordinal: int + datum_id: _DatumId + + +@dataclass(frozen=True, slots=True, repr=False) +class _CompiledContextProjection(_PrivateContextPlanValue): + owner_task: _TaskKey + target_datum_id: _DatumId + scope: _ContextScopeKey + context_datum_ids: tuple[_DatumId, ...] + bindings: tuple[_CompiledContextBinding, ...] + context_bytes: int + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextPlanProof(_PrivateContextPlanValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +_CONTEXT_ADMISSION_SEAL = object() + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextPlan(_PrivateContextPlanValue): + accounting: _AccountingPlan + context_only_datums: tuple[_TextDatum, ...] + projections: tuple[_CompiledContextProjection, ...] + contract: _ContextExecutionContract + preflight_capability: _ContextBackendCapability + _proof: _ContextPlanProof | None = field(default=None, compare=False) + + +_ContextAdmissionResult: TypeAlias = _ContextPlan | _ContextRejected + + +def _compile_context_plan( + graph: object, + *, + accounting_limits: _AccountingLimits, + contract: object, + capability: object, + stages: tuple[str, ...] = ("protect",), +) -> _ContextAdmissionResult: + """Compile one detached target/context projection before invocation effects.""" + rejected_or_datums = _compile_all_datums(graph, accounting_limits) + if isinstance(rejected_or_datums, _ContextRejected): + return rejected_or_datums + if not isinstance(graph, _ProtectionGraph): + return _ContextRejected(_ContextAdmissionCode.MALFORMED_GRAPH) + datums = rejected_or_datums + target_datums = tuple(datum for datum in datums if datum.purpose is _DatumPurpose.TARGET) + context_datums = tuple(datum for datum in datums if datum.purpose is _DatumPurpose.CONTEXT_ONLY) + if not target_datums: + return _ContextRejected(_AccountingAdmissionCode.MALFORMED_GRAPH) + target_ids = frozenset(datum.id for datum in target_datums) + projected = _ProtectionGraph( + datums=target_datums, + links=graph.links, + context_scopes=tuple(_ContextScope(datum.id) for datum in target_datums), + coherence_scopes=graph.coherence_scopes, + atomic_groups=graph.atomic_groups, + dependencies=graph.dependencies, + ) + accounting = _compile_accounting_plan(projected, limits=accounting_limits, stages=stages) + if isinstance(accounting, _AccountingRejected): + return _ContextRejected(accounting.code) + if not _valid_contract(contract): + return _ContextRejected(_ContextAdmissionCode.UNSUPPORTED_CONTEXT_CONTRACT) + scopes_or_rejected = _compile_scopes(graph.context_scopes, datums, target_ids, context_datums, contract) + if isinstance(scopes_or_rejected, _ContextRejected): + return scopes_or_rejected + scopes = scopes_or_rejected + limit_rejection = _check_context_limits(scopes, datums, target_datums, contract.limits) + if limit_rejection is not None: + return limit_rejection + if not isinstance(capability, _ContextBackendCapability) or not _capability_satisfies(capability, contract): + return _ContextRejected(_ContextAdmissionCode.BACKEND_INCOMPATIBLE) + projections = _materialize_projections(accounting, scopes, datums) + return _admit_context_plan(accounting, context_datums, projections, contract, capability) + + +def _is_admitted_context_plan(value: object) -> bool: + if not isinstance(value, _ContextPlan) or value._proof is None: + return False + return ( + value._proof.seal is _CONTEXT_ADMISSION_SEAL + and _is_admitted_accounting_plan(value.accounting) + and value._proof.snapshot == _context_plan_snapshot(value) + ) + + +def _compile_all_datums( + graph: object, + limits: _AccountingLimits, +) -> tuple[_TextDatum, ...] | _ContextRejected: + if not isinstance(graph, _ProtectionGraph) or not isinstance(getattr(graph, "datums", None), tuple): + return _ContextRejected(_ContextAdmissionCode.MALFORMED_GRAPH) + if not graph.datums: + return _ContextRejected(_AccountingAdmissionCode.MALFORMED_GRAPH) + if len(graph.datums) > limits.max_datums: + return _ContextRejected(_AccountingAdmissionCode.TOO_MANY_DATUMS) + detached: list[_TextDatum] = [] + seen: set[str] = set() + total_bytes = 0 + for candidate in graph.datums: + if not isinstance(candidate, _TextDatum): + return _ContextRejected(_ContextAdmissionCode.MALFORMED_GRAPH) + datum_id = getattr(candidate, "id", None) + value = getattr(datum_id, "value", None) + text = getattr(candidate, "text", None) + purpose = getattr(candidate, "purpose", None) + if not isinstance(datum_id, _DatumId) or not isinstance(value, str) or not value or not isinstance(text, str): + return _ContextRejected(_ContextAdmissionCode.MALFORMED_GRAPH) + if purpose not in {_DatumPurpose.TARGET, _DatumPurpose.CONTEXT_ONLY}: + return _ContextRejected(_ContextAdmissionCode.INVALID_DATUM_PURPOSE) + try: + id_bytes = len(value.encode("utf-8")) + text_bytes = len(text.encode("utf-8")) + except UnicodeEncodeError: + return _ContextRejected(_ContextAdmissionCode.MALFORMED_GRAPH) + if id_bytes > limits.max_id_bytes: + return _ContextRejected(_AccountingAdmissionCode.MALFORMED_GRAPH) + if text_bytes > limits.max_datum_bytes: + return _ContextRejected(_AccountingAdmissionCode.DATUM_TOO_LARGE) + if value in seen: + return _ContextRejected(_AccountingAdmissionCode.DUPLICATE_DATUM_ID) + seen.add(value) + total_bytes += text_bytes + detached.append(_TextDatum(_DatumId(value), text, purpose)) + if total_bytes > limits.max_graph_bytes: + return _ContextRejected(_AccountingAdmissionCode.GRAPH_TOO_LARGE) + return tuple(detached) + + +def _compile_scopes( + source: object, + datums: tuple[_TextDatum, ...], + target_ids: frozenset[_DatumId], + context_only_datums: tuple[_TextDatum, ...], + contract: _ContextExecutionContract, +) -> tuple[tuple[_DatumId, tuple[_DatumId, ...]], ...] | _ContextRejected: + if not isinstance(source, tuple) or not all(isinstance(scope, _ContextScope) for scope in source): + return _ContextRejected(_ContextAdmissionCode.MALFORMED_GRAPH) + by_value = {datum.id.value: datum.id for datum in datums} + target_values = {datum_id.value for datum_id in target_ids} + observed_targets: set[str] = set() + referenced_context_only: set[str] = set() + compiled: list[tuple[_DatumId, tuple[_DatumId, ...]]] = [] + for scope in source: + scope_result = _compile_scope(scope, by_value, target_values, observed_targets, contract) + if isinstance(scope_result, _ContextRejected): + return scope_result + target_id, member_ids, context_values = scope_result + compiled.append((target_id, member_ids)) + referenced_context_only.update(context_values) + if observed_targets != target_values: + return _ContextRejected(_ContextAdmissionCode.MISSING_CONTEXT_SCOPE) + if referenced_context_only != {datum.id.value for datum in context_only_datums}: + return _ContextRejected(_ContextAdmissionCode.ORPHAN_CONTEXT_DATUM) + position = {datum.id.value: index for index, datum in enumerate(datums)} + compiled.sort(key=lambda item: position[item[0].value]) + return tuple(compiled) + + +def _compile_scope( + scope: _ContextScope, + by_value: dict[str, _DatumId], + target_values: set[str], + observed_targets: set[str], + contract: _ContextExecutionContract, +) -> tuple[_DatumId, tuple[_DatumId, ...], set[str]] | _ContextRejected: + target_value = getattr(getattr(scope, "target", None), "value", None) + members = getattr(scope, "context", None) + if not isinstance(target_value, str) or not isinstance(members, tuple): + return _ContextRejected(_ContextAdmissionCode.MALFORMED_GRAPH) + if target_value not in by_value: + return _ContextRejected(_ContextAdmissionCode.UNKNOWN_CONTEXT_TARGET) + if target_value not in target_values: + return _ContextRejected(_ContextAdmissionCode.CONTEXT_ONLY_TARGET) + if target_value in observed_targets: + return _ContextRejected(_ContextAdmissionCode.DUPLICATE_CONTEXT_SCOPE) + observed_targets.add(target_value) + member_values = _compile_scope_members(members, target_value, by_value) + if isinstance(member_values, _ContextRejected): + return member_values + if not contract.allow_target_as_context and any(value in target_values for value in member_values): + return _ContextRejected(_ContextAdmissionCode.TARGET_CONTEXT_DISABLED) + context_values = {value for value in member_values if value not in target_values} + return by_value[target_value], tuple(by_value[value] for value in member_values), context_values + + +def _compile_scope_members( + members: tuple[object, ...], + target_value: str, + by_value: dict[str, _DatumId], +) -> tuple[str, ...] | _ContextRejected: + values: list[str] = [] + for member in members: + value = getattr(member, "value", None) + if not isinstance(member, _DatumId) or not isinstance(value, str): + return _ContextRejected(_ContextAdmissionCode.MALFORMED_GRAPH) + if value not in by_value: + return _ContextRejected(_ContextAdmissionCode.UNKNOWN_CONTEXT_DATUM) + if value == target_value: + return _ContextRejected(_ContextAdmissionCode.SELF_CONTEXT) + values.append(value) + if len(set(values)) != len(values): + return _ContextRejected(_ContextAdmissionCode.DUPLICATE_CONTEXT_MEMBER) + return tuple(values) + + +def _check_context_limits( + scopes: tuple[tuple[_DatumId, tuple[_DatumId, ...]], ...], + datums: tuple[_TextDatum, ...], + target_datums: tuple[_TextDatum, ...], + limits: _ContextLimits, +) -> _ContextRejected | None: + if not _valid_context_limits(limits): + return _ContextRejected(_ContextAdmissionCode.UNSUPPORTED_CONTEXT_CONTRACT) + text_by_id = {datum.id: datum.text for datum in datums} + total_references = 0 + expanded_bytes = sum(len(datum.text.encode("utf-8")) for datum in target_datums) + for _target_id, members in scopes: + if len(members) > limits.max_context_members_per_target: + return _ContextRejected(_ContextAdmissionCode.CONTEXT_MEMBERS_EXCEEDED) + context_bytes = sum(len(text_by_id[member].encode("utf-8")) for member in members) + if context_bytes > limits.max_context_bytes_per_target: + return _ContextRejected(_ContextAdmissionCode.CONTEXT_BYTES_EXCEEDED) + total_references += len(members) + expanded_bytes += context_bytes + if total_references > limits.max_total_context_references: + return _ContextRejected(_ContextAdmissionCode.TOTAL_CONTEXT_REFERENCES_EXCEEDED) + if expanded_bytes > limits.max_expanded_frame_bytes: + return _ContextRejected(_ContextAdmissionCode.EXPANDED_FRAME_BYTES_EXCEEDED) + return None + + +def _valid_contract(contract: object) -> TypeGuard[_ContextExecutionContract]: + return ( + isinstance(contract, _ContextExecutionContract) + and contract.profile is _ContextProfile.TARGET_CONTEXT_V1 + and contract.schema_version is _ContextSchemaVersion.V1 + and _valid_context_limits(contract.limits) + and type(contract.allow_target_as_context) is bool + and contract.ordering is _ContextOrdering.DECLARED + and contract.retention is _RetentionPosture.DISABLED + and isinstance(contract.required_artifacts, tuple) + and contract.required_artifacts == (_BackendArtifactClass.CONTEXT_REQUEST,) + ) + + +def _materialize_projections( + accounting: _AccountingPlan, + scopes: tuple[tuple[_DatumId, tuple[_DatumId, ...]], ...], + datums: tuple[_TextDatum, ...], +) -> tuple[_CompiledContextProjection, ...]: + member_by_target = dict(scopes) + text_by_id = {datum.id: datum.text for datum in datums} + final_stage = accounting.stages[-1] + projections: list[_CompiledContextProjection] = [] + for target in accounting.datums: + task = _TaskKey(final_stage, _DatumTaskSubject(target.id)) + scope = _ContextScopeKey() + members = member_by_target[target.id] + bindings = tuple( + _CompiledContextBinding(task, scope, ordinal, datum_id) for ordinal, datum_id in enumerate(members) + ) + projections.append( + _CompiledContextProjection( + task, + target.id, + scope, + members, + bindings, + sum(len(text_by_id[member].encode("utf-8")) for member in members), + ) + ) + return tuple(projections) + + +def _admit_context_plan( + accounting: _AccountingPlan, + context_only_datums: tuple[_TextDatum, ...], + projections: tuple[_CompiledContextProjection, ...], + contract: _ContextExecutionContract, + capability: _ContextBackendCapability, +) -> _ContextPlan: + values = (accounting, context_only_datums, projections, contract, capability) + candidate = _ContextPlan(*values) + snapshot = _context_plan_snapshot(candidate) + if snapshot is None: + raise TypeError("private context plan admission failed") + return _ContextPlan(*values, _ContextPlanProof(_CONTEXT_ADMISSION_SEAL, snapshot)) + + +def _context_plan_snapshot(plan: _ContextPlan) -> tuple[object, ...] | None: + try: + return ( + plan.accounting._proof, + tuple((datum.id.value, datum.text, datum.purpose.value) for datum in plan.context_only_datums), + _projection_snapshot(plan.projections), + _contract_snapshot(plan.contract), + _capability_snapshot(plan.preflight_capability), + ) + except (AttributeError, TypeError): + return None + + +def _projection_snapshot(projections: tuple[_CompiledContextProjection, ...]) -> tuple[object, ...]: + snapshots: list[tuple[object, ...]] = [] + for projection in projections: + if not isinstance(projection.owner_task.subject, _DatumTaskSubject): + raise TypeError("private context projection owner is not datum-owned") + bindings: list[tuple[object, ...]] = [] + for binding in projection.bindings: + if not isinstance(binding.owner_task.subject, _DatumTaskSubject): + raise TypeError("private context binding owner is not datum-owned") + bindings.append( + ( + binding.owner_task.stage.value, + binding.owner_task.subject.datum_id.value, + binding.scope, + binding.ordinal, + binding.datum_id.value, + ) + ) + snapshots.append( + ( + projection.owner_task.stage.value, + projection.target_datum_id.value, + projection.scope, + tuple(member.value for member in projection.context_datum_ids), + tuple(bindings), + projection.context_bytes, + ) + ) + return tuple(snapshots) + + +def _contract_snapshot(contract: _ContextExecutionContract) -> tuple[object, ...]: + limits = contract.limits + return ( + contract.profile.value, + contract.schema_version.value, + limits.max_context_members_per_target, + limits.max_context_bytes_per_target, + limits.max_total_context_references, + limits.max_expanded_frame_bytes, + contract.allow_target_as_context, + contract.ordering.value, + tuple(value.value for value in contract.required_artifacts), + contract.retention.value, + ) + + +def _capability_snapshot(capability: _ContextBackendCapability) -> tuple[object, ...]: + limits = capability.limits + return ( + capability.profile.value, + capability.schema_version.value, + limits.max_context_members_per_target, + limits.max_context_bytes_per_target, + limits.max_total_context_references, + limits.max_expanded_frame_bytes, + capability.allow_target_as_context, + capability.ordering.value, + tuple(value.value for value in capability.artifact_classes), + capability.retention.value, + ) diff --git a/src/anonymizer/engine/execution/context_contract.py b/src/anonymizer/engine/execution/context_contract.py new file mode 100644 index 00000000..14c887a1 --- /dev/null +++ b/src/anonymizer/engine/execution/context_contract.py @@ -0,0 +1,119 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Typed private contract for bounded target and context execution.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum + + +class _PrivateContextContractValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private context execution values are not serializable") + + +class _ContextProfile(str, Enum): + TARGET_CONTEXT_V1 = "target-context-v1" + + +class _ContextSchemaVersion(str, Enum): + V1 = "context-workframe-v1" + + +class _ContextOrdering(str, Enum): + DECLARED = "declared" + + +class _RetentionPosture(str, Enum): + DISABLED = "retention_disabled" + ENABLED = "retention_enabled" + UNKNOWN = "unknown" + + +class _BackendArtifactClass(str, Enum): + CONTEXT_REQUEST = "context_request" + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextLimits(_PrivateContextContractValue): + max_context_members_per_target: int + max_context_bytes_per_target: int + max_total_context_references: int + max_expanded_frame_bytes: int + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextExecutionContract(_PrivateContextContractValue): + profile: _ContextProfile + schema_version: _ContextSchemaVersion + limits: _ContextLimits + allow_target_as_context: bool + ordering: _ContextOrdering + required_artifacts: tuple[_BackendArtifactClass, ...] + retention: _RetentionPosture = _RetentionPosture.DISABLED + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextBackendCapability(_PrivateContextContractValue): + profile: _ContextProfile + schema_version: _ContextSchemaVersion + limits: _ContextLimits + allow_target_as_context: bool + ordering: _ContextOrdering + artifact_classes: tuple[_BackendArtifactClass, ...] + retention: _RetentionPosture + + +def _snapshot_context_capability(backend: object) -> _ContextBackendCapability | None: + """Take one fail-closed typed capability snapshot from a private backend.""" + try: + capability_getter = getattr(backend, "context_capability", None) + capability = capability_getter() if callable(capability_getter) else None + except Exception: + return None + return capability if isinstance(capability, _ContextBackendCapability) else None + + +def _capability_satisfies( + capability: object, + contract: object, +) -> bool: + """Return whether one immutable backend snapshot satisfies the frozen contract.""" + if not isinstance(capability, _ContextBackendCapability) or not isinstance(contract, _ContextExecutionContract): + return False + try: + actual = capability.limits + required = contract.limits + return ( + _valid_context_limits(actual) + and _valid_context_limits(required) + and capability.profile is contract.profile + and capability.schema_version is contract.schema_version + and capability.ordering is contract.ordering + and capability.retention is contract.retention is _RetentionPosture.DISABLED + and (capability.allow_target_as_context or not contract.allow_target_as_context) + and set(contract.required_artifacts).issubset(capability.artifact_classes) + and actual.max_context_members_per_target >= required.max_context_members_per_target + and actual.max_context_bytes_per_target >= required.max_context_bytes_per_target + and actual.max_total_context_references >= required.max_total_context_references + and actual.max_expanded_frame_bytes >= required.max_expanded_frame_bytes + ) + except (AttributeError, TypeError): + return False + + +def _valid_context_limits(limits: object) -> bool: + if not isinstance(limits, _ContextLimits): + return False + values = ( + limits.max_context_members_per_target, + limits.max_context_bytes_per_target, + limits.max_total_context_references, + limits.max_expanded_frame_bytes, + ) + return all(type(value) is int and value >= 0 for value in values) diff --git a/src/anonymizer/engine/execution/context_observations.py b/src/anonymizer/engine/execution/context_observations.py new file mode 100644 index 00000000..176eb01a --- /dev/null +++ b/src/anonymizer/engine/execution/context_observations.py @@ -0,0 +1,129 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Content-free, non-authoritative observations for private context framing.""" + +from __future__ import annotations + +import time +from collections.abc import Iterator +from contextlib import contextmanager +from contextvars import ContextVar +from dataclasses import dataclass + +from anonymizer.measurement.session import current_collector + +_SCHEMA_VERSION = "context-observation-v1" +_SEMANTIC_PROFILE = "target-context-v1" +_IMPLEMENTATION_PROFILE = "pandas-runtime-v1" +_ROUTE = "private_dataframe" +_PRIVATE_CONTEXT_COLLECTOR: ContextVar[object | None] = ContextVar( + "anonymizer_private_context_observation_collector", + default=None, +) +_RECORDING_CONTEXT_OBSERVATION: ContextVar[bool] = ContextVar( + "anonymizer_recording_context_observation", + default=False, +) + + +@dataclass(slots=True) +class _ContextObservationTerminal: + outcome: str = "completed" + reason: str = "none" + reconciliation: str = "not_entered" + cleanup: str = "not_entered" + + +@contextmanager +def _observe_context_boundary( + boundary: str, + *, + target_count: int, + context_count: int, + byte_count: int = 0, +) -> Iterator[_ContextObservationTerminal]: + """Record one best-effort start/terminal pair without semantic authority.""" + terminal = _ContextObservationTerminal() + collector = current_collector() or _PRIVATE_CONTEXT_COLLECTOR.get() + started = time.perf_counter() + common = { + "observation_schema": _SCHEMA_VERSION, + "semantic_profile": _SEMANTIC_PROFILE, + "implementation_profile": _IMPLEMENTATION_PROFILE, + "route": _ROUTE, + "boundary": boundary, + "target_count_bucket": _count_bucket(target_count), + "context_count_bucket": _count_bucket(context_count), + "byte_count_bucket": _byte_bucket(byte_count), + } + _record_safely(collector, event="start", duration_sec=0.0, outcome="started", reason="none", **common) + try: + yield terminal + except BaseException: + terminal.outcome = "error" + if terminal.reason == "none": + terminal.reason = "boundary_error" + raise + finally: + _record_safely( + collector, + event="terminal", + duration_sec=max(0.0, time.perf_counter() - started), + outcome=terminal.outcome, + reason=terminal.reason, + reconciliation=terminal.reconciliation, + cleanup=terminal.cleanup, + **common, + ) + + +@contextmanager +def _private_context_observation_session() -> Iterator[None]: + """Preserve only the safe Phase 5 observer across private trace suppression.""" + token = _PRIVATE_CONTEXT_COLLECTOR.set(current_collector()) + try: + yield + finally: + _PRIVATE_CONTEXT_COLLECTOR.reset(token) + + +def _record_safely(collector: object, **fields: object) -> None: + if _RECORDING_CONTEXT_OBSERVATION.get(): + return + record = getattr(collector, "record", None) + if not callable(record): + return + token = _RECORDING_CONTEXT_OBSERVATION.set(True) + try: + record("context_workframe", **fields) + except BaseException: + return + finally: + _RECORDING_CONTEXT_OBSERVATION.reset(token) + + +def _count_bucket(value: int) -> str: + if value <= 0: + return "0" + if value == 1: + return "1" + if value <= 4: + return "2-4" + if value <= 16: + return "5-16" + if value <= 64: + return "17-64" + return "65+" + + +def _byte_bucket(value: int) -> str: + if value <= 0: + return "0" + if value <= 256: + return "1-256" + if value <= 4096: + return "257-4096" + if value <= 65_536: + return "4097-65536" + return "65537+" diff --git a/src/anonymizer/engine/execution/context_workframes.py b/src/anonymizer/engine/execution/context_workframes.py new file mode 100644 index 00000000..1d8a0184 --- /dev/null +++ b/src/anonymizer/engine/execution/context_workframes.py @@ -0,0 +1,484 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Private target/context framing, reconciliation, and cleanup.""" + +from __future__ import annotations + +import secrets +from collections.abc import Callable +from dataclasses import dataclass +from enum import Enum + +import pandas as pd + +from anonymizer.engine.constants import ( + COL_ATTEMPT_ID, + COL_CONTEXT_BINDING_ID, + COL_CONTEXT_ORDINAL, + COL_CONTEXT_OWNER_WORK_ID, + COL_CONTEXT_TEXT, + COL_TARGET_WORK_ID, + COL_TASK_ID, + COL_TEXT, +) +from anonymizer.engine.execution.accounting_evidence import _Dispatch +from anonymizer.engine.execution.accounting_plan import _DatumTaskSubject, _TaskKey +from anonymizer.engine.execution.context_admission import ( + _CompiledContextBinding, + _CompiledContextProjection, + _ContextPlan, + _is_admitted_context_plan, +) +from anonymizer.engine.execution.context_contract import _BackendArtifactClass, _ContextSchemaVersion +from anonymizer.engine.execution.graph import _DatumId + + +class _PrivateWorkframeValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private context workframe values are not serializable") + + +@dataclass(frozen=True, slots=True, repr=False) +class _TargetWorkId(_PrivateWorkframeValue): + value: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextBindingId(_PrivateWorkframeValue): + value: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextPayloadToken(_PrivateWorkframeValue): + value: str + + +class _ContextPayload(str): + _token: _ContextPayloadToken + + def __new__(cls, text: str, token: _ContextPayloadToken) -> _ContextPayload: + value = super().__new__(cls, text) + value._token = token + return value + + @property + def token(self) -> _ContextPayloadToken: + return self._token + + def __repr__(self) -> str: + return "" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private context payload is not serializable") + + +_ExpectedBinding = tuple[ + _ContextBindingId, + _TargetWorkId, + int, + _CompiledContextBinding, + _ContextPayloadToken, +] + + +@dataclass(frozen=True, slots=True, repr=False) +class _BackendArtifactId(_PrivateWorkframeValue): + value: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextBindingEvidence(_PrivateWorkframeValue): + binding_id: _ContextBindingId + owner_target_work_id: _TargetWorkId + ordinal: int + payload_token: _ContextPayloadToken + + +@dataclass(frozen=True, slots=True, repr=False) +class _BackendClosureAttestation(_PrivateWorkframeValue): + artifact_id: _BackendArtifactId + artifact_class: _BackendArtifactClass + closed: bool + schema_version: _ContextSchemaVersion = _ContextSchemaVersion.V1 + + +class _ContextReconciliationStatus(str, Enum): + VERIFIED = "verified" + LOCAL_INVALID = "local_invalid" + GLOBAL_INVALID = "global_invalid" + + +class _ContextBindingFault(str, Enum): + MISSING = "missing" + DUPLICATE = "duplicate" + CONTRADICTORY = "contradictory" + + +class _ContextCleanupStatus(str, Enum): + VERIFIED = "verified" + FAILED = "failed" + UNCONFIRMED = "unconfirmed" + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextReconciliation(_PrivateWorkframeValue): + status: _ContextReconciliationStatus + affected_tasks: tuple[_TaskKey, ...] = () + faults: tuple[tuple[_TaskKey, _ContextBindingFault], ...] = () + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextCleanup(_PrivateWorkframeValue): + status: _ContextCleanupStatus + + +class _WorkframeStateError(RuntimeError): + def __init__(self, message: str = "private context workframe state violation") -> None: + super().__init__(message) + + def __repr__(self) -> str: + return "" + + +class _WorkframeClosedError(_WorkframeStateError): + pass + + +class _WorkframeConstructionError(_WorkframeStateError): + pass + + +def _default_identity() -> str: + return secrets.token_hex(16) + + +class _ContextWorkframes(_PrivateWorkframeValue): + """Owned bounded frames and maps for one dispatched target frontier.""" + + def __init__( + self, + *, + target_frame: pd.DataFrame, + context_frame: pd.DataFrame, + tasks: tuple[_TaskKey, ...], + expected: tuple[_ExpectedBinding, ...], + artifact_id: _BackendArtifactId, + required_artifacts: tuple[_BackendArtifactClass, ...], + schema_version: _ContextSchemaVersion, + ) -> None: + self._target_frame = target_frame + self._context_frame = context_frame + self._tasks = tasks + self._expected = expected + self._artifact_id: _BackendArtifactId | None = artifact_id + self._required_artifacts = required_artifacts + self._schema_version = schema_version + self._closed = False + self._reconciled = False + self._dispatches_bound = False + + @property + def target_frame(self) -> pd.DataFrame: + if self._closed: + return self._target_frame.iloc[0:0].copy() + return self._target_frame.copy() + + @property + def context_frame(self) -> pd.DataFrame: + if self._closed: + return self._context_frame.iloc[0:0].copy() + return self._context_frame.copy() + + @property + def tasks(self) -> tuple[_TaskKey, ...]: + if self._closed: + return () + return self._tasks + + @property + def artifact_id(self) -> _BackendArtifactId: + self._require_active() + if self._artifact_id is None: + raise _WorkframeClosedError + return self._artifact_id + + def expected_bindings(self) -> tuple[tuple[_ContextBindingId, _TargetWorkId, int], ...]: + self._require_active() + return tuple((binding_id, owner, ordinal) for binding_id, owner, ordinal, _binding, _digest in self._expected) + + def target_work_ids(self) -> tuple[_TargetWorkId, ...]: + self._require_active() + return tuple(_TargetWorkId(value) for value in self._target_frame[COL_TARGET_WORK_ID]) + + def bind_dispatches(self, dispatches: tuple[_Dispatch, ...]) -> None: + """Attach the exact accepted phase-4 identities before backend invocation.""" + self._require_active() + target_work_ids = self.target_work_ids() + if ( + self._dispatches_bound + or self._reconciled + or len(dispatches) != len(self._tasks) + or len(dispatches) != len(target_work_ids) + or len({dispatch.attempt_id for dispatch in dispatches}) != len(dispatches) + or any( + dispatch.task != task or dispatch.row_token.value != work_id.value + for dispatch, task, work_id in zip(dispatches, self._tasks, target_work_ids, strict=True) + ) + ): + raise _WorkframeStateError + self._target_frame.loc[:, COL_ATTEMPT_ID] = [dispatch.attempt_id for dispatch in dispatches] + self._dispatches_bound = True + + def reconcile(self, evidence: object) -> _ContextReconciliation: + self._require_active() + if self._reconciled: + raise _WorkframeStateError + self._reconciled = True + if not self._dispatches_bound: + return _ContextReconciliation(_ContextReconciliationStatus.GLOBAL_INVALID) + if not isinstance(evidence, tuple) or not all(isinstance(item, _ContextBindingEvidence) for item in evidence): + return _ContextReconciliation(_ContextReconciliationStatus.GLOBAL_INVALID) + if not all(_valid_binding_evidence(item) for item in evidence): + return _ContextReconciliation(_ContextReconciliationStatus.GLOBAL_INVALID) + expected_by_id = { + binding_id: (owner, ordinal, binding, payload_token) + for binding_id, owner, ordinal, binding, payload_token in self._expected + } + observed_ids = tuple(item.binding_id for item in evidence) + if len(expected_by_id) != len(self._expected) or any( + binding_id not in expected_by_id for binding_id in observed_ids + ): + return _ContextReconciliation(_ContextReconciliationStatus.GLOBAL_INVALID) + faults: dict[_TaskKey, _ContextBindingFault] = {} + for item in evidence: + owner, ordinal, binding, payload_token = expected_by_id[item.binding_id] + if item.owner_target_work_id != owner: + return _ContextReconciliation(_ContextReconciliationStatus.GLOBAL_INVALID) + if type(item.ordinal) is not int: + return _ContextReconciliation(_ContextReconciliationStatus.GLOBAL_INVALID) + if item.ordinal != ordinal: + faults[binding.owner_task] = _ContextBindingFault.CONTRADICTORY + if item.payload_token != payload_token: + faults[binding.owner_task] = _ContextBindingFault.CONTRADICTORY + for binding_id, (_owner, _ordinal, binding, _payload_token) in expected_by_id.items(): + count = observed_ids.count(binding_id) + if count == 0: + faults.setdefault(binding.owner_task, _ContextBindingFault.MISSING) + elif count > 1: + faults.setdefault(binding.owner_task, _ContextBindingFault.DUPLICATE) + if faults: + ordered = tuple(task for task in self._tasks if task in faults) + return _ContextReconciliation( + _ContextReconciliationStatus.LOCAL_INVALID, + ordered, + tuple((task, faults[task]) for task in ordered), + ) + return _ContextReconciliation(_ContextReconciliationStatus.VERIFIED) + + def close(self, attestations: object) -> _ContextCleanup: + self._require_active() + try: + if not isinstance(attestations, tuple) or not all( + isinstance(item, _BackendClosureAttestation) for item in attestations + ): + return _ContextCleanup(_ContextCleanupStatus.UNCONFIRMED) + if len(attestations) != 1: + return _ContextCleanup(_ContextCleanupStatus.UNCONFIRMED) + attestation = attestations[0] + if ( + attestation.artifact_id != self._artifact_id + or self._required_artifacts != (attestation.artifact_class,) + or attestation.schema_version is not self._schema_version + or type(attestation.closed) is not bool + ): + return _ContextCleanup(_ContextCleanupStatus.UNCONFIRMED) + if not self._dispatches_bound or not self._reconciled: + return _ContextCleanup(_ContextCleanupStatus.UNCONFIRMED) + status = _ContextCleanupStatus.VERIFIED if attestation.closed else _ContextCleanupStatus.FAILED + return _ContextCleanup(status) + finally: + self._discard_owned_state() + + def discard_before_dispatch(self) -> None: + """Close only owned frames after dispatch could not commit a backend handoff.""" + self._require_active() + if self._reconciled: + raise _WorkframeStateError + self._discard_owned_state() + + def contain_discard_failure(self) -> None: + """Make owned state inaccessible after a failed pre-dispatch discard.""" + self._closed = True + try: + self._erase_owned_state() + except Exception: + pass + + def _discard_owned_state(self) -> None: + self._erase_owned_state() + self._closed = True + + def _erase_owned_state(self) -> None: + self._target_frame = self._target_frame.iloc[0:0].copy() + self._context_frame = self._context_frame.iloc[0:0].copy() + self._tasks = () + self._expected = () + self._artifact_id = None + self._required_artifacts = () + + def _require_active(self) -> None: + if self._closed: + raise _WorkframeClosedError + + +def _lower_context_workframes( + plan: _ContextPlan, + tasks: tuple[_TaskKey, ...], + *, + target_work_ids: tuple[str, ...] | None = None, + identity_factory: Callable[[], str] = _default_identity, +) -> _ContextWorkframes: + """Lower a ready target frontier from the sealed compiled snapshot only.""" + if not _is_admitted_context_plan(plan): + raise _WorkframeConstructionError + datum_subjects: list[_DatumTaskSubject] = [] + for task in tasks: + if not isinstance(task.subject, _DatumTaskSubject): + raise _WorkframeConstructionError + datum_subjects.append(task.subject) + target_work_ids = _resolve_target_work_ids(tasks, target_work_ids, identity_factory) + projection_by_task = {projection.owner_task: projection for projection in plan.projections} + if len(set(tasks)) != len(tasks) or any(task not in projection_by_task for task in tasks): + raise _WorkframeConstructionError + target_ids = tuple(_TargetWorkId(value) for value in target_work_ids) + target_text = {datum.id: datum.text for datum in plan.accounting.datums} + context_text = {datum.id: datum.text for datum in (*plan.accounting.datums, *plan.context_only_datums)} + expected, context_rows, artifact_value = _lower_binding_rows( + tasks, + target_ids, + projection_by_task, + context_text, + used=set(target_work_ids), + identity_factory=identity_factory, + ) + target_frame = pd.DataFrame( + ( + { + COL_TARGET_WORK_ID: owner.value, + COL_TASK_ID: task, + COL_ATTEMPT_ID: None, + COL_TEXT: target_text[subject.datum_id], + } + for task, subject, owner in zip(tasks, datum_subjects, target_ids, strict=True) + ), + columns=pd.Index([COL_TARGET_WORK_ID, COL_TASK_ID, COL_ATTEMPT_ID, COL_TEXT]), + ) + context_frame = pd.DataFrame( + context_rows, + columns=pd.Index([COL_CONTEXT_BINDING_ID, COL_CONTEXT_OWNER_WORK_ID, COL_CONTEXT_ORDINAL, COL_CONTEXT_TEXT]), + ) + return _ContextWorkframes( + target_frame=target_frame, + context_frame=context_frame, + tasks=tasks, + expected=expected, + artifact_id=_BackendArtifactId(artifact_value), + required_artifacts=plan.contract.required_artifacts, + schema_version=plan.contract.schema_version, + ) + + +def _resolve_target_work_ids( + tasks: tuple[_TaskKey, ...], + supplied: tuple[str, ...] | None, + identity_factory: Callable[[], str], +) -> tuple[str, ...]: + try: + values = supplied if supplied is not None else tuple(identity_factory() for _task in tasks) + except (StopIteration, TypeError): + raise _WorkframeConstructionError from None + if ( + len(tasks) != len(values) + or len(set(values)) != len(values) + or not all(isinstance(value, str) and value for value in values) + ): + raise _WorkframeConstructionError + return values + + +def _lower_binding_rows( + tasks: tuple[_TaskKey, ...], + target_ids: tuple[_TargetWorkId, ...], + projection_by_task: dict[_TaskKey, _CompiledContextProjection], + context_text: dict[_DatumId, str], + *, + used: set[str], + identity_factory: Callable[[], str], +) -> tuple[tuple[_ExpectedBinding, ...], list[dict[str, object]], str]: + expected: list[_ExpectedBinding] = [] + rows: list[dict[str, object]] = [] + try: + for task, owner in zip(tasks, target_ids, strict=True): + for binding in projection_by_task[task].bindings: + binding_id = _ContextBindingId(_claim_work_identity(identity_factory(), used)) + text = context_text[binding.datum_id] + payload_token = _ContextPayloadToken(binding_id.value) + payload = _ContextPayload(text, payload_token) + expected.append((binding_id, owner, binding.ordinal, binding, payload_token)) + rows.append( + { + COL_CONTEXT_BINDING_ID: binding_id.value, + COL_CONTEXT_OWNER_WORK_ID: owner.value, + COL_CONTEXT_ORDINAL: binding.ordinal, + COL_CONTEXT_TEXT: payload, + } + ) + artifact_value = _claim_work_identity(identity_factory(), used) + except (KeyError, StopIteration, TypeError): + raise _WorkframeConstructionError from None + return tuple(expected), rows, artifact_value + + +def _claim_work_identity(value: object, used: set[str]) -> str: + if not isinstance(value, str) or not value or value in used: + raise _WorkframeConstructionError + used.add(value) + return value + + +def _make_context_binding_evidence( + binding_id: str, + owner_target_work_id: str, + ordinal: int, + text: object, +) -> _ContextBindingEvidence: + """Create typed consumption evidence over the exact context-row payload.""" + if not isinstance(text, _ContextPayload): + raise _WorkframeStateError + private_binding_id = _ContextBindingId(binding_id) + private_owner = _TargetWorkId(owner_target_work_id) + return _ContextBindingEvidence( + private_binding_id, + private_owner, + ordinal, + text.token, + ) + + +def _valid_binding_evidence(value: _ContextBindingEvidence) -> bool: + return ( + isinstance(value.binding_id, _ContextBindingId) + and isinstance(value.binding_id.value, str) + and bool(value.binding_id.value) + and isinstance(value.owner_target_work_id, _TargetWorkId) + and isinstance(value.owner_target_work_id.value, str) + and bool(value.owner_target_work_id.value) + and type(value.ordinal) is int + and isinstance(value.payload_token, _ContextPayloadToken) + and isinstance(value.payload_token.value, str) + and bool(value.payload_token.value) + ) diff --git a/src/anonymizer/engine/execution/graph.py b/src/anonymizer/engine/execution/graph.py new file mode 100644 index 00000000..8e20f7a3 --- /dev/null +++ b/src/anonymizer/engine/execution/graph.py @@ -0,0 +1,89 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Private source-neutral protection graph vocabulary.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum + + +class _PrivateGraphValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private protection graph values are not serializable") + + +@dataclass(frozen=True, slots=True, repr=False) +class _DatumId(_PrivateGraphValue): + value: str + + +class _DatumPurpose(str, Enum): + TARGET = "target" + CONTEXT_ONLY = "context_only" + + +class _RelationKind(str, Enum): + RELATED = "related" + + +@dataclass(frozen=True, slots=True, repr=False) +class _TextDatum(_PrivateGraphValue): + id: _DatumId + text: str + purpose: _DatumPurpose = _DatumPurpose.TARGET + + +@dataclass(frozen=True, slots=True, repr=False) +class _DatumLink(_PrivateGraphValue): + source: _DatumId + target: _DatumId + relation: _RelationKind + + +@dataclass(frozen=True, slots=True, repr=False) +class _DatumDependency(_PrivateGraphValue): + prerequisite: _DatumId + dependent: _DatumId + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextScope(_PrivateGraphValue): + target: _DatumId + context: tuple[_DatumId, ...] = () + + +@dataclass(frozen=True, slots=True, repr=False) +class _CoherenceScope(_PrivateGraphValue): + members: tuple[_DatumId, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _AtomicGroup(_PrivateGraphValue): + members: tuple[_DatumId, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _ProtectionGraph(_PrivateGraphValue): + datums: tuple[_TextDatum, ...] + links: tuple[_DatumLink, ...] + context_scopes: tuple[_ContextScope, ...] + coherence_scopes: tuple[_CoherenceScope, ...] + atomic_groups: tuple[_AtomicGroup, ...] + dependencies: tuple[_DatumDependency, ...] = () + + +def _trivial_graph(datums: tuple[_TextDatum, ...]) -> _ProtectionGraph: + """Build explicit independent scopes for a compatibility workload.""" + ids = tuple(datum.id for datum in datums) + return _ProtectionGraph( + datums=datums, + links=(), + context_scopes=tuple(_ContextScope(datum_id) for datum_id in ids), + coherence_scopes=tuple(_CoherenceScope((datum_id,)) for datum_id in ids), + atomic_groups=tuple(_AtomicGroup((datum_id,)) for datum_id in ids), + ) diff --git a/src/anonymizer/engine/execution/graph_runtime.py b/src/anonymizer/engine/execution/graph_runtime.py new file mode 100644 index 00000000..08563abc --- /dev/null +++ b/src/anonymizer/engine/execution/graph_runtime.py @@ -0,0 +1,673 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Execute admitted accounting graphs through the pandas backend.""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass +from inspect import Signature, signature +from typing import Generic, Protocol, TypeGuard, TypeVar, assert_never + +import pandas as pd + +from anonymizer.engine.constants import COL_TEXT +from anonymizer.engine.execution.accounting_admission import ( + _AccountingAdmissionCode, +) +from anonymizer.engine.execution.accounting_evidence import ( + _Dispatch, + _FailureRecord, + _SuccessRecord, + _TerminalRecord, +) +from anonymizer.engine.execution.accounting_ledger import _AccountingLedger +from anonymizer.engine.execution.accounting_outcomes import _AccountingResult, _CauseCode +from anonymizer.engine.execution.accounting_plan import ( + _AccountingPlan, + _DatumTaskSubject, + _is_admitted_accounting_plan, + _TaskKey, +) +from anonymizer.engine.execution.context_admission import ( + _ContextAdmissionCode, + _ContextPlan, + _is_admitted_context_plan, +) +from anonymizer.engine.execution.context_contract import ( + _capability_satisfies, + _ContextBackendCapability, + _snapshot_context_capability, +) +from anonymizer.engine.execution.context_observations import _observe_context_boundary +from anonymizer.engine.execution.context_workframes import ( + _BackendArtifactId, + _BackendClosureAttestation, + _ContextBindingFault, + _ContextCleanupStatus, + _ContextReconciliationStatus, + _ContextWorkframes, + _lower_context_workframes, + _WorkframeConstructionError, +) +from anonymizer.engine.execution.graph import _DatumId, _TextDatum +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.execution.pandas_runtime import _PandasExecutionResult +from anonymizer.engine.ndd.adapter import FailedRecord, _FailedRowEvidence +from anonymizer.engine.private_row_verification import ( + PrivateRowVerificationError, + _InvocationRowVerifier, + _TerminalOutcome, +) + +T = TypeVar("T") + + +class _FrameExecutionBackend(Protocol): + """Private effect boundary implemented by the current pandas runtime.""" + + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: ... + + +@dataclass(frozen=True, slots=True, repr=False) +class _ContextRunnerAdapter: + """Validated private invocation shape for context-capable backends.""" + + runner: Callable[..., object] + + def run( + self, + dataframe: pd.DataFrame, + *, + context_dataframe: pd.DataFrame, + artifact_id: _BackendArtifactId, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> object: + return self.runner( + dataframe, + context_dataframe=context_dataframe, + artifact_id=artifact_id, + invocation=invocation, + data_summary=data_summary, + preview_num_records=preview_num_records, + verifier=verifier, + ) + + +def _adapt_context_runner(backend: object) -> _ContextRunnerAdapter | None: + """Reject missing or incompatible private context runners before execution opens.""" + try: + runner = getattr(backend, "run_context", None) + if not callable(runner): + return None + runner_signature: Signature = signature(runner) + runner_signature.bind( + object(), + context_dataframe=object(), + artifact_id=object(), + invocation=object(), + data_summary=None, + preview_num_records=None, + verifier=object(), + ) + except Exception: + return None + return _ContextRunnerAdapter(runner) + + +class _AccountingGraphAdmissionError(TypeError): + def __init__(self, code: _AccountingAdmissionCode) -> None: + self.code = code + super().__init__("private accounting plan required") + + def __repr__(self) -> str: + return "" + + +class _ContextGraphAdmissionError(TypeError): + def __init__(self, code: _ContextAdmissionCode) -> None: + self.code = code + super().__init__("compatible private context plan and backend required") + + def __repr__(self) -> str: + return "" + + +@dataclass(frozen=True, slots=True, repr=False) +class _AccountingGraphExecution(Generic[T]): + plan: _AccountingPlan + accounting: _AccountingResult[T] + failed_records: tuple[FailedRecord, ...] + + def __repr__(self) -> str: + return "" + + +@dataclass(frozen=True, slots=True, repr=False) +class _PreparedRuntimePlan: + accounting: _AccountingPlan + context: _ContextPlan | None + context_count: int + context_runner: _ContextRunnerAdapter | None + + +@dataclass(slots=True, repr=False) +class _ExecutionFrontier: + ready: tuple[_TaskKey, ...] + dispatches: tuple[_Dispatch, ...] + verifier: _InvocationRowVerifier + bound: pd.DataFrame + workframes: _ContextWorkframes | None + context_runner: _ContextRunnerAdapter | None + + +class _AccountingGraphRuntime: + """Schedule a compiled task DAG through bounded pandas frontiers.""" + + def __init__(self, backend: _FrameExecutionBackend) -> None: + self._backend = backend + + def context_capability(self) -> _ContextBackendCapability | None: + """Take one typed preflight snapshot from the selected backend.""" + return _snapshot_context_capability(self._backend) + + def run( + self, + plan: _AccountingPlan | _ContextPlan, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + hydrate: Callable[[_TextDatum, pd.Series], T], + datum_release_predicate: Callable[[_DatumId, T], bool] = lambda _datum_id, _candidate: True, + group_release_predicate: Callable[[tuple[tuple[_DatumId, T], ...]], bool] = lambda _outputs: True, + ) -> _AccountingGraphExecution[T]: + if preview_num_records is not None: + raise _AccountingGraphAdmissionError(_AccountingAdmissionCode.UNSUPPORTED_TASK_CARDINALITY) + prepared = self._prepare_runtime_plan(plan) + ledger: _AccountingLedger[T] = _AccountingLedger( + prepared.accounting, + datum_release_predicate=datum_release_predicate, + ) + ledger.open() + failed_records: list[FailedRecord] = [] + datum_by_id = {datum.id: datum for datum in prepared.accounting.datums} + while ready := ledger.ready_tasks(): + frontier = self._build_frontier(ledger, prepared, ready, datum_by_id) + if frontier is None: + continue + result = self._invoke_frontier( + ledger, + frontier, + invocation=invocation, + data_summary=data_summary, + ) + if result is None: + break + if _is_well_formed_result(result): + failed_records.extend(result.failed_records) + if not self._accept_frontier(ledger, prepared.accounting, frontier, result, hydrate): + break + if prepared.context is not None: + with _observe_context_boundary( + "release", + target_count=len(prepared.accounting.datums), + context_count=prepared.context_count, + ): + accounting = ledger.finish(group_release_predicate=group_release_predicate) + else: + accounting = ledger.finish(group_release_predicate=group_release_predicate) + return _AccountingGraphExecution(prepared.accounting, accounting, tuple(failed_records)) + + def _prepare_runtime_plan(self, plan: _AccountingPlan | _ContextPlan) -> _PreparedRuntimePlan: + if not isinstance(plan, _ContextPlan): + if not _is_admitted_accounting_plan(plan): + raise _AccountingGraphAdmissionError(_AccountingAdmissionCode.MALFORMED_GRAPH) + if any(not isinstance(task.subject, _DatumTaskSubject) for task in plan.tasks): + raise _AccountingGraphAdmissionError(_AccountingAdmissionCode.UNSUPPORTED_TASK_CARDINALITY) + return _PreparedRuntimePlan(plan, None, 0, None) + if not _is_admitted_context_plan(plan): + raise _ContextGraphAdmissionError(_ContextAdmissionCode.MALFORMED_GRAPH) + if any(not isinstance(task.subject, _DatumTaskSubject) for task in plan.accounting.tasks): + raise _ContextGraphAdmissionError(_ContextAdmissionCode.MALFORMED_GRAPH) + context_count = sum(len(projection.bindings) for projection in plan.projections) + with _observe_context_boundary( + "capability_recheck", + target_count=len(plan.accounting.datums), + context_count=context_count, + ) as observation: + context_runner = _adapt_context_runner(self._backend) + if not _capability_satisfies(self.context_capability(), plan.contract) or context_runner is None: + observation.outcome = "rejected" + observation.reason = _ContextAdmissionCode.BACKEND_INCOMPATIBLE.value + raise _ContextGraphAdmissionError(_ContextAdmissionCode.BACKEND_INCOMPATIBLE) + return _PreparedRuntimePlan(plan.accounting, plan, context_count, context_runner) + + def _build_frontier( + self, + ledger: _AccountingLedger[T], + prepared: _PreparedRuntimePlan, + ready: tuple[_TaskKey, ...], + datum_by_id: dict[_DatumId, _TextDatum], + ) -> _ExecutionFrontier | None: + datum_subjects: list[_DatumTaskSubject] = [] + for task in ready: + if not isinstance(task.subject, _DatumTaskSubject): + raise _AccountingGraphAdmissionError(_AccountingAdmissionCode.UNSUPPORTED_TASK_CARDINALITY) + datum_subjects.append(task.subject) + if prepared.context is None: + frame = pd.DataFrame({COL_TEXT: [datum_by_id[subject.datum_id].text for subject in datum_subjects]}) + dispatches = self._dispatch_frontier(ledger, prepared, ready) + correlations = tuple(dispatch.row_token.value for dispatch in dispatches) + verifier = _InvocationRowVerifier(frame, correlations=correlations) + return _ExecutionFrontier(ready, dispatches, verifier, verifier.bind(frame), None, None) + with _observe_context_boundary( + "workframe_construction", + target_count=len(ready), + context_count=prepared.context_count, + ) as observation: + try: + workframes = _lower_context_workframes(prepared.context, ready) + except _WorkframeConstructionError: + observation.outcome = "failed" + observation.reason = "binding_construction_failed" + for task in ready: + ledger.mark_task_failed(task) + return None + correlations = tuple(work_id.value for work_id in workframes.target_work_ids()) + try: + dispatches = self._dispatch_frontier(ledger, prepared, ready, correlations=correlations) + except Exception: + self._discard_context_workframes(ledger, workframes, target_count=len(ready)) + for task in ready: + ledger.mark_task_failed(task) + return None + try: + workframes.bind_dispatches(dispatches) + except Exception: + ledger.reconcile(dispatches, (), trusted_run_record=False) + self._close_context_workframes( + ledger, + workframes, + (), + target_count=len(ready), + ) + return None + frame = workframes.target_frame.loc[:, [COL_TEXT]] + verifier = _InvocationRowVerifier(frame, correlations=correlations) + return _ExecutionFrontier( + ready, + dispatches, + verifier, + workframes.target_frame, + workframes, + prepared.context_runner, + ) + + @staticmethod + def _dispatch_frontier( + ledger: _AccountingLedger[T], + prepared: _PreparedRuntimePlan, + ready: tuple[_TaskKey, ...], + *, + correlations: tuple[str, ...] | None = None, + ) -> tuple[_Dispatch, ...]: + if prepared.context is None: + return tuple(ledger.dispatch(task) for task in ready) + if correlations is None or len(correlations) != len(ready): + raise _WorkframeConstructionError + with _observe_context_boundary( + "dispatch", + target_count=len(ready), + context_count=prepared.context_count, + ): + return ledger.dispatch_batch(ready, row_token_values=correlations) + + def _invoke_frontier( + self, + ledger: _AccountingLedger[T], + frontier: _ExecutionFrontier, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + ) -> object | None: + try: + return self._call_frontier_backend(frontier, invocation=invocation, data_summary=data_summary) + except KeyboardInterrupt: + ledger.request_cancellation() + frontier.verifier.abort(cancelled=True) + ledger.reconcile(frontier.dispatches, (), trusted_run_record=False) + self._close_frontier_without_evidence(ledger, frontier) + raise + except PrivateRowVerificationError: + frontier.verifier.abort(cancelled=False) + ledger.mark_inconsistent(_CauseCode.CONTRADICTORY) + self._close_frontier_without_evidence(ledger, frontier) + return None + except Exception: + frontier.verifier.abort(cancelled=False) + ledger.reconcile(frontier.dispatches, (), trusted_run_record=False) + self._close_frontier_without_evidence(ledger, frontier) + return None + + def _call_frontier_backend( + self, + frontier: _ExecutionFrontier, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + ) -> object: + if frontier.workframes is None: + return self._backend.run( + frontier.bound, + invocation=invocation, + data_summary=data_summary, + preview_num_records=None, + verifier=frontier.verifier, + ) + if frontier.context_runner is None: + raise TypeError("private context backend is unavailable") + return self._run_context_backend( + frontier.context_runner, + frontier.workframes, + frontier.bound, + invocation=invocation, + data_summary=data_summary, + verifier=frontier.verifier, + ) + + def _accept_frontier( + self, + ledger: _AccountingLedger[T], + plan: _AccountingPlan, + frontier: _ExecutionFrontier, + value: object, + hydrate: Callable[[_TextDatum, pd.Series], T], + ) -> bool: + if not _is_well_formed_result(value): + frontier.verifier.abort(cancelled=False) + ledger.mark_inconsistent(_CauseCode.CONTRADICTORY) + self._close_frontier_without_evidence(ledger, frontier) + return False + try: + frontier.verifier.verify_returned_rows(value.dataframe, value.result_row_tokens) + except PrivateRowVerificationError: + ledger.mark_inconsistent(_CauseCode.CONTRADICTORY) + self._close_frontier(ledger, frontier, value.closure_attestations) + return False + context_status = self._accept_context_evidence(ledger, frontier, value) + if context_status is not _ContextReconciliationStatus.GLOBAL_INVALID: + self._reconcile_frontier(ledger, plan, frontier.dispatches, value, hydrate) + cleanup = self._close_frontier(ledger, frontier, value.closure_attestations) + return cleanup is _ContextCleanupStatus.VERIFIED + + def _accept_context_evidence( + self, + ledger: _AccountingLedger[T], + frontier: _ExecutionFrontier, + result: _PandasExecutionResult, + ) -> _ContextReconciliationStatus: + if frontier.workframes is None: + return _ContextReconciliationStatus.VERIFIED + return self._reconcile_context_bindings( + ledger, + frontier.workframes, + result, + target_count=len(frontier.ready), + ) + + def _close_frontier_without_evidence( + self, + ledger: _AccountingLedger[T], + frontier: _ExecutionFrontier, + ) -> None: + self._close_frontier(ledger, frontier, ()) + + def _close_frontier( + self, + ledger: _AccountingLedger[T], + frontier: _ExecutionFrontier, + attestations: tuple[_BackendClosureAttestation, ...], + ) -> _ContextCleanupStatus: + if frontier.workframes is None: + return _ContextCleanupStatus.VERIFIED + return self._close_context_workframes( + ledger, + frontier.workframes, + attestations, + target_count=len(frontier.ready), + ) + + def _run_context_backend( + self, + context_runner: _ContextRunnerAdapter, + workframes: _ContextWorkframes, + target_frame: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + verifier: _InvocationRowVerifier, + ) -> object: + context_frame = workframes.context_frame + with _observe_context_boundary( + "backend_execution", + target_count=len(target_frame), + context_count=len(context_frame), + ): + return context_runner.run( + target_frame, + context_dataframe=context_frame, + artifact_id=workframes.artifact_id, + invocation=invocation, + data_summary=data_summary, + preview_num_records=None, + verifier=verifier, + ) + + @staticmethod + def _reconcile_context_bindings( + ledger: _AccountingLedger[T], + workframes: _ContextWorkframes, + result: _PandasExecutionResult, + *, + target_count: int, + ) -> _ContextReconciliationStatus: + with _observe_context_boundary( + "reconciliation", + target_count=target_count, + context_count=len(result.context_binding_evidence), + ) as observation: + context_reconciliation = workframes.reconcile(result.context_binding_evidence) + observation.reconciliation = context_reconciliation.status.value + if context_reconciliation.status is _ContextReconciliationStatus.GLOBAL_INVALID: + observation.outcome = "inconsistent" + observation.reason = "binding_attribution_invalid" + ledger.mark_inconsistent(_CauseCode.CONTRADICTORY) + elif context_reconciliation.status is _ContextReconciliationStatus.LOCAL_INVALID: + observation.outcome = "localized" + observation.reason = "binding_evidence_invalid" + cause_by_fault = { + _ContextBindingFault.MISSING: _CauseCode.MISSING, + _ContextBindingFault.DUPLICATE: _CauseCode.DUPLICATE, + _ContextBindingFault.CONTRADICTORY: _CauseCode.CONTRADICTORY, + } + for task, fault in context_reconciliation.faults: + ledger.mark_task_inconsistent(task, cause_by_fault[fault]) + return context_reconciliation.status + + @staticmethod + def _close_context_workframes( + ledger: _AccountingLedger[T], + workframes: _ContextWorkframes, + attestations: tuple[_BackendClosureAttestation, ...], + *, + target_count: int, + ) -> _ContextCleanupStatus: + with _observe_context_boundary( + "cleanup", + target_count=target_count, + context_count=len(workframes.context_frame), + ) as observation: + try: + cleanup = workframes.close(attestations) + except Exception: + observation.outcome = "failed" + observation.reason = _CauseCode.CLEANUP_FAILED.value + observation.cleanup = _ContextCleanupStatus.FAILED.value + ledger.mark_cleanup_failed() + return _ContextCleanupStatus.FAILED + observation.cleanup = cleanup.status.value + if cleanup.status is _ContextCleanupStatus.FAILED: + observation.outcome = "failed" + observation.reason = _CauseCode.CLEANUP_FAILED.value + ledger.mark_cleanup_failed() + elif cleanup.status is _ContextCleanupStatus.UNCONFIRMED: + observation.outcome = "inconsistent" + observation.reason = _CauseCode.CLEANUP_UNCONFIRMED.value + ledger.mark_cleanup_unconfirmed() + return cleanup.status + + @staticmethod + def _discard_context_workframes( + ledger: _AccountingLedger[T], + workframes: _ContextWorkframes, + *, + target_count: int, + ) -> None: + """Discard unopened owned frames when atomic dispatch never committed.""" + with _observe_context_boundary( + "cleanup", + target_count=target_count, + context_count=len(workframes.context_frame), + ) as observation: + try: + workframes.discard_before_dispatch() + except Exception: + observation.outcome = "failed" + observation.reason = _CauseCode.CLEANUP_FAILED.value + observation.cleanup = _ContextCleanupStatus.FAILED.value + try: + workframes.contain_discard_failure() + except Exception: + pass + ledger.mark_cleanup_failed() + return + observation.cleanup = _ContextCleanupStatus.VERIFIED.value + + @staticmethod + def _reconcile_frontier( + ledger: _AccountingLedger[T], + plan: _AccountingPlan, + dispatches: tuple[_Dispatch, ...], + result: _PandasExecutionResult, + hydrate: Callable[[_TextDatum, pd.Series], T], + ) -> None: + dispatch_by_token = {dispatch.row_token.value: dispatch for dispatch in dispatches} + terminal_tokens = tuple(token for token, _status in result.terminal_outcomes) + result_tokens = result.result_row_tokens + successful_tokens = tuple( + token for token, status in result.terminal_outcomes if status is _TerminalOutcome.SUCCESS + ) + failed_tokens = tuple(token for token, status in result.terminal_outcomes if status is _TerminalOutcome.FAILED) + failure_evidence = result.failed_row_evidence + failure_tokens = tuple(evidence.row_token for evidence in failure_evidence) + trusted_stop_tokens = result.trusted_stop_tokens + if ( + len(dispatch_by_token) != len(dispatches) + or len(set(terminal_tokens)) != len(terminal_tokens) + or len(set(result_tokens)) != len(result_tokens) + or not set(terminal_tokens).issubset(dispatch_by_token) + or not set(result_tokens).issubset(dispatch_by_token) + or not set(result_tokens).issubset(successful_tokens) + or len(set(failure_tokens)) != len(failure_tokens) + or not set(failure_tokens).issubset(failed_tokens) + or tuple(evidence.record for evidence in failure_evidence) != tuple(result.failed_records) + or len(set(trusted_stop_tokens)) != len(trusted_stop_tokens) + or not set(trusted_stop_tokens).issubset( + token for token, status in result.terminal_outcomes if status is _TerminalOutcome.CANCELLED + ) + ): + ledger.mark_inconsistent(_CauseCode.FOREIGN) + return + try: + row_by_token = { + token: row for token, (_index, row) in zip(result_tokens, result.dataframe.iterrows(), strict=True) + } + except ValueError: + ledger.mark_inconsistent(_CauseCode.CONTRADICTORY) + return + datum_by_id = {datum.id: datum for datum in plan.datums} + records: list[_TerminalRecord[T]] = [] + lost_dispatches: list[_Dispatch] = [] + if trusted_stop_tokens: + ledger.request_cancellation() + for token, status in result.terminal_outcomes: + dispatch = dispatch_by_token[token] + match status: + case _TerminalOutcome.SUCCESS: + try: + if not isinstance(dispatch.task.subject, _DatumTaskSubject): + raise TypeError + candidate = hydrate(datum_by_id[dispatch.task.subject.datum_id], row_by_token[token]) + except Exception: + records.append(_FailureRecord(dispatch)) + else: + records.append(_SuccessRecord(dispatch, candidate)) + case _TerminalOutcome.FAILED: + records.append(_FailureRecord(dispatch)) + case _TerminalOutcome.CANCELLED: + if token in trusted_stop_tokens: + ledger.acknowledge_stop(dispatch) + else: + lost_dispatches.append(dispatch) + case unreachable: + assert_never(unreachable) + for dispatch in lost_dispatches: + ledger.mark_transport_lost(dispatch) + closed_tokens = {*trusted_stop_tokens, *(dispatch.row_token.value for dispatch in lost_dispatches)} + reconciled = tuple(dispatch for dispatch in dispatches if dispatch.row_token.value not in closed_tokens) + ledger.reconcile(reconciled, tuple(records), trusted_run_record=True) + + +def _is_well_formed_result(value: object) -> TypeGuard[_PandasExecutionResult]: + return ( + isinstance(value, _PandasExecutionResult) + and isinstance(value.dataframe, pd.DataFrame) + and isinstance(value.failed_records, list) + and all(isinstance(record, FailedRecord) for record in value.failed_records) + and isinstance(value.terminal_outcomes, tuple) + and all( + isinstance(item, tuple) + and len(item) == 2 + and isinstance(item[0], str) + and bool(item[0]) + and isinstance(item[1], _TerminalOutcome) + for item in value.terminal_outcomes + ) + and isinstance(value.result_row_tokens, tuple) + and all(isinstance(token, str) and token for token in value.result_row_tokens) + and isinstance(value.failed_row_evidence, tuple) + and all( + isinstance(evidence, _FailedRowEvidence) + and isinstance(evidence.row_token, str) + and bool(evidence.row_token) + and isinstance(evidence.record, FailedRecord) + for evidence in value.failed_row_evidence + ) + and isinstance(value.trusted_stop_tokens, tuple) + and all(isinstance(token, str) and token for token in value.trusted_stop_tokens) + and isinstance(value.context_binding_evidence, tuple) + and isinstance(value.closure_attestations, tuple) + ) diff --git a/src/anonymizer/engine/execution/invocation.py b/src/anonymizer/engine/execution/invocation.py new file mode 100644 index 00000000..818b8534 --- /dev/null +++ b/src/anonymizer/engine/execution/invocation.py @@ -0,0 +1,73 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Private, non-serializable description of one normalized execution.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from data_designer.config.models import ModelConfig + +from anonymizer.config.anonymizer_config import AnonymizerConfig +from anonymizer.config.models import ModelSelection +from anonymizer.config.replace_strategies import ReplaceMethod +from anonymizer.config.rewrite import EvaluationCriteria, PrivacyGoal + + +@dataclass(frozen=True) +class _CompiledRewrite: + """Execution-only rewrite choices, detached from the public config object.""" + + privacy_goal: PrivacyGoal + evaluation: EvaluationCriteria + use_combined_graph: bool + strict_entity_protection: bool + + +@dataclass(frozen=True) +class _CompiledInvocation: + """Immutable, private execution plan; deliberately excludes input and runtime state.""" + + model_configs: tuple[ModelConfig, ...] + selected_models: ModelSelection + gliner_detection_threshold: float + validation_max_entities_per_call: int + validation_excerpt_window_chars: int + entity_labels: tuple[str, ...] | None + replace_method: ReplaceMethod | None + rewrite: _CompiledRewrite | None + + @classmethod + def compile( + cls, + config: AnonymizerConfig, + selected_models: ModelSelection, + model_configs: list[ModelConfig] | None = None, + ) -> _CompiledInvocation: + """Capture only workflow inputs after the public request has been normalized.""" + rewrite = config.rewrite + compiled_rewrite = None + if rewrite is not None: + privacy_goal = rewrite.privacy_goal + if privacy_goal is None: + raise ValueError("rewrite.privacy_goal must not be None") + compiled_rewrite = _CompiledRewrite( + privacy_goal=privacy_goal.model_copy(deep=True), + evaluation=rewrite.evaluation, + use_combined_graph=rewrite.use_combined_graph, + strict_entity_protection=rewrite.strict_entity_protection, + ) + return cls( + model_configs=tuple(model_config.model_copy(deep=True) for model_config in model_configs or ()), + selected_models=selected_models.model_copy(deep=True), + gliner_detection_threshold=config.detect.gliner_threshold, + validation_max_entities_per_call=config.detect.validation_max_entities_per_call, + validation_excerpt_window_chars=config.detect.validation_excerpt_window_chars, + entity_labels=tuple(config.detect.entity_labels) if config.detect.entity_labels is not None else None, + replace_method=config.replace.model_copy(deep=True) if config.replace is not None else None, + rewrite=compiled_rewrite, + ) + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("compiled invocation is not serializable") diff --git a/src/anonymizer/engine/execution/mention_admission.py b/src/anonymizer/engine/execution/mention_admission.py new file mode 100644 index 00000000..91cdaf33 --- /dev/null +++ b/src/anonymizer/engine/execution/mention_admission.py @@ -0,0 +1,313 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Strict target-anchored mention finalization for the private graph profile.""" + +from __future__ import annotations + +from collections import Counter +from dataclasses import dataclass, replace +from enum import Enum + +from anonymizer.engine.execution.graph import _DatumId + + +class _PrivateMentionValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private mention values are not serializable") + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _MentionTargetToken(_PrivateMentionValue): + """Executor-issued identity for one current target.""" + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _CandidateToken(_PrivateMentionValue): + """Executor-issued identity for one provisional candidate lineage.""" + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _MentionId(_PrivateMentionValue): + """Executor-issued graph-scoped mention identity.""" + + +class _MentionProvenance(str, Enum): + SPAN_DETECTOR = "span_detector" + EXACT_AUGMENTER = "exact_augmenter" + + +class _ValidationDecisionKind(str, Enum): + KEEP = "keep" + RECLASS = "reclass" + DROP = "drop" + + +class _MentionRejectionCode(str, Enum): + UNKNOWN_TARGET = "unknown_target" + INVALID_OFFSET = "invalid_offset" + SOURCE_SLICE_MISMATCH = "source_slice_mismatch" + UNSUPPORTED_PROVENANCE = "unsupported_provenance" + MISSING_DECISION = "missing_decision" + DUPLICATE_DECISION = "duplicate_decision" + OVERLAP = "overlap" + FOREIGN_TOKEN = "foreign_token" + STALE_TOKEN = "stale_token" + CONTRADICTORY_CANDIDATE = "contradictory_candidate" + + +@dataclass(frozen=True, slots=True, repr=False) +class _MentionTarget(_PrivateMentionValue): + token: _MentionTargetToken + datum_id: _DatumId + text: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _ProvisionalCandidate(_PrivateMentionValue): + token: _CandidateToken + target_token: _MentionTargetToken + start: int + end: int + source_slice: str + detector_label: str + provenance: _MentionProvenance + + +@dataclass(frozen=True, slots=True, repr=False) +class _ValidationDecision(_PrivateMentionValue): + candidate_token: _CandidateToken + kind: _ValidationDecisionKind + proposed_label: str | None = None + + +@dataclass(frozen=True, slots=True, repr=False) +class _AnchoredMention(_PrivateMentionValue): + id: _MentionId + target_datum_id: _DatumId + start: int + end: int + source_slice: str + detector_label: str + provenance: _MentionProvenance + + +@dataclass(frozen=True, slots=True, repr=False) +class _DetectedGraph(_PrivateMentionValue): + targets: tuple[_MentionTarget, ...] + mentions: tuple[_AnchoredMention, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _MentionRejected(_PrivateMentionValue): + code: _MentionRejectionCode + owner: _MentionTargetToken | None = None + + +@dataclass(frozen=True, slots=True, repr=False) +class _MentionLimits(_PrivateMentionValue): + max_candidates_per_target: int + max_mentions_per_target: int + max_label_bytes: int + max_source_slice_bytes: int + + +def _finalize_mentions( + targets: tuple[_MentionTarget, ...], + candidates: tuple[_ProvisionalCandidate, ...], + decisions: tuple[_ValidationDecision, ...], + *, + limits: _MentionLimits, +) -> _DetectedGraph | _MentionRejected: + target_by_token = _validate_targets(targets) + if target_by_token is None or not _valid_limits(limits): + return _MentionRejected(_MentionRejectionCode.UNKNOWN_TARGET) + normalized = _normalize_candidates(candidates, target_by_token, limits) + if isinstance(normalized, _MentionRejected): + return normalized + decision_by_token = _index_decisions(decisions, normalized) + if isinstance(decision_by_token, _MentionRejected): + return decision_by_token + finalized = _apply_decisions(normalized, decision_by_token, limits) + if isinstance(finalized, _MentionRejected): + return finalized + return _build_detected_graph(targets, target_by_token, finalized, limits) + + +def _validate_targets(targets: object) -> dict[_MentionTargetToken, _MentionTarget] | None: + if ( + not isinstance(targets, tuple) + or not targets + or not all(isinstance(target, _MentionTarget) for target in targets) + ): + return None + target_by_token: dict[_MentionTargetToken, _MentionTarget] = {} + datum_ids: set[_DatumId] = set() + for target in targets: + if ( + not isinstance(target.token, _MentionTargetToken) + or not isinstance(target.datum_id, _DatumId) + or not isinstance(target.text, str) + or target.token in target_by_token + or target.datum_id in datum_ids + ): + return None + target_by_token[target.token] = target + datum_ids.add(target.datum_id) + return target_by_token + + +def _normalize_candidates( + candidates: object, + target_by_token: dict[_MentionTargetToken, _MentionTarget], + limits: _MentionLimits, +) -> tuple[_ProvisionalCandidate, ...] | _MentionRejected: + if not isinstance(candidates, tuple) or not all(isinstance(item, _ProvisionalCandidate) for item in candidates): + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE) + by_token: dict[_CandidateToken, _ProvisionalCandidate] = {} + for candidate in candidates: + target = target_by_token.get(candidate.target_token) + if target is None: + return _MentionRejected(_MentionRejectionCode.UNKNOWN_TARGET) + prior = by_token.get(candidate.token) + if prior is not None: + if prior != candidate: + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE, target.token) + continue + rejection = _validate_candidate(candidate, target, limits) + if rejection is not None: + return rejection + by_token[candidate.token] = candidate + counts = Counter(candidate.target_token for candidate in by_token.values()) + if any(count > limits.max_candidates_per_target for count in counts.values()): + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE) + return tuple(by_token.values()) + + +def _validate_candidate( + candidate: _ProvisionalCandidate, + target: _MentionTarget, + limits: _MentionLimits, +) -> _MentionRejected | None: + owner = target.token + if not isinstance(candidate.token, _CandidateToken): + return _MentionRejected(_MentionRejectionCode.STALE_TOKEN, owner) + if type(candidate.start) is not int or type(candidate.end) is not int: + return _MentionRejected(_MentionRejectionCode.INVALID_OFFSET, owner) + if candidate.start < 0 or candidate.end <= candidate.start or candidate.end > len(target.text): + return _MentionRejected(_MentionRejectionCode.INVALID_OFFSET, owner) + if ( + not isinstance(candidate.source_slice, str) + or target.text[candidate.start : candidate.end] != candidate.source_slice + ): + return _MentionRejected(_MentionRejectionCode.SOURCE_SLICE_MISMATCH, owner) + if not isinstance(candidate.provenance, _MentionProvenance): + return _MentionRejected(_MentionRejectionCode.UNSUPPORTED_PROVENANCE, owner) + if not _valid_bounded_text(candidate.detector_label, limits.max_label_bytes): + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE, owner) + if not _valid_bounded_text(candidate.source_slice, limits.max_source_slice_bytes): + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE, owner) + return None + + +def _index_decisions( + decisions: object, + candidates: tuple[_ProvisionalCandidate, ...], +) -> dict[_CandidateToken, _ValidationDecision] | _MentionRejected: + if not isinstance(decisions, tuple) or not all(isinstance(item, _ValidationDecision) for item in decisions): + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE) + candidate_by_token = {candidate.token: candidate for candidate in candidates} + decision_by_token: dict[_CandidateToken, _ValidationDecision] = {} + for decision in decisions: + candidate = candidate_by_token.get(decision.candidate_token) + if candidate is None: + return _MentionRejected(_MentionRejectionCode.FOREIGN_TOKEN) + if decision.candidate_token in decision_by_token: + return _MentionRejected(_MentionRejectionCode.DUPLICATE_DECISION, candidate.target_token) + decision_by_token[decision.candidate_token] = decision + missing = next((candidate for candidate in candidates if candidate.token not in decision_by_token), None) + if missing is not None: + return _MentionRejected(_MentionRejectionCode.MISSING_DECISION, missing.target_token) + return decision_by_token + + +def _apply_decisions( + candidates: tuple[_ProvisionalCandidate, ...], + decisions: dict[_CandidateToken, _ValidationDecision], + limits: _MentionLimits, +) -> tuple[_ProvisionalCandidate, ...] | _MentionRejected: + finalized: list[_ProvisionalCandidate] = [] + for candidate in candidates: + decision = decisions[candidate.token] + if not isinstance(decision.kind, _ValidationDecisionKind): + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE, candidate.target_token) + if decision.kind is _ValidationDecisionKind.RECLASS: + if not _valid_bounded_text(decision.proposed_label, limits.max_label_bytes): + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE, candidate.target_token) + finalized.append(replace(candidate, detector_label=decision.proposed_label)) + elif decision.proposed_label not in {None, ""}: + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE, candidate.target_token) + elif decision.kind is _ValidationDecisionKind.KEEP: + finalized.append(candidate) + return tuple(finalized) + + +def _build_detected_graph( + targets: tuple[_MentionTarget, ...], + target_by_token: dict[_MentionTargetToken, _MentionTarget], + candidates: tuple[_ProvisionalCandidate, ...], + limits: _MentionLimits, +) -> _DetectedGraph | _MentionRejected: + target_position = {target.token: index for index, target in enumerate(targets)} + ordered = sorted(candidates, key=lambda item: (target_position[item.target_token], item.start, item.end)) + by_span: dict[tuple[_MentionTargetToken, int, int], _ProvisionalCandidate] = {} + previous_end: dict[_MentionTargetToken, int] = {} + mention_counts: Counter[_MentionTargetToken] = Counter() + mentions: list[_AnchoredMention] = [] + for candidate in ordered: + span = (candidate.target_token, candidate.start, candidate.end) + if span in by_span: + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE, candidate.target_token) + if candidate.start < previous_end.get(candidate.target_token, 0): + return _MentionRejected(_MentionRejectionCode.OVERLAP, candidate.target_token) + by_span[span] = candidate + previous_end[candidate.target_token] = candidate.end + mention_counts[candidate.target_token] += 1 + if mention_counts[candidate.target_token] > limits.max_mentions_per_target: + return _MentionRejected(_MentionRejectionCode.CONTRADICTORY_CANDIDATE, candidate.target_token) + mentions.append( + _AnchoredMention( + _MentionId(), + target_by_token[candidate.target_token].datum_id, + candidate.start, + candidate.end, + candidate.source_slice, + candidate.detector_label, + candidate.provenance, + ) + ) + return _DetectedGraph(targets, tuple(mentions)) + + +def _valid_limits(limits: object) -> bool: + return isinstance(limits, _MentionLimits) and all( + type(value) is int and value > 0 + for value in ( + limits.max_candidates_per_target, + limits.max_mentions_per_target, + limits.max_label_bytes, + limits.max_source_slice_bytes, + ) + ) + + +def _valid_bounded_text(value: object, limit: int) -> bool: + if not isinstance(value, str) or not value: + return False + try: + return len(value.encode("utf-8")) <= limit + except UnicodeEncodeError: + return False diff --git a/src/anonymizer/engine/execution/mention_resolution.py b/src/anonymizer/engine/execution/mention_resolution.py new file mode 100644 index 00000000..5b778e4a --- /dev/null +++ b/src/anonymizer/engine/execution/mention_resolution.py @@ -0,0 +1,307 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Explicit-evidence clustering for private target-anchored mentions.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from typing import TypeAlias + +from anonymizer.engine.execution.mention_admission import ( + _AnchoredMention, + _DetectedGraph, + _MentionId, + _MentionTargetToken, +) + + +class _PrivateResolutionValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private resolution values are not serializable") + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _ClusterId(_PrivateResolutionValue): + """Executor-issued graph-scoped cluster identity.""" + + +class _EvidenceVersion(str, Enum): + V1 = "same-subject-evidence/v1" + + +class _EvidenceProvenance(str, Enum): + RESOLVER = "resolver" + + +@dataclass(frozen=True, slots=True, repr=False) +class _ResolverScope(_PrivateResolutionValue): + owner: _MentionTargetToken + eligible_targets: tuple[_MentionTargetToken, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _SameSubjectEvidence(_PrivateResolutionValue): + owner: _MentionTargetToken + left: _MentionId + right: _MentionId + version: _EvidenceVersion + provenance: _EvidenceProvenance = _EvidenceProvenance.RESOLVER + + +@dataclass(frozen=True, slots=True, repr=False) +class _DistinctSubjectEvidence(_PrivateResolutionValue): + owner: _MentionTargetToken + left: _MentionId + right: _MentionId + version: _EvidenceVersion + provenance: _EvidenceProvenance = _EvidenceProvenance.RESOLVER + + +_SubjectEvidence: TypeAlias = _SameSubjectEvidence | _DistinctSubjectEvidence + + +@dataclass(frozen=True, slots=True, repr=False) +class _EntityCluster(_PrivateResolutionValue): + id: _ClusterId + ordered_mention_ids: tuple[_MentionId, ...] + accepted_evidence: tuple[_SameSubjectEvidence, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _ClusteredGraph(_PrivateResolutionValue): + detected: _DetectedGraph + clusters: tuple[_EntityCluster, ...] + accepted_evidence: tuple[_SubjectEvidence, ...] + + +class _ResolutionRejectionCode(str, Enum): + FOREIGN_TOKEN = "foreign_token" + STALE_TOKEN = "stale_token" + INVALID_EVIDENCE = "invalid_evidence" + EVIDENCE_CONTRADICTION = "evidence_contradiction" + + +@dataclass(frozen=True, slots=True, repr=False) +class _ResolutionRejected(_PrivateResolutionValue): + code: _ResolutionRejectionCode + owner: _MentionTargetToken | None = None + + +def _resolve_mentions( + detected: _DetectedGraph, + scopes: tuple[_ResolverScope, ...], + evidence: tuple[_SubjectEvidence, ...], +) -> _ClusteredGraph | _ResolutionRejected: + indexed = _index_detected_graph(detected) + if indexed is None: + return _ResolutionRejected(_ResolutionRejectionCode.STALE_TOKEN) + mention_by_id, mention_owner, mention_position, target_position = indexed + scope_by_owner = _validate_scopes(detected, scopes) + if scope_by_owner is None: + return _ResolutionRejected(_ResolutionRejectionCode.INVALID_EVIDENCE) + normalized = _normalize_evidence( + evidence, + scope_by_owner, + mention_by_id, + mention_owner, + mention_position, + target_position, + ) + if isinstance(normalized, _ResolutionRejected): + return normalized + return _cluster(detected, normalized, mention_position) + + +def _index_detected_graph( + detected: object, +) -> ( + tuple[ + dict[_MentionId, _AnchoredMention], + dict[_MentionId, _MentionTargetToken], + dict[_MentionId, int], + dict[_MentionTargetToken, int], + ] + | None +): + if not isinstance(detected, _DetectedGraph): + return None + target_position = {target.token: index for index, target in enumerate(detected.targets)} + if len(target_position) != len(detected.targets): + return None + target_by_datum = {target.datum_id: target.token for target in detected.targets} + if len(target_by_datum) != len(detected.targets): + return None + mention_by_id: dict[_MentionId, _AnchoredMention] = {} + mention_owner: dict[_MentionId, _MentionTargetToken] = {} + mention_position: dict[_MentionId, int] = {} + for position, mention in enumerate(detected.mentions): + if not isinstance(mention, _AnchoredMention): + return None + owner = target_by_datum.get(mention.target_datum_id) + if owner is None or mention.id in mention_by_id: + return None + mention_by_id[mention.id] = mention + mention_owner[mention.id] = owner + mention_position[mention.id] = position + return mention_by_id, mention_owner, mention_position, target_position + + +def _validate_scopes( + detected: _DetectedGraph, + scopes: object, +) -> dict[_MentionTargetToken, frozenset[_MentionTargetToken]] | None: + if not isinstance(scopes, tuple) or not all(isinstance(scope, _ResolverScope) for scope in scopes): + return None + known = frozenset(target.token for target in detected.targets) + by_owner: dict[_MentionTargetToken, frozenset[_MentionTargetToken]] = {} + for scope in scopes: + eligible = scope.eligible_targets + if ( + scope.owner not in known + or scope.owner in by_owner + or not isinstance(eligible, tuple) + or not eligible + or len(set(eligible)) != len(eligible) + or scope.owner not in eligible + or not set(eligible).issubset(known) + ): + return None + by_owner[scope.owner] = frozenset(eligible) + return by_owner if set(by_owner) == set(known) else None + + +def _normalize_evidence( + evidence: object, + scopes: dict[_MentionTargetToken, frozenset[_MentionTargetToken]], + mention_by_id: dict[_MentionId, _AnchoredMention], + mention_owner: dict[_MentionId, _MentionTargetToken], + mention_position: dict[_MentionId, int], + target_position: dict[_MentionTargetToken, int], +) -> tuple[_SubjectEvidence, ...] | _ResolutionRejected: + if not isinstance(evidence, tuple) or not all( + isinstance(item, (_SameSubjectEvidence, _DistinctSubjectEvidence)) for item in evidence + ): + return _ResolutionRejected(_ResolutionRejectionCode.INVALID_EVIDENCE) + selected: dict[tuple[type[object], _MentionId, _MentionId], _SubjectEvidence] = {} + owner_keys: set[tuple[_MentionTargetToken, type[object], _MentionId, _MentionId]] = set() + pair_kinds: dict[tuple[_MentionId, _MentionId], type[object]] = {} + for item in evidence: + validated = _validate_evidence_item(item, scopes, mention_by_id, mention_owner, mention_position) + if isinstance(validated, _ResolutionRejected): + return validated + left, right = validated + owner_key = (item.owner, type(item), left, right) + if owner_key in owner_keys: + return _ResolutionRejected(_ResolutionRejectionCode.INVALID_EVIDENCE, item.owner) + owner_keys.add(owner_key) + pair = (left, right) + if pair in pair_kinds and pair_kinds[pair] is not type(item): + return _ResolutionRejected(_ResolutionRejectionCode.EVIDENCE_CONTRADICTION) + pair_kinds[pair] = type(item) + key = (type(item), left, right) + prior = selected.get(key) + if prior is None or target_position[item.owner] < target_position[prior.owner]: + selected[key] = type(item)(item.owner, left, right, item.version, item.provenance) + return tuple( + selected[key] + for key in sorted( + selected, + key=lambda value: ( + 0 if value[0] is _SameSubjectEvidence else 1, + mention_position[value[1]], + mention_position[value[2]], + ), + ) + ) + + +def _validate_evidence_item( + item: _SubjectEvidence, + scopes: dict[_MentionTargetToken, frozenset[_MentionTargetToken]], + mention_by_id: dict[_MentionId, _AnchoredMention], + mention_owner: dict[_MentionId, _MentionTargetToken], + mention_position: dict[_MentionId, int], +) -> tuple[_MentionId, _MentionId] | _ResolutionRejected: + if ( + item.owner not in scopes + or item.version is not _EvidenceVersion.V1 + or item.provenance is not _EvidenceProvenance.RESOLVER + ): + return _ResolutionRejected(_ResolutionRejectionCode.INVALID_EVIDENCE) + if item.left not in mention_by_id or item.right not in mention_by_id: + return _ResolutionRejected(_ResolutionRejectionCode.FOREIGN_TOKEN) + if item.left is item.right: + return _ResolutionRejected(_ResolutionRejectionCode.INVALID_EVIDENCE, item.owner) + left_owner = mention_owner[item.left] + right_owner = mention_owner[item.right] + if ( + left_owner not in scopes[item.owner] + or right_owner not in scopes[item.owner] + or item.owner not in {left_owner, right_owner} + ): + return _ResolutionRejected(_ResolutionRejectionCode.INVALID_EVIDENCE, item.owner) + if mention_position[item.left] < mention_position[item.right]: + return item.left, item.right + return item.right, item.left + + +def _cluster( + detected: _DetectedGraph, + evidence: tuple[_SubjectEvidence, ...], + mention_position: dict[_MentionId, int], +) -> _ClusteredGraph | _ResolutionRejected: + parents = {mention.id: mention.id for mention in detected.mentions} + for item in evidence: + if isinstance(item, _SameSubjectEvidence): + _union(parents, item.left, item.right, mention_position) + if any( + _find(parents, item.left) is _find(parents, item.right) + for item in evidence + if isinstance(item, _DistinctSubjectEvidence) + ): + return _ResolutionRejected(_ResolutionRejectionCode.EVIDENCE_CONTRADICTION) + components: dict[_MentionId, list[_MentionId]] = {} + for mention in detected.mentions: + components.setdefault(_find(parents, mention.id), []).append(mention.id) + same = tuple(item for item in evidence if isinstance(item, _SameSubjectEvidence)) + clusters = tuple( + _EntityCluster( + _ClusterId(), + tuple(members), + tuple(item for item in same if item.left in members and item.right in members), + ) + for _root, members in sorted(components.items(), key=lambda item: mention_position[item[1][0]]) + ) + return _ClusteredGraph(detected, clusters, evidence) + + +def _find(parents: dict[_MentionId, _MentionId], mention_id: _MentionId) -> _MentionId: + root = mention_id + while parents[root] is not root: + root = parents[root] + while parents[mention_id] is not mention_id: + parent = parents[mention_id] + parents[mention_id] = root + mention_id = parent + return root + + +def _union( + parents: dict[_MentionId, _MentionId], + left: _MentionId, + right: _MentionId, + positions: dict[_MentionId, int], +) -> None: + left_root = _find(parents, left) + right_root = _find(parents, right) + if left_root is right_root: + return + if positions[left_root] < positions[right_root]: + parents[right_root] = left_root + else: + parents[left_root] = right_root diff --git a/src/anonymizer/engine/execution/pandas_runtime.py b/src/anonymizer/engine/execution/pandas_runtime.py new file mode 100644 index 00000000..53e1b0e8 --- /dev/null +++ b/src/anonymizer/engine/execution/pandas_runtime.py @@ -0,0 +1,320 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""The single private pandas runtime for normalized anonymizer invocations.""" + +from __future__ import annotations + +import logging +import time +from collections import Counter +from dataclasses import dataclass, replace +from typing import TYPE_CHECKING, Protocol, TypeGuard + +from anonymizer.engine.constants import ( + COL_CONTEXT_BINDING_ID, + COL_CONTEXT_ORDINAL, + COL_CONTEXT_OWNER_WORK_ID, + COL_CONTEXT_TEXT, + COL_DETECTED_ENTITIES, + COL_TEXT, + DEFAULT_ENTITY_LABELS, +) +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextBackendCapability, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.context_workframes import ( + _BackendArtifactId, + _BackendClosureAttestation, + _ContextBindingEvidence, + _make_context_binding_evidence, +) +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.ndd.adapter import FailedRecord, _FailedRowEvidence +from anonymizer.engine.private_row_verification import _InvocationRowVerifier, _TerminalOutcome + +logger = logging.getLogger("anonymizer") + + +class _ListConvertible(Protocol): + def tolist(self) -> object: ... + + +def _is_list_convertible(value: object) -> TypeGuard[_ListConvertible]: + return callable(getattr(value, "tolist", None)) + + +def _entity_counts(dataframe: pd.DataFrame) -> Counter[str]: + counts: Counter[str] = Counter() + for raw in dataframe.get(COL_DETECTED_ENTITIES, []): + if isinstance(raw, dict): + entities = raw.get("entities", []) + elif isinstance(raw, list): + entities = raw + else: + entities = getattr(raw, "entities", []) + if _is_list_convertible(entities): + entities = entities.tolist() + if not isinstance(entities, list): + continue + for entity in entities: + label = entity.get("label") if isinstance(entity, dict) else getattr(entity, "label", None) + if isinstance(label, str): + counts[label] += 1 + return counts + + +if TYPE_CHECKING: + import pandas as pd + + from anonymizer.engine.detection.detection_workflow import EntityDetectionWorkflow + from anonymizer.engine.replace.replace_runner import ReplacementWorkflow + from anonymizer.engine.rewrite.combined_rewrite_workflow import CombinedRewriteWorkflow + from anonymizer.engine.rewrite.rewrite_workflow import RewriteWorkflow + + +@dataclass(frozen=True, repr=False) +class _PandasExecutionResult: + dataframe: pd.DataFrame + failed_records: list[FailedRecord] + terminal_outcomes: tuple[tuple[str, _TerminalOutcome], ...] = () + result_row_tokens: tuple[str, ...] = () + failed_row_evidence: tuple[_FailedRowEvidence, ...] = () + trusted_stop_tokens: tuple[str, ...] = () + context_binding_evidence: tuple[_ContextBindingEvidence, ...] = () + closure_attestations: tuple[_BackendClosureAttestation, ...] = () + + def __repr__(self) -> str: + return "" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private pandas execution results are not serializable") + + +class _PandasRuntime: + """Coordinate existing workflows over one normalized pandas dataframe.""" + + def __init__( + self, + *, + detection_workflow: EntityDetectionWorkflow, + replace_runner: ReplacementWorkflow, + rewrite_runner: RewriteWorkflow, + combined_rewrite_runner: CombinedRewriteWorkflow, + ) -> None: + self._detection_workflow = detection_workflow + self._replace_runner = replace_runner + self._rewrite_runner = rewrite_runner + self._combined_rewrite_runner = combined_rewrite_runner + + def context_capability(self) -> _ContextBackendCapability: + """Declare the bounded framing profile supported by this private runtime.""" + return _ContextBackendCapability( + profile=_ContextProfile.TARGET_CONTEXT_V1, + schema_version=_ContextSchemaVersion.V1, + limits=_ContextLimits( + max_context_members_per_target=128, + max_context_bytes_per_target=1_048_576, + max_total_context_references=16_384, + max_expanded_frame_bytes=2_097_152, + ), + allow_target_as_context=True, + ordering=_ContextOrdering.DECLARED, + artifact_classes=(_BackendArtifactClass.CONTEXT_REQUEST,), + retention=_RetentionPosture.DISABLED, + ) + + def run_context( + self, + dataframe: pd.DataFrame, + *, + context_dataframe: pd.DataFrame, + artifact_id: _BackendArtifactId, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + """Execute targets unchanged and attest consumption of the separate context frame. + + Phase 5 qualifies framing only. Context is therefore reconciled as typed + input evidence but is not added to a prompt or used for entity decisions. + """ + required = {COL_CONTEXT_BINDING_ID, COL_CONTEXT_OWNER_WORK_ID, COL_CONTEXT_ORDINAL} + if set(context_dataframe.columns) != {*required, COL_CONTEXT_TEXT}: + raise TypeError("private context frame is malformed") + evidence = tuple( + _make_context_binding_evidence( + row[COL_CONTEXT_BINDING_ID], + row[COL_CONTEXT_OWNER_WORK_ID], + row[COL_CONTEXT_ORDINAL], + row[COL_CONTEXT_TEXT], + ) + for _index, row in context_dataframe.iterrows() + ) + result = self.run( + dataframe, + invocation=invocation, + data_summary=data_summary, + preview_num_records=preview_num_records, + verifier=verifier, + ) + return replace( + result, + context_binding_evidence=evidence, + closure_attestations=( + _BackendClosureAttestation(artifact_id, _BackendArtifactClass.CONTEXT_REQUEST, True), + ), + ) + + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + """Run detection then the compiled replacement or rewrite workflow.""" + num_records = len(dataframe) + if preview_num_records is not None and preview_num_records != num_records: + effective_records = min(preview_num_records, num_records) + if effective_records < preview_num_records: + logger.info( + " |-- 🔍 Running entity detection on capped %d records (requested %d, available %d)", + effective_records, + preview_num_records, + num_records, + ) + else: + logger.info(" |-- 🔍 Running entity detection on %d of %d records", effective_records, num_records) + preview_num_records = effective_records + else: + logger.info("🔍 Running entity detection on %d records", num_records) + if logger.isEnabledFor(logging.DEBUG): + text_lengths = dataframe[COL_TEXT].astype(str).str.len() + logger.debug( + "input text lengths: min=%d, max=%d, mean=%.0f chars (%d records)", + text_lengths.min(), + text_lengths.max(), + text_lengths.mean(), + num_records, + ) + logger.debug( + "detection config: threshold=%.2f, labels=%s", + invocation.gliner_detection_threshold, + invocation.entity_labels + or f"(default: {len(DEFAULT_ENTITY_LABELS)} labels; see anonymizer.DEFAULT_ENTITY_LABELS for list)", + ) + else: + logger.info( + "detection labels in scope: %s", + invocation.entity_labels + or f"(default: {len(DEFAULT_ENTITY_LABELS)} labels; see anonymizer.DEFAULT_ENTITY_LABELS for list)", + ) + started = time.perf_counter() + detection_result = self._detection_workflow.run( + dataframe, + model_configs=list(invocation.model_configs), + selected_models=invocation.selected_models.detection, + gliner_detection_threshold=invocation.gliner_detection_threshold, + validation_max_entities_per_call=invocation.validation_max_entities_per_call, + validation_excerpt_window_chars=invocation.validation_excerpt_window_chars, + entity_labels=list(invocation.entity_labels) if invocation.entity_labels is not None else None, + privacy_goal=invocation.rewrite.privacy_goal if invocation.rewrite is not None else None, + data_summary=data_summary, + tag_latent_entities=invocation.rewrite is not None, + compute_grouped_entities=invocation.replace_method is not None or invocation.rewrite is not None, + preview_num_records=preview_num_records, + ) + logger.info( + " |-- 📋 Detection complete — %d entities found across %d records (%d failed) [%.1fs]", + sum(_entity_counts(detection_result.dataframe).values()), + len(detection_result.dataframe), + len(detection_result.failed_records), + time.perf_counter() - started, + ) + label_counts = _entity_counts(detection_result.dataframe) + if label_counts: + logger.info( + " |-- labels: %s", ", ".join(f"{label}={count}" for label, count in label_counts.most_common()) + ) + detected = verifier.bind_complete_stage_output(detection_result.dataframe) + verifier.freeze_accepted_detections(detected) + if invocation.replace_method is not None: + logger.info("🔄 Running %s replacement", type(invocation.replace_method).__name__) + started = time.perf_counter() + result = self._replace_runner.run( + detected, + replace_method=invocation.replace_method, + model_configs=list(invocation.model_configs), + selected_models=invocation.selected_models.replace, + preview_num_records=preview_num_records, + ) + logger.info( + " |-- 📋 Replacement complete (%d failed) [%.1fs]", + len(result.failed_records), + time.perf_counter() - started, + ) + elif invocation.rewrite is not None: + runner = self._combined_rewrite_runner if invocation.rewrite.use_combined_graph else self._rewrite_runner + logger.info("✏️ Running rewrite pipeline") + started = time.perf_counter() + result = runner.run( + detected, + model_configs=list(invocation.model_configs), + selected_models=invocation.selected_models.rewrite, + replace_model_selection=invocation.selected_models.replace, + privacy_goal=invocation.rewrite.privacy_goal, + evaluation=invocation.rewrite.evaluation, + data_summary=data_summary, + preview_num_records=preview_num_records, + strict_entity_protection=invocation.rewrite.strict_entity_protection, + ) + logger.info( + " |-- 📋 Rewrite complete (%d failed) [%.1fs]", + len(result.failed_records), + time.perf_counter() - started, + ) + else: + final = verifier.finish(verifier.bind_complete_stage_output(detected)) + if detection_result.failed_records: + logger.warning("%d record(s) failed during pipeline processing.", len(detection_result.failed_records)) + logger.info( + "🎉 Pipeline complete — %d records processed, %d total failures", + num_records, + len(detection_result.failed_records), + ) + return _PandasExecutionResult( + dataframe=final, + failed_records=detection_result.failed_records, + terminal_outcomes=verifier.take_terminal_outcomes(), + result_row_tokens=verifier.take_result_order(), + failed_row_evidence=detection_result.failed_row_evidence, + ) + failed_records = [*detection_result.failed_records, *result.failed_records] + if failed_records: + logger.warning("%d record(s) failed during pipeline processing.", len(failed_records)) + final = verifier.finish(verifier.bind_complete_stage_output(result.dataframe)) + logger.info( + "🎉 Pipeline complete — %d records processed, %d total failures", + num_records, + len(detection_result.failed_records) + len(result.failed_records), + ) + return _PandasExecutionResult( + dataframe=final, + failed_records=failed_records, + terminal_outcomes=verifier.take_terminal_outcomes(), + result_row_tokens=verifier.take_result_order(), + failed_row_evidence=( + *detection_result.failed_row_evidence, + *result.failed_row_evidence, + ), + ) diff --git a/src/anonymizer/engine/execution/phase6_ndd_backend.py b/src/anonymizer/engine/execution/phase6_ndd_backend.py new file mode 100644 index 00000000..15376332 --- /dev/null +++ b/src/anonymizer/engine/execution/phase6_ndd_backend.py @@ -0,0 +1,303 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Stage-specific NDD effects for the private Phase 6 Redact profile.""" + +from __future__ import annotations + +import json +from enum import Enum +from typing import TypeVar + +import pandas as pd +from data_designer.config.column_configs import LLMStructuredColumnConfig, LLMTextColumnConfig +from pydantic import BaseModel, ConfigDict, StrictInt, StrictStr + +from anonymizer.engine.constants import ( + COL_PHASE6_AUGMENTED, + COL_PHASE6_CANDIDATES, + COL_PHASE6_CONTEXT, + COL_PHASE6_VALIDATION, + COL_RAW_DETECTED, + COL_TEXT, + DEFAULT_ENTITY_LABELS, + _jinja, +) +from anonymizer.engine.detection.detection_workflow import _inject_detector_params +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextBackendCapability, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.execution.mention_admission import ( + _ValidationDecision, + _ValidationDecisionKind, +) +from anonymizer.engine.execution.mention_resolution import _SubjectEvidence +from anonymizer.engine.execution.phase6_runtime import ( + _CandidateProposal, + _Phase6AugmentationWork, + _Phase6CandidateWork, + _Phase6ResolverWork, + _Phase6ValidationWork, +) +from anonymizer.engine.ndd.adapter import NddAdapter +from anonymizer.engine.ndd.model_loader import resolve_model_alias, resolve_model_aliases + +T = TypeVar("T", bound=BaseModel) + + +class _PrivatePhase6BackendValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Phase 6 backend values are not serializable") + + +class _Phase6NddStageError(RuntimeError): + def __init__(self) -> None: + super().__init__("private Phase 6 provider stage failed") + + def __repr__(self) -> str: + return "" + + +class _AugmentedSpan(BaseModel): + model_config = ConfigDict(extra="forbid") + + start: StrictInt + end: StrictInt + source_slice: StrictStr + detector_label: StrictStr + + +class _AugmentedSpans(BaseModel): + model_config = ConfigDict(extra="forbid") + + entities: list[_AugmentedSpan] + + +class _DecisionKind(str, Enum): + KEEP = "keep" + RECLASS = "reclass" + DROP = "drop" + + +class _CandidateDecision(BaseModel): + model_config = ConfigDict(extra="forbid") + + ordinal: StrictInt + decision: _DecisionKind + proposed_label: StrictStr | None = None + + +class _CandidateDecisions(BaseModel): + model_config = ConfigDict(extra="forbid") + + decisions: list[_CandidateDecision] + + +class _Phase6NddBackend(_PrivatePhase6BackendValue): + """Execute separately accounted detector, augmenter, and validator stages.""" + + def __init__(self, adapter: NddAdapter, invocation: _CompiledInvocation) -> None: + self._adapter = adapter + self._invocation = invocation + + def context_capability(self) -> _ContextBackendCapability: + return _ContextBackendCapability( + _ContextProfile.TARGET_CONTEXT_V1, + _ContextSchemaVersion.V1, + _ContextLimits(0, 0, 0, 65_536), + False, + _ContextOrdering.DECLARED, + (_BackendArtifactClass.CONTEXT_REQUEST,), + _RetentionPosture.DISABLED, + ) + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + labels = list(self._invocation.entity_labels or DEFAULT_ENTITY_LABELS) + model_configs = _inject_detector_params( + model_configs=list(self._invocation.model_configs), + selected_models=self._invocation.selected_models.detection, + labels=labels, + gliner_detection_threshold=self._invocation.gliner_detection_threshold, + ) + result = self._adapter.run_workflow( + pd.DataFrame({COL_TEXT: [work.target.text]}), + model_configs=model_configs, + columns=[ + LLMTextColumnConfig( + name=COL_RAW_DETECTED, + prompt=_jinja(COL_TEXT), + model_alias=resolve_model_alias("entity_detector", self._invocation.selected_models.detection), + ) + ], + workflow_name="phase6-detect", + ) + raw = _single_stage_value(result, COL_RAW_DETECTED) + return _decode_detector(raw) + + def augment(self, work: _Phase6AugmentationWork) -> tuple[_CandidateProposal, ...]: + context = json.dumps([datum.text for datum in work.context], ensure_ascii=False, separators=(",", ":")) + result = self._adapter.run_workflow( + pd.DataFrame({COL_TEXT: [work.target.text], COL_PHASE6_CONTEXT: [context]}), + model_configs=list(self._invocation.model_configs), + columns=[ + LLMStructuredColumnConfig( + name=COL_PHASE6_AUGMENTED, + prompt="""Find additional privacy-sensitive spans in the target only. +Target: """ + + _jinja(COL_TEXT) + + """ +Declared context (evidence only): """ + + _jinja(COL_PHASE6_CONTEXT) + + """ +Return Python-character start/end offsets, the exact target source_slice, and detector_label. +Return an empty entities list when no additional exact target span is justified.""", + model_alias=resolve_model_alias( + "entity_augmenter", + self._invocation.selected_models.detection, + ), + output_format=_AugmentedSpans, + ) + ], + workflow_name="phase6-augment", + ) + parsed = _coerce_model(_single_stage_value(result, COL_PHASE6_AUGMENTED), _AugmentedSpans) + return tuple( + _CandidateProposal(item.start, item.end, item.source_slice, item.detector_label) for item in parsed.entities + ) + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + if not work.candidates: + return () + candidate_payload = json.dumps( + [ + { + "ordinal": ordinal, + "start": candidate.start, + "end": candidate.end, + "source_slice": candidate.source_slice, + "detector_label": candidate.detector_label, + "provenance": candidate.provenance.value, + } + for ordinal, candidate in enumerate(work.candidates) + ], + ensure_ascii=False, + separators=(",", ":"), + ) + aliases = resolve_model_aliases("entity_validator", self._invocation.selected_models.detection) + if not aliases: + raise _Phase6NddStageError + result = self._adapter.run_workflow( + pd.DataFrame({COL_TEXT: [work.target.text], COL_PHASE6_CANDIDATES: [candidate_payload]}), + model_configs=list(self._invocation.model_configs), + columns=[ + LLMStructuredColumnConfig( + name=COL_PHASE6_VALIDATION, + prompt="""Validate every candidate against the target. Return exactly one decision per ordinal. +Target: """ + + _jinja(COL_TEXT) + + """ +Candidates: """ + + _jinja(COL_PHASE6_CANDIDATES) + + """ +Allowed decisions are keep, reclass, and drop. proposed_label is required only for reclass.""", + model_alias=aliases[0], + output_format=_CandidateDecisions, + ) + ], + workflow_name="phase6-validate", + ) + parsed = _coerce_model(_single_stage_value(result, COL_PHASE6_VALIDATION), _CandidateDecisions) + ordinals = tuple(item.ordinal for item in parsed.decisions) + expected = tuple(range(len(work.candidates))) + if len(set(ordinals)) != len(ordinals) or set(ordinals) != set(expected): + raise _Phase6NddStageError + by_ordinal = {item.ordinal: item for item in parsed.decisions} + decisions: list[_ValidationDecision] = [] + for ordinal, candidate in enumerate(work.candidates): + item = by_ordinal[ordinal] + if item.decision is _DecisionKind.RECLASS: + if not isinstance(item.proposed_label, str) or not item.proposed_label: + raise _Phase6NddStageError + decisions.append( + _ValidationDecision(candidate.token, _ValidationDecisionKind.RECLASS, item.proposed_label) + ) + else: + if item.proposed_label is not None: + raise _Phase6NddStageError + kind = ( + _ValidationDecisionKind.KEEP + if item.decision is _DecisionKind.KEEP + else _ValidationDecisionKind.DROP + ) + decisions.append(_ValidationDecision(candidate.token, kind)) + return tuple(decisions) + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SubjectEvidence, ...]: + del work + return () + + def close_phase6(self) -> bool: + return True + + +def _single_stage_value(result: object, column: str) -> object: + dataframe = getattr(result, "dataframe", None) + failures = getattr(result, "failed_records", None) + if ( + not isinstance(dataframe, pd.DataFrame) + or failures != [] + or len(dataframe) != 1 + or column not in dataframe.columns + ): + raise _Phase6NddStageError + return dataframe.iloc[0][column] + + +def _decode_detector(raw: object) -> tuple[_CandidateProposal, ...]: + try: + payload = json.loads(raw) if isinstance(raw, str) else raw + if not isinstance(payload, dict) or set(payload) != {"entities"}: + raise TypeError + entities = payload["entities"] + if not isinstance(entities, list): + raise TypeError + proposals: list[_CandidateProposal] = [] + for entity in entities: + if not isinstance(entity, dict): + raise TypeError + allowed = {"text", "label", "start", "end", "score"} + if not {"text", "label", "start", "end"}.issubset(entity) or set(entity) - allowed: + raise TypeError + proposals.append( + _CandidateProposal( + entity["start"], + entity["end"], + entity["text"], + entity["label"], + ) + ) + return tuple(proposals) + except (KeyError, TypeError, ValueError, json.JSONDecodeError): + raise _Phase6NddStageError from None + + +def _coerce_model(raw: object, model: type[T]) -> T: + try: + if isinstance(raw, model): + return raw + if isinstance(raw, str): + raw = json.loads(raw) + return model.model_validate(raw) + except Exception as error: + del error + raise _Phase6NddStageError from None diff --git a/src/anonymizer/engine/execution/phase6_plan.py b/src/anonymizer/engine/execution/phase6_plan.py new file mode 100644 index 00000000..ee4ae5ce --- /dev/null +++ b/src/anonymizer/engine/execution/phase6_plan.py @@ -0,0 +1,322 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Sealed compiler output for the private Phase 6 Redact profile.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum +from typing import TypeAlias + +from anonymizer.engine.execution.accounting_admission import _AccountingAdmissionCode +from anonymizer.engine.execution.accounting_plan import ( + _AccountingLimits, + _AccountingPlan, + _DatumTaskSubject, + _is_admitted_accounting_plan, + _TaskKey, + _TaskPredecessor, +) +from anonymizer.engine.execution.context_admission import ( + _compile_context_plan, + _ContextAdmissionCode, + _ContextPlan, + _ContextRejected, + _is_admitted_context_plan, +) +from anonymizer.engine.execution.context_contract import ( + _ContextBackendCapability, + _ContextExecutionContract, +) +from anonymizer.engine.execution.graph import _CoherenceScope, _ProtectionGraph +from anonymizer.engine.execution.mention_admission import ( + _MentionLimits, + _MentionTarget, + _MentionTargetToken, +) +from anonymizer.engine.execution.mention_resolution import _ResolverScope +from anonymizer.engine.execution.role_policy import ( + _is_admitted_policy, + _load_redact_role_policy, + _load_substitute_role_policy, + _RolePolicy, + _RolePolicyRejected, +) + + +class _PrivatePhase6PlanValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Phase 6 plan values are not serializable") + + +class _Phase6ProfileVersion(str, Enum): + REDACT_V1 = "phase6-redact-graph/v1" + SUBSTITUTE_V1 = "phase6-substitute-graph/v1" + + +class _Phase6PlanRejectionCode(str, Enum): + INVALID_PROFILE = "invalid_profile" + + +_Phase6RejectionCode: TypeAlias = _AccountingAdmissionCode | _ContextAdmissionCode | _Phase6PlanRejectionCode + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6Rejected(_PrivatePhase6PlanValue): + code: _Phase6RejectionCode + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _Phase6ComponentKey(_PrivatePhase6PlanValue): + """Compiler-issued identity for one target-context resolution component.""" + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6Component(_PrivatePhase6PlanValue): + key: _Phase6ComponentKey + target_tokens: tuple[_MentionTargetToken, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6PlanProof(_PrivatePhase6PlanValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6Plan(_PrivatePhase6PlanValue): + accounting: _AccountingPlan + context: _ContextPlan + targets: tuple[_MentionTarget, ...] + resolver_scopes: tuple[_ResolverScope, ...] + components: tuple[_Phase6Component, ...] + coherence_scopes: tuple[_CoherenceScope, ...] + mention_limits: _MentionLimits + role_policy: _RolePolicy + profile_version: _Phase6ProfileVersion + _proof: _Phase6PlanProof | None = field(default=None, compare=False) + + +_PHASE6_PLAN_SEAL = object() +_PHASE6_REDACT_STAGES = ( + "detect", + "augment", + "validate", + "finalize", + "resolve", + "classify", + "transform", + "verify", +) +_PHASE6_SUBSTITUTE_STAGES = _PHASE6_REDACT_STAGES[:6] + + +def _compile_phase6_plan( + graph: object, + *, + accounting_limits: _AccountingLimits, + context_contract: _ContextExecutionContract, + capability: _ContextBackendCapability, + mention_limits: _MentionLimits, + profile_version: _Phase6ProfileVersion = _Phase6ProfileVersion.REDACT_V1, +) -> _Phase6Plan | _Phase6Rejected: + """Compile target-only mention and resolver identities before invocation effects.""" + selected = _select_profile(profile_version) + if selected is None: + return _Phase6Rejected(_Phase6PlanRejectionCode.INVALID_PROFILE) + stages, policy = selected + context = _compile_context_plan( + graph, + accounting_limits=accounting_limits, + contract=context_contract, + capability=capability, + stages=stages, + ) + if isinstance(context, _ContextRejected): + return _Phase6Rejected(context.code) + if not _valid_mention_limits(mention_limits): + return _Phase6Rejected(_Phase6PlanRejectionCode.INVALID_PROFILE) + if ( + isinstance(policy, _RolePolicyRejected) + or not _is_admitted_policy(policy) + or not isinstance(graph, _ProtectionGraph) + ): + return _Phase6Rejected(_Phase6PlanRejectionCode.INVALID_PROFILE) + coherence_scopes = _detach_coherence_scopes(graph, context) + if coherence_scopes is None: + return _Phase6Rejected(_Phase6PlanRejectionCode.INVALID_PROFILE) + return _materialize_phase6_plan(context, mention_limits, policy, profile_version, coherence_scopes) + + +def _materialize_phase6_plan( + context: _ContextPlan, + mention_limits: _MentionLimits, + policy: _RolePolicy, + profile_version: _Phase6ProfileVersion, + coherence_scopes: tuple[_CoherenceScope, ...], +) -> _Phase6Plan | _Phase6Rejected: + targets = tuple(_MentionTarget(_MentionTargetToken(), datum.id, datum.text) for datum in context.accounting.datums) + scopes = _compile_resolver_scopes(context, targets) + components = _compile_components(targets, scopes) + predecessors = _compile_predecessors(context.accounting, targets, scopes, components) + accounting = context.accounting.with_task_predecessors(predecessors) + values = ( + accounting, + context, + targets, + scopes, + components, + coherence_scopes, + mention_limits, + policy, + profile_version, + ) + candidate = _Phase6Plan(*values) + snapshot = _phase6_plan_snapshot(candidate) + if snapshot is None: + return _Phase6Rejected(_Phase6PlanRejectionCode.INVALID_PROFILE) + return _Phase6Plan(*values, _Phase6PlanProof(_PHASE6_PLAN_SEAL, snapshot)) + + +def _compile_resolver_scopes( + context: _ContextPlan, + targets: tuple[_MentionTarget, ...], +) -> tuple[_ResolverScope, ...]: + token_by_datum = {target.datum_id: target.token for target in targets} + projection_by_target = {projection.target_datum_id: projection for projection in context.projections} + return tuple( + _ResolverScope( + target.token, + ( + target.token, + *tuple( + token_by_datum[datum_id] + for datum_id in projection_by_target[target.datum_id].context_datum_ids + if datum_id in token_by_datum and datum_id != target.datum_id + ), + ), + ) + for target in targets + ) + + +def _is_admitted_phase6_plan(value: object) -> bool: + if not isinstance(value, _Phase6Plan) or value._proof is None: + return False + return ( + value._proof.seal is _PHASE6_PLAN_SEAL + and _is_admitted_accounting_plan(value.accounting) + and _is_admitted_context_plan(value.context) + and _is_admitted_policy(value.role_policy) + and value._proof.snapshot == _phase6_plan_snapshot(value) + ) + + +def _compile_components( + targets: tuple[_MentionTarget, ...], + scopes: tuple[_ResolverScope, ...], +) -> tuple[_Phase6Component, ...]: + parents = {target.token: target.token for target in targets} + + def find(token: _MentionTargetToken) -> _MentionTargetToken: + while parents[token] is not token: + token = parents[token] + return token + + for scope in scopes: + owner_root = find(scope.owner) + for eligible in scope.eligible_targets: + eligible_root = find(eligible) + if owner_root is not eligible_root: + parents[eligible_root] = owner_root + grouped: dict[_MentionTargetToken, list[_MentionTargetToken]] = {} + for target in targets: + grouped.setdefault(find(target.token), []).append(target.token) + return tuple(_Phase6Component(_Phase6ComponentKey(), tuple(members)) for members in grouped.values()) + + +def _compile_predecessors( + accounting: _AccountingPlan, + targets: tuple[_MentionTarget, ...], + scopes: tuple[_ResolverScope, ...], + components: tuple[_Phase6Component, ...], +) -> tuple[_TaskPredecessor, ...]: + datum_by_token = {target.token: target.datum_id for target in targets} + stage_by_value = {stage.value: stage for stage in accounting.stages} + edges: list[_TaskPredecessor] = [] + for scope in scopes: + resolve = _TaskKey(stage_by_value["resolve"], _DatumTaskSubject(datum_by_token[scope.owner])) + for token in scope.eligible_targets: + if token is scope.owner: + continue + edges.append( + _TaskPredecessor( + _TaskKey(stage_by_value["finalize"], _DatumTaskSubject(datum_by_token[token])), + resolve, + ) + ) + return tuple(edges) + + +def _phase6_plan_snapshot(plan: _Phase6Plan) -> tuple[object, ...] | None: + try: + return ( + plan.accounting._proof, + plan.context._proof, + tuple((target.token, target.datum_id.value, target.text) for target in plan.targets), + tuple((scope.owner, scope.eligible_targets) for scope in plan.resolver_scopes), + tuple((component.key, component.target_tokens) for component in plan.components), + tuple(tuple(member.value for member in scope.members) for scope in plan.coherence_scopes), + ( + plan.mention_limits.max_candidates_per_target, + plan.mention_limits.max_mentions_per_target, + plan.mention_limits.max_label_bytes, + plan.mention_limits.max_source_slice_bytes, + ), + plan.role_policy._proof, + plan.profile_version.value, + ) + except (AttributeError, TypeError): + return None + + +def _select_profile( + profile_version: object, +) -> tuple[tuple[str, ...], _RolePolicy | _RolePolicyRejected] | None: + match profile_version: + case _Phase6ProfileVersion.REDACT_V1: + return _PHASE6_REDACT_STAGES, _load_redact_role_policy() + case _Phase6ProfileVersion.SUBSTITUTE_V1: + return _PHASE6_SUBSTITUTE_STAGES, _load_substitute_role_policy() + case _: + return None + + +def _detach_coherence_scopes( + graph: _ProtectionGraph, + context: _ContextPlan, +) -> tuple[_CoherenceScope, ...] | None: + by_value = {datum.id.value: datum.id for datum in context.accounting.datums} + try: + return tuple( + _CoherenceScope(tuple(by_value[member.value] for member in scope.members)) + for scope in graph.coherence_scopes + ) + except (AttributeError, KeyError, TypeError): + return None + + +def _valid_mention_limits(limits: object) -> bool: + return isinstance(limits, _MentionLimits) and all( + type(value) is int and value > 0 + for value in ( + limits.max_candidates_per_target, + limits.max_mentions_per_target, + limits.max_label_bytes, + limits.max_source_slice_bytes, + ) + ) diff --git a/src/anonymizer/engine/execution/phase6_redact_role_policy.json b/src/anonymizer/engine/execution/phase6_redact_role_policy.json new file mode 100644 index 00000000..a82b6bcd --- /dev/null +++ b/src/anonymizer/engine/execution/phase6_redact_role_policy.json @@ -0,0 +1,5 @@ +{ + "digest": "e11a29db1af26c9572e1b4dec9e0a91e80966c6de1d813a378b64210a3bdfc40", + "mappings": [], + "version": "phase6-role-result/v1" +} diff --git a/src/anonymizer/engine/execution/phase6_runtime.py b/src/anonymizer/engine/execution/phase6_runtime.py new file mode 100644 index 00000000..71473cc0 --- /dev/null +++ b/src/anonymizer/engine/execution/phase6_runtime.py @@ -0,0 +1,850 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Invocation-private coordinator for the Phase 6 local Redact profile.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Protocol, TypeAlias + +from anonymizer.engine.execution.accounting_ledger import _AccountingLedger +from anonymizer.engine.execution.accounting_outcomes import ( + _AccountingResult, + _CauseCode, + _DatumQualified, + _GroupReleased, + _InvocationCompleted, + _TaskSucceeded, +) +from anonymizer.engine.execution.accounting_plan import _AtomicGroupKey, _DatumTaskSubject, _TaskKey +from anonymizer.engine.execution.context_contract import _capability_satisfies, _snapshot_context_capability +from anonymizer.engine.execution.context_observations import _observe_context_boundary +from anonymizer.engine.execution.graph import _DatumId, _TextDatum +from anonymizer.engine.execution.mention_admission import ( + _AnchoredMention, + _CandidateToken, + _DetectedGraph, + _finalize_mentions, + _MentionProvenance, + _MentionRejected, + _MentionTarget, + _ProvisionalCandidate, + _ValidationDecision, +) +from anonymizer.engine.execution.mention_resolution import ( + _ClusteredGraph, + _ResolutionRejected, + _ResolutionRejectionCode, + _resolve_mentions, + _ResolverScope, + _SubjectEvidence, +) +from anonymizer.engine.execution.phase6_plan import ( + _is_admitted_phase6_plan, + _Phase6Component, + _Phase6ComponentKey, + _Phase6Plan, + _Phase6ProfileVersion, +) +from anonymizer.engine.execution.redact_patches import ( + _apply_redact_patches, + _bind_patch_manifest, + _BoundPatchManifest, + _build_patch_manifest, + _materialize_redact_patches, + _PatchRejected, + _RedactPatch, + _ReturnedRedact, + _VerifiedDatum, + _VerifiedGraph, + _verify_redact_patches, +) +from anonymizer.engine.execution.role_policy import ( + _ClassifiedRole, + _classify_roles, + _is_admitted_resolved_graph, + _ResolvedGraph, + _RolePolicyRejected, +) + + +class _PrivatePhase6RuntimeValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Phase 6 runtime values are not serializable") + + +@dataclass(frozen=True, slots=True, repr=False) +class _CandidateProposal(_PrivatePhase6RuntimeValue): + start: int + end: int + source_slice: str + detector_label: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6CandidateWork(_PrivatePhase6RuntimeValue): + target: _MentionTarget + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6AugmentationWork(_PrivatePhase6RuntimeValue): + target: _MentionTarget + context: tuple[_TextDatum, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6ValidationWork(_PrivatePhase6RuntimeValue): + target: _MentionTarget + candidates: tuple[_ProvisionalCandidate, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6ResolverWork(_PrivatePhase6RuntimeValue): + owner: _MentionTarget + eligible_mentions: tuple[_AnchoredMention, ...] + + +class _Phase6EffectBackend(Protocol): + def context_capability(self) -> object: ... + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: ... + + def augment(self, work: _Phase6AugmentationWork) -> tuple[_CandidateProposal, ...]: ... + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: ... + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SubjectEvidence, ...]: ... + + def close_phase6(self) -> bool: ... + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6StageReceipt(_PrivatePhase6RuntimeValue): + task: _TaskKey + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6ResolvedDatum(_PrivatePhase6RuntimeValue): + datum_id: _DatumId + component: _Phase6ComponentKey + resolved: _ResolvedGraph + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6TerminalEvidence(_PrivatePhase6RuntimeValue): + tasks: tuple[_TaskKey, ...] + datum_ids: tuple[_DatumId, ...] + groups: tuple[_AtomicGroupKey, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6HandoffProof(_PrivatePhase6RuntimeValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6SubstituteHandoff(_PrivatePhase6RuntimeValue): + component: _Phase6ComponentKey + resolved: _ResolvedGraph + result_version: str + policy_version: str + policy_digest: str + terminal_evidence: _Phase6TerminalEvidence + _proof: _Phase6HandoffProof | None = field(default=None, compare=False) + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6HandoffRejected(_PrivatePhase6RuntimeValue): + pass + + +_Phase6Candidate: TypeAlias = _Phase6StageReceipt | _Phase6ResolvedDatum | _VerifiedDatum + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6ExecutionProof(_PrivatePhase6RuntimeValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase6Execution(_PrivatePhase6RuntimeValue): + accounting: _AccountingResult[_Phase6Candidate] + released: tuple[_VerifiedDatum, ...] + handoffs: tuple[_Phase6SubstituteHandoff, ...] = () + _proof: _Phase6ExecutionProof | None = field(default=None, compare=False) + + +_PHASE6_HANDOFF_SEAL = object() +_PHASE6_EXECUTION_SEAL = object() + + +def _is_admitted_phase6_execution(value: object, plan: object) -> bool: + """Accept only the terminal result sealed for this exact Phase 6 plan.""" + return ( + isinstance(value, _Phase6Execution) + and isinstance(plan, _Phase6Plan) + and value._proof is not None + and value._proof.seal is _PHASE6_EXECUTION_SEAL + and value._proof.snapshot == _phase6_execution_snapshot(plan, value) + ) + + +def _phase6_execution_snapshot(plan: _Phase6Plan, execution: _Phase6Execution) -> tuple[object, ...] | None: + try: + # Keep object identity here: a Phase 7 handoff may only consume the + # terminal prefix produced by this invocation, never a look-alike + # result reconstructed from task keys or candidate values. + return (plan._proof, id(execution.accounting), id(execution.released), id(execution.handoffs)) + except AttributeError: + return None + + +class _Phase6RuntimeAdmissionError(TypeError): + def __init__(self) -> None: + super().__init__("admitted private Phase 6 plan and compatible backend required") + + def __repr__(self) -> str: + return "" + + +class _Phase6TransportLost(RuntimeError): + """Backend signal that a dispatched attempt has no trusted terminal evidence.""" + + +class _GlobalPhase6Fault(Exception): + pass + + +class _ComponentPhase6Fault(Exception): + pass + + +@dataclass(slots=True, repr=False) +class _RuntimeStore(_PrivatePhase6RuntimeValue): + candidates: dict[_DatumId, list[_ProvisionalCandidate]] = field(default_factory=dict) + decisions: dict[_DatumId, tuple[_ValidationDecision, ...]] = field(default_factory=dict) + detected: dict[_DatumId, _DetectedGraph] = field(default_factory=dict) + evidence: dict[_DatumId, tuple[_SubjectEvidence, ...]] = field(default_factory=dict) + clustered: dict[_Phase6ComponentKey, _ClusteredGraph] = field(default_factory=dict) + resolved: dict[_Phase6ComponentKey, _ResolvedGraph] = field(default_factory=dict) + bound: dict[_Phase6ComponentKey, _BoundPatchManifest] = field(default_factory=dict) + patches: dict[_Phase6ComponentKey, tuple[_RedactPatch, ...]] = field(default_factory=dict) + returned: dict[_Phase6ComponentKey, tuple[_ReturnedRedact, ...]] = field(default_factory=dict) + verified: dict[_Phase6ComponentKey, _VerifiedGraph] = field(default_factory=dict) + + def close(self) -> None: + self.candidates.clear() + self.decisions.clear() + self.detected.clear() + self.evidence.clear() + self.clustered.clear() + self.resolved.clear() + self.bound.clear() + self.patches.clear() + self.returned.clear() + self.verified.clear() + + +class _Phase6Runtime: + """Account every Phase 6 semantic task and expose only released Redact results.""" + + def __init__(self, backend: _Phase6EffectBackend) -> None: + self._backend = backend + + def run(self, plan: _Phase6Plan) -> _Phase6Execution: + counts = _observation_counts(plan) + with _observe_context_boundary("capability_recheck", **counts): + self._preflight(plan) + with _observe_context_boundary("workframe_construction", **counts): + ledger: _AccountingLedger[_Phase6Candidate] = _AccountingLedger(plan.accounting) + store = _RuntimeStore() + ledger.open() + try: + with _observe_context_boundary("dispatch", **counts): + with _observe_context_boundary("backend_execution", **counts): + self._drive_ledger(plan, ledger, store) + finally: + with _observe_context_boundary("cleanup", **counts): + self._close_before_release(ledger, store) + with _observe_context_boundary("reconciliation", **counts): + accounting = ledger.finish( + datum_release_predicate=lambda datum_id, candidate: _qualified_datum_predicate( + plan, datum_id, candidate + ), + group_release_predicate=lambda outputs: _qualified_group_predicate(plan, outputs), + ) + with _observe_context_boundary("release", **counts): + released = _collect_released(plan, accounting) + handoff_result = _build_substitute_handoffs(plan, accounting) + handoffs = () if isinstance(handoff_result, _Phase6HandoffRejected) else handoff_result + execution = _Phase6Execution(accounting, released, handoffs) + snapshot = _phase6_execution_snapshot(plan, execution) + if snapshot is None: + raise _Phase6RuntimeAdmissionError + return _Phase6Execution(accounting, released, handoffs, _Phase6ExecutionProof(_PHASE6_EXECUTION_SEAL, snapshot)) + + def _drive_ledger( + self, + plan: _Phase6Plan, + ledger: _AccountingLedger[_Phase6Candidate], + store: _RuntimeStore, + ) -> None: + while ready := ledger.ready_tasks(): + task = ready[0] + dispatch = ledger.dispatch(task) + try: + candidate = self._execute_task(plan, task, store) + except _Phase6TransportLost: + ledger.mark_transport_lost(dispatch) + except _GlobalPhase6Fault: + ledger.mark_inconsistent(_CauseCode.CONTRADICTORY) + except _ComponentPhase6Fault: + ledger.accept_failure(dispatch) + except KeyboardInterrupt: + ledger.request_cancellation() + raise + except Exception: + ledger.accept_failure(dispatch) + else: + ledger.accept_success(dispatch, candidate) + + def _preflight(self, plan: object) -> None: + methods = ("detect", "augment", "validate", "resolve", "close_phase6") + if ( + not isinstance(plan, _Phase6Plan) + or not _is_admitted_phase6_plan(plan) + or not _capability_satisfies(_snapshot_context_capability(self._backend), plan.context.contract) + or not all(callable(getattr(self._backend, method, None)) for method in methods) + ): + raise _Phase6RuntimeAdmissionError + + def _execute_task( + self, + plan: _Phase6Plan, + task: _TaskKey, + store: _RuntimeStore, + ) -> _Phase6Candidate: + if not isinstance(task.subject, _DatumTaskSubject): + raise _GlobalPhase6Fault + target = _target_for_datum(plan, task.subject.datum_id) + if task.stage.value in {"detect", "augment", "validate", "finalize"}: + self._execute_mention_task(plan, task, target, store) + return _Phase6StageReceipt(task) + match task.stage.value: + case "resolve": + self._resolve_target(plan, target, store) + case "classify": + self._classify_component(plan, target, store) + if plan.profile_version is _Phase6ProfileVersion.SUBSTITUTE_V1: + return self._resolved_datum(plan, target, store) + case "transform": + self._transform_component(plan, target, store) + case "verify": + return self._verify_component(plan, target, store) + case _: + raise _GlobalPhase6Fault + return _Phase6StageReceipt(task) + + def _execute_mention_task( + self, + plan: _Phase6Plan, + task: _TaskKey, + target: _MentionTarget, + store: _RuntimeStore, + ) -> None: + match task.stage.value: + case "detect": + self._add_proposals(plan, target, self._backend.detect(_Phase6CandidateWork(target)), store, False) + case "augment": + self._add_proposals(plan, target, self._backend.augment(_augmentation_work(plan, target)), store, True) + case "validate": + candidates = tuple(store.candidates.get(target.datum_id, ())) + decisions = self._backend.validate(_Phase6ValidationWork(target, candidates)) + if not isinstance(decisions, tuple) or not all( + isinstance(decision, _ValidationDecision) for decision in decisions + ): + raise _ComponentPhase6Fault + store.decisions[target.datum_id] = decisions + case "finalize": + self._finalize_target(plan, target, store) + case _: + raise _GlobalPhase6Fault + + @staticmethod + def _finalize_target(plan: _Phase6Plan, target: _MentionTarget, store: _RuntimeStore) -> None: + result = _finalize_mentions( + (target,), + tuple(store.candidates.get(target.datum_id, ())), + store.decisions.get(target.datum_id, ()), + limits=plan.mention_limits, + ) + if isinstance(result, _MentionRejected): + if result.owner is None: + raise _GlobalPhase6Fault + raise _ComponentPhase6Fault + store.detected[target.datum_id] = result + + def _add_proposals( + self, + plan: _Phase6Plan, + target: _MentionTarget, + proposals: object, + store: _RuntimeStore, + augmented: bool, + ) -> None: + if not isinstance(proposals, tuple) or not all(isinstance(item, _CandidateProposal) for item in proposals): + raise _ComponentPhase6Fault + current = store.candidates.setdefault(target.datum_id, []) + provenance = _MentionProvenance.EXACT_AUGMENTER if augmented else _MentionProvenance.SPAN_DETECTOR + for proposal in proposals: + candidate = _candidate_from_proposal(target, proposal, provenance, plan) + current.append(candidate) + if len(current) > plan.mention_limits.max_candidates_per_target: + raise _ComponentPhase6Fault + + def _resolve_target( + self, + plan: _Phase6Plan, + target: _MentionTarget, + store: _RuntimeStore, + ) -> None: + scope = next(scope for scope in plan.resolver_scopes if scope.owner is target.token) + detected = _combine_detected(plan, scope.eligible_targets, store) + work = _Phase6ResolverWork(target, detected.mentions) + evidence = self._backend.resolve(work) + if not isinstance(evidence, tuple): + raise _ComponentPhase6Fault + validation_scopes = tuple( + scope if candidate.token is target.token else _ResolverScope(candidate.token, (candidate.token,)) + for candidate in detected.targets + ) + validated = _resolve_mentions(detected, validation_scopes, evidence) + if isinstance(validated, _ResolutionRejected): + if validated.code in { + _ResolutionRejectionCode.FOREIGN_TOKEN, + _ResolutionRejectionCode.STALE_TOKEN, + }: + raise _GlobalPhase6Fault + raise _ComponentPhase6Fault + store.evidence[target.datum_id] = evidence + + @staticmethod + def _classify_component(plan: _Phase6Plan, target: _MentionTarget, store: _RuntimeStore) -> None: + component = _component_for_target(plan, target) + if component.key in store.resolved: + return + detected = _combine_detected(plan, component.target_tokens, store) + scopes = tuple(scope for scope in plan.resolver_scopes if scope.owner in component.target_tokens) + evidence = tuple( + item + for member in _targets_for_component(plan, component) + for item in store.evidence.get(member.datum_id, ()) + ) + clustered = _resolve_mentions(detected, scopes, evidence) + if isinstance(clustered, _ResolutionRejected): + raise _ComponentPhase6Fault + resolved = _classify_roles(clustered, plan.role_policy) + if isinstance(resolved, _RolePolicyRejected): + raise _ComponentPhase6Fault + store.clustered[component.key] = clustered + store.resolved[component.key] = resolved + + @staticmethod + def _resolved_datum( + plan: _Phase6Plan, + target: _MentionTarget, + store: _RuntimeStore, + ) -> _Phase6ResolvedDatum: + component = _component_for_target(plan, target) + resolved = store.resolved.get(component.key) + if resolved is None or not _valid_substitute_resolved_graph(plan, component, resolved): + raise _ComponentPhase6Fault + return _Phase6ResolvedDatum(target.datum_id, component.key, resolved) + + @staticmethod + def _transform_component(plan: _Phase6Plan, target: _MentionTarget, store: _RuntimeStore) -> None: + component = _component_for_target(plan, target) + if component.key in store.bound: + return + resolved = store.resolved.get(component.key) + if resolved is None: + raise _ComponentPhase6Fault + manifest = _build_patch_manifest(resolved) + if isinstance(manifest, _PatchRejected): + raise _ComponentPhase6Fault + bound = _bind_patch_manifest(manifest) + if isinstance(bound, _PatchRejected): + raise _ComponentPhase6Fault + patches = _materialize_redact_patches(bound) + if isinstance(patches, _PatchRejected): + raise _ComponentPhase6Fault + store.bound[component.key] = bound + store.patches[component.key] = patches + returned = _apply_redact_patches(resolved, patches) + if isinstance(returned, _PatchRejected): + raise _ComponentPhase6Fault + store.returned[component.key] = returned + + @staticmethod + def _verify_component( + plan: _Phase6Plan, + target: _MentionTarget, + store: _RuntimeStore, + ) -> _VerifiedDatum: + component = _component_for_target(plan, target) + verified = store.verified.get(component.key) + if verified is None: + bound = store.bound.get(component.key) + if bound is None: + raise _ComponentPhase6Fault + result = _verify_redact_patches( + bound, + store.patches.get(component.key, ()), + store.returned.get(component.key, ()), + ) + if isinstance(result, _PatchRejected): + raise _ComponentPhase6Fault + store.verified[component.key] = result + verified = result + candidate = next((datum for datum in verified.datums if datum.datum_id == target.datum_id), None) + if candidate is None: + raise _ComponentPhase6Fault + return candidate + + def _close_before_release(self, ledger: _AccountingLedger[_Phase6Candidate], store: _RuntimeStore) -> None: + try: + closed = self._backend.close_phase6() + except Exception: + ledger.mark_cleanup_unconfirmed() + else: + if type(closed) is not bool or not closed: + ledger.mark_cleanup_failed() + finally: + store.close() + + +def _augmentation_work(plan: _Phase6Plan, target: _MentionTarget) -> _Phase6AugmentationWork: + projection = next(item for item in plan.context.projections if item.target_datum_id == target.datum_id) + datum_by_id = {datum.id: datum for datum in (*plan.context.accounting.datums, *plan.context.context_only_datums)} + return _Phase6AugmentationWork( + target, + tuple(datum_by_id[datum_id] for datum_id in projection.context_datum_ids), + ) + + +def _observation_counts(plan: _Phase6Plan) -> dict[str, int]: + context_ids = {datum_id for projection in plan.context.projections for datum_id in projection.context_datum_ids} + return { + "target_count": len(plan.accounting.datums), + "context_count": len(context_ids), + "byte_count": sum(len(target.text.encode("utf-8")) for target in plan.targets), + } + + +def _candidate_from_proposal( + target: _MentionTarget, + proposal: _CandidateProposal, + provenance: _MentionProvenance, + plan: _Phase6Plan, +) -> _ProvisionalCandidate: + values = (proposal.start, proposal.end) + if ( + any(type(value) is not int for value in values) + or proposal.start < 0 + or proposal.end <= proposal.start + or proposal.end > len(target.text) + or target.text[proposal.start : proposal.end] != proposal.source_slice + or not _bounded_text(proposal.source_slice, plan.mention_limits.max_source_slice_bytes) + or not _bounded_text(proposal.detector_label, plan.mention_limits.max_label_bytes) + ): + raise _ComponentPhase6Fault + return _ProvisionalCandidate( + _CandidateToken(), + target.token, + proposal.start, + proposal.end, + proposal.source_slice, + proposal.detector_label, + provenance, + ) + + +def _combine_detected( + plan: _Phase6Plan, + tokens: tuple[object, ...], + store: _RuntimeStore, +) -> _DetectedGraph: + selected = tuple(target for target in plan.targets if target.token in tokens) + graphs = tuple(store.detected.get(target.datum_id) for target in selected) + if any(graph is None for graph in graphs): + raise _ComponentPhase6Fault + mentions = tuple(mention for graph in graphs if graph is not None for mention in graph.mentions) + return _DetectedGraph(selected, mentions) + + +def _target_for_datum(plan: _Phase6Plan, datum_id: _DatumId) -> _MentionTarget: + target = next((candidate for candidate in plan.targets if candidate.datum_id == datum_id), None) + if target is None: + raise _GlobalPhase6Fault + return target + + +def _component_for_target(plan: _Phase6Plan, target: _MentionTarget) -> _Phase6Component: + component = next( + (candidate for candidate in plan.components if target.token in candidate.target_tokens), + None, + ) + if component is None: + raise _GlobalPhase6Fault + return component + + +def _targets_for_component( + plan: _Phase6Plan, + component: _Phase6Component, +) -> tuple[_MentionTarget, ...]: + return tuple(target for target in plan.targets if target.token in component.target_tokens) + + +def _qualified_datum_predicate( + plan: _Phase6Plan, + datum_id: _DatumId, + candidate: _Phase6Candidate, +) -> bool: + if plan.profile_version is _Phase6ProfileVersion.REDACT_V1: + return isinstance(candidate, _VerifiedDatum) and candidate.datum_id == datum_id + if not isinstance(candidate, _Phase6ResolvedDatum) or candidate.datum_id != datum_id: + return False + component = next((item for item in plan.components if item.key is candidate.component), None) + return component is not None and _valid_substitute_resolved_graph(plan, component, candidate.resolved) + + +def _qualified_group_predicate( + plan: _Phase6Plan, + outputs: tuple[tuple[_DatumId, _Phase6Candidate], ...], +) -> bool: + ids = tuple(datum_id for datum_id, _candidate in outputs) + return len(set(ids)) == len(ids) and all( + _qualified_datum_predicate(plan, datum_id, candidate) for datum_id, candidate in outputs + ) + + +def _collect_released( + plan: _Phase6Plan, + accounting: _AccountingResult[_Phase6Candidate], +) -> tuple[_VerifiedDatum, ...]: + released_by_id = { + datum_id: candidate + for group in accounting.groups + if isinstance(group, _GroupReleased) + for datum_id, candidate in group.outputs + if isinstance(candidate, _VerifiedDatum) + } + return tuple(released_by_id[datum.id] for datum in plan.accounting.datums if datum.id in released_by_id) + + +def _build_substitute_handoffs( + plan: _Phase6Plan, + accounting: _AccountingResult[_Phase6Candidate], +) -> tuple[_Phase6SubstituteHandoff, ...] | _Phase6HandoffRejected: + if plan.profile_version is not _Phase6ProfileVersion.SUBSTITUTE_V1 or not _is_admitted_phase6_plan(plan): + return () + if not isinstance(accounting, _AccountingResult) or not isinstance(accounting.invocation, _InvocationCompleted): + return _Phase6HandoffRejected() + candidate_by_datum = _released_substitute_candidates(plan, accounting) + if isinstance(candidate_by_datum, _Phase6HandoffRejected): + return candidate_by_datum + handoffs: list[_Phase6SubstituteHandoff] = [] + for component in plan.components: + handoff = _build_component_handoff(plan, accounting, component, candidate_by_datum) + if isinstance(handoff, _Phase6HandoffRejected): + return handoff + if handoff is not None: + handoffs.append(handoff) + return tuple(handoffs) + + +def _released_substitute_candidates( + plan: _Phase6Plan, + accounting: _AccountingResult[_Phase6Candidate], +) -> dict[_DatumId, _Phase6Candidate] | _Phase6HandoffRejected: + expected_groups = tuple(group.key for group in plan.accounting.atomic_groups) + observed_groups = tuple(outcome.group for outcome in accounting.groups) + if len(set(observed_groups)) != len(observed_groups) or set(observed_groups) != set(expected_groups): + return _Phase6HandoffRejected() + released = tuple(outcome for outcome in accounting.groups if isinstance(outcome, _GroupReleased)) + released_candidates = tuple(item for outcome in released for item in outcome.outputs) + if len({datum_id for datum_id, _candidate in released_candidates}) != len(released_candidates): + return _Phase6HandoffRejected() + return dict(released_candidates) + + +def _build_component_handoff( + plan: _Phase6Plan, + accounting: _AccountingResult[_Phase6Candidate], + component: _Phase6Component, + candidate_by_datum: dict[_DatumId, _Phase6Candidate], +) -> _Phase6SubstituteHandoff | _Phase6HandoffRejected | None: + targets = _targets_for_component(plan, component) + present = tuple(candidate_by_datum[target.datum_id] for target in targets if target.datum_id in candidate_by_datum) + if not present: + return None + if len(present) != len(targets) or not all(isinstance(candidate, _Phase6ResolvedDatum) for candidate in present): + return _Phase6HandoffRejected() + candidates = tuple(candidate for candidate in present if isinstance(candidate, _Phase6ResolvedDatum)) + resolved = candidates[0].resolved + if any( + candidate.component is not component.key + or candidate.resolved is not resolved + or candidate.datum_id != target.datum_id + for candidate, target in zip(candidates, targets, strict=True) + ) or not _valid_substitute_resolved_graph(plan, component, resolved): + return _Phase6HandoffRejected() + terminal = _terminal_evidence(plan, accounting, component, candidates) + if terminal is None: + return _Phase6HandoffRejected() + values = ( + component.key, + resolved, + plan.role_policy.version.value, + plan.role_policy.policy_version, + plan.role_policy.digest, + terminal, + ) + candidate = _Phase6SubstituteHandoff(*values) + snapshot = _handoff_snapshot(candidate) + if snapshot is None: + return _Phase6HandoffRejected() + return _Phase6SubstituteHandoff(*values, _Phase6HandoffProof(_PHASE6_HANDOFF_SEAL, snapshot)) + + +def _is_admitted_substitute_handoff(value: object, plan: _Phase6Plan) -> bool: + if ( + not isinstance(value, _Phase6SubstituteHandoff) + or value._proof is None + or not _is_admitted_phase6_plan(plan) + or plan.profile_version is not _Phase6ProfileVersion.SUBSTITUTE_V1 + or value._proof.seal is not _PHASE6_HANDOFF_SEAL + or value._proof.snapshot != _handoff_snapshot(value) + or value.result_version != plan.role_policy.version.value + or value.policy_version != plan.role_policy.policy_version + or value.policy_digest != plan.role_policy.digest + ): + return False + component = next((candidate for candidate in plan.components if candidate.key is value.component), None) + if component is None or not _valid_substitute_resolved_graph(plan, component, value.resolved): + return False + expected_ids = tuple(target.datum_id for target in _targets_for_component(plan, component)) + expected_tasks = tuple( + task + for task in plan.accounting.tasks + if isinstance(task.subject, _DatumTaskSubject) and task.subject.datum_id in expected_ids + ) + expected_groups = tuple( + group.key for group in plan.accounting.atomic_groups if any(member in expected_ids for member in group.members) + ) + return value.terminal_evidence == _Phase6TerminalEvidence(expected_tasks, expected_ids, expected_groups) + + +def _terminal_evidence( + plan: _Phase6Plan, + accounting: _AccountingResult[_Phase6Candidate], + component: _Phase6Component, + candidates: tuple[_Phase6ResolvedDatum, ...], +) -> _Phase6TerminalEvidence | None: + targets = _targets_for_component(plan, component) + datum_ids = tuple(target.datum_id for target in targets) + tasks = tuple( + task + for task in plan.accounting.tasks + if isinstance(task.subject, _DatumTaskSubject) and task.subject.datum_id in datum_ids + ) + task_by_key = {outcome.task: outcome for outcome in accounting.tasks} + datum_by_id = {outcome.datum_id: outcome for outcome in accounting.datums} + candidate_by_id = {candidate.datum_id: candidate for candidate in candidates} + if len(task_by_key) != len(accounting.tasks) or len(datum_by_id) != len(accounting.datums): + return None + if any(not isinstance(task_by_key.get(task), _TaskSucceeded) for task in tasks): + return None + final_tasks = tuple(_TaskKey(plan.accounting.stages[-1], _DatumTaskSubject(datum_id)) for datum_id in datum_ids) + for task, datum_id in zip(final_tasks, datum_ids, strict=True): + task_outcome = task_by_key[task] + datum_outcome = datum_by_id.get(datum_id) + expected = candidate_by_id[datum_id] + if ( + not isinstance(task_outcome, _TaskSucceeded) + or task_outcome.candidate is not expected + or not isinstance(datum_outcome, _DatumQualified) + or datum_outcome.candidate is not expected + ): + return None + groups = tuple( + group.key for group in plan.accounting.atomic_groups if any(member in datum_ids for member in group.members) + ) + released_by_key = {outcome.group: outcome for outcome in accounting.groups if isinstance(outcome, _GroupReleased)} + if any(group not in released_by_key for group in groups): + return None + return _Phase6TerminalEvidence(tasks, datum_ids, groups) + + +def _valid_substitute_resolved_graph( + plan: _Phase6Plan, + component: _Phase6Component, + resolved: _ResolvedGraph, +) -> bool: + if not _is_admitted_resolved_graph(resolved, plan.role_policy): + return False + targets = _targets_for_component(plan, component) + if tuple(target.token for target in resolved.clustered.detected.targets) != component.target_tokens: + return False + target_ids = {target.datum_id for target in targets} + if any( + item.mention.target_datum_id not in target_ids or not isinstance(item.role_result, _ClassifiedRole) + for item in resolved.mentions + ): + return False + scope_by_datum = { + datum_id: scope_index for scope_index, scope in enumerate(plan.coherence_scopes) for datum_id in scope.members + } + mention_by_id = {item.mention.id: item.mention for item in resolved.mentions} + return all( + len( + { + scope_by_datum.get(mention_by_id[mention_id].target_datum_id) + for mention_id in cluster.ordered_mention_ids + } + ) + == 1 + for cluster in resolved.clustered.clusters + ) + + +def _handoff_snapshot(value: _Phase6SubstituteHandoff) -> tuple[object, ...] | None: + try: + return ( + value.component, + value.resolved._proof, + value.result_version, + value.policy_version, + value.policy_digest, + value.terminal_evidence.tasks, + value.terminal_evidence.datum_ids, + value.terminal_evidence.groups, + ) + except (AttributeError, TypeError): + return None + + +def _bounded_text(value: object, limit: int) -> bool: + if not isinstance(value, str) or not value: + return False + try: + return len(value.encode("utf-8")) <= limit + except UnicodeEncodeError: + return False diff --git a/src/anonymizer/engine/execution/phase6_substitute_role_policy.json b/src/anonymizer/engine/execution/phase6_substitute_role_policy.json new file mode 100644 index 00000000..a53f723b --- /dev/null +++ b/src/anonymizer/engine/execution/phase6_substitute_role_policy.json @@ -0,0 +1,71 @@ +{ + "dispositions": { + "account_number": null, + "age": null, + "api_key": null, + "bank_routing_number": null, + "biometric_identifier": null, + "blood_type": null, + "certificate_license_number": null, + "city": null, + "company_name": null, + "coordinate": null, + "country": null, + "county": null, + "court_name": null, + "credit_debit_card": null, + "customer_id": null, + "cvv": null, + "date": null, + "date_of_birth": null, + "date_time": null, + "degree": null, + "device_identifier": null, + "education_level": null, + "email": "email_address", + "employee_id": null, + "employment_status": null, + "fax_number": "fax_number", + "field_of_study": null, + "first_name": "person_given_name", + "gender": null, + "health_plan_beneficiary_number": null, + "http_cookie": null, + "ipv4": null, + "ipv6": null, + "landmark": null, + "language": null, + "last_name": "person_family_name", + "license_plate": null, + "mac_address": null, + "medical_record_number": null, + "monetary_amount": null, + "national_id": null, + "nationality": null, + "occupation": null, + "organization_name": null, + "password": null, + "phone_number": "voice_phone_number", + "pin": null, + "place_name": null, + "political_view": null, + "postcode": null, + "prison_detention_facility": null, + "race_ethnicity": null, + "religious_belief": null, + "sexuality": null, + "ssn": null, + "state": null, + "street_address": null, + "swift_bic": null, + "tax_id": null, + "time": null, + "unique_id": null, + "university": null, + "url": null, + "user_name": "user_name", + "vehicle_identifier": null + }, + "result_version": "phase6-role-result/v1", + "version": "phase6-substitute-role-policy/v1" +} diff --git a/src/anonymizer/engine/execution/phase7_admission.py b/src/anonymizer/engine/execution/phase7_admission.py new file mode 100644 index 00000000..657121e9 --- /dev/null +++ b/src/anonymizer/engine/execution/phase7_admission.py @@ -0,0 +1,774 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pure admission for the private Phase 7 stable-Substitute profile.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum + +from anonymizer.engine.execution.accounting_plan import ( + _AccountingPlan, + _DatumTaskSubject, + _ScopeTaskSubject, + _StageId, + _TaskKey, + _TaskPredecessor, +) +from anonymizer.engine.execution.context_admission import _CompiledContextProjection +from anonymizer.engine.execution.graph import _CoherenceScope, _DatumId, _TextDatum +from anonymizer.engine.execution.mention_admission import _MentionId +from anonymizer.engine.execution.mention_resolution import _ClusterId +from anonymizer.engine.execution.phase6_plan import ( + _is_admitted_phase6_plan, + _Phase6Plan, + _Phase6ProfileVersion, +) +from anonymizer.engine.execution.phase6_runtime import ( + _is_admitted_substitute_handoff, + _Phase6SubstituteHandoff, +) +from anonymizer.engine.execution.phase7_contract import ( + _is_admitted_phase7_contract, + _Phase7StableSubstituteContract, +) +from anonymizer.engine.execution.role_policy import _ClassifiedRole + + +class _PrivatePhase7AdmissionValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Phase 7 admission values are not serializable") + + +class _Phase7AdmissionCode(str, Enum): + INVALID_INPUT = "invalid_input" + LIMIT_EXCEEDED = "limit_exceeded" + EMPTY_SCOPE = "empty_scope" + DUPLICATE_SCOPE = "duplicate_scope" + DUPLICATE_SCOPE_MEMBER = "duplicate_scope_member" + UNKNOWN_SCOPE_DATUM = "unknown_scope_datum" + SCOPE_COVERAGE_GAP = "scope_coverage_gap" + SCOPE_OVERLAP = "scope_overlap" + UNSUPPORTED_SCOPE_NESTING = "unsupported_scope_nesting" + PHASE6_HANDOFF_MISMATCH = "phase6_handoff_mismatch" + UNSUPPORTED_SELECTOR = "unsupported_selector" + SELECTOR_MISSING = "selector_missing" + SELECTOR_AMBIGUOUS = "selector_ambiguous" + UNSUPPORTED_RELATION = "unsupported_relation" + CROSS_SCOPE_RELATION = "cross_scope_relation" + RELATION_ROLE_MISMATCH = "relation_role_mismatch" + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7Rejected(_PrivatePhase7AdmissionValue): + code: _Phase7AdmissionCode + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7Declarations(_PrivatePhase7AdmissionValue): + coherence_scopes: tuple[_CoherenceScope, ...] + relations: tuple[object, ...] = () + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _Phase7ScopeId(_PrivatePhase7AdmissionValue): + """Compiler-issued opaque capability for one flat coherence scope.""" + + +@dataclass(frozen=True, slots=True, repr=False) +class _ClusterRoleSelector(_PrivatePhase7AdmissionValue): + version: str + cluster: _ClusterId + role: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _RelationDeclaration(_PrivatePhase7AdmissionValue): + version: str + upstream: tuple[_ClusterRoleSelector, ...] + downstream: _ClusterRoleSelector + + +@dataclass(frozen=True, slots=True, repr=False) +class _PreSlot(_PrivatePhase7AdmissionValue): + scope_id: _Phase7ScopeId + cluster_id: _ClusterId + role: str + format: str + mask: str + mention_ids: tuple[_MentionId, ...] + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _ReplacementSlotId(_PrivatePhase7AdmissionValue): + """Compiler-issued identity for one structural scope-cluster-role key.""" + + +@dataclass(frozen=True, slots=True, repr=False) +class _ReplacementSlot(_PrivatePhase7AdmissionValue): + id: _ReplacementSlotId + cluster_id: _ClusterId + role: str + format: str + mask: str + mention_ids: tuple[_MentionId, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _CompiledRelation(_PrivatePhase7AdmissionValue): + version: str + upstream: tuple[_ReplacementSlotId, ...] + downstream: _ReplacementSlotId + + +@dataclass(frozen=True, slots=True, repr=False) +class _RequiredDistinctPair(_PrivatePhase7AdmissionValue): + left: _ReplacementSlotId + right: _ReplacementSlotId + + +@dataclass(frozen=True, slots=True, repr=False) +class _PreSlotRelation(_PrivatePhase7AdmissionValue): + version: str + upstream: tuple[_PreSlot, ...] + downstream: _PreSlot + + +@dataclass(frozen=True, slots=True, repr=False) +class _ScopeManifestProof(_PrivatePhase7AdmissionValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _ScopeManifest(_PrivatePhase7AdmissionValue): + id: _Phase7ScopeId + members: tuple[_DatumId, ...] + slots: tuple[_ReplacementSlot, ...] = () + required_pairs: tuple[_RequiredDistinctPair, ...] = () + relations: tuple[_CompiledRelation, ...] = () + _proof: _ScopeManifestProof | None = field(default=None, compare=False) + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7PlanProof(_PrivatePhase7AdmissionValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7Plan(_PrivatePhase7AdmissionValue): + manifests: tuple[_ScopeManifest, ...] + # These are issued while compiling the Phase 7 declaration. Runtime must + # consume them verbatim; reconstructing scope tasks later loses Phase 4 + # conservation and turns an opaque owner capability into presentation data. + accounting: _AccountingPlan + scope_tasks: tuple[_TaskKey, ...] + application_tasks: tuple[_TaskKey, ...] + application_predecessors: tuple[_TaskPredecessor, ...] + _proof: _Phase7PlanProof | None = field(default=None, compare=False) + + +_SCOPE_MANIFEST_SEAL = object() +_PHASE7_PLAN_SEAL = object() + + +def _compile_phase7_plan( + phase6: object, + handoffs: object, + declarations: object, + contract: object, +) -> _Phase7Plan | _Phase7Rejected: + """Validate authored Phase 7 scope shape without creating runtime state.""" + if ( + not isinstance(phase6, _Phase6Plan) + or not _is_admitted_phase6_plan(phase6) + or not isinstance(handoffs, tuple) + or not isinstance(declarations, _Phase7Declarations) + or not isinstance(contract, _Phase7StableSubstituteContract) + or not _is_admitted_phase7_contract(contract) + or not isinstance(declarations.coherence_scopes, tuple) + or not isinstance(declarations.relations, tuple) + ): + return _Phase7Rejected(_Phase7AdmissionCode.INVALID_INPUT) + if _total_limits_exceeded(phase6, handoffs, declarations, contract): + return _Phase7Rejected(_Phase7AdmissionCode.LIMIT_EXCEEDED) + rejected = _validate_partition(declarations.coherence_scopes, phase6.accounting.datums) + if rejected is not None: + return rejected + if not _valid_phase6_handoffs(phase6, handoffs, contract): + return _Phase7Rejected(_Phase7AdmissionCode.PHASE6_HANDOFF_MISMATCH) + materialized = _materialize_scope_manifests( + declarations.coherence_scopes, + phase6, + handoffs, + declarations.relations, + contract, + ) + if isinstance(materialized, _Phase7Rejected): + return materialized + manifests = materialized + subjects = tuple(_ScopeTaskSubject() for _manifest in manifests) + planning_stage = _StageId("phase7-plan") + application_stage = _StageId("phase7-apply") + accounting = phase6.accounting.with_scope_tasks(planning_stage, subjects).with_datum_stage(application_stage) + scope_tasks = tuple(_TaskKey(planning_stage, subject) for subject in subjects) + application_tasks = tuple(_TaskKey(application_stage, _DatumTaskSubject(datum.id)) for datum in accounting.datums) + application_by_datum = { + task.subject.datum_id: task for task in application_tasks if isinstance(task.subject, _DatumTaskSubject) + } + scope_predecessors = tuple( + _TaskPredecessor(scope_task, application_by_datum[member]) + for manifest, scope_task in zip(manifests, scope_tasks, strict=True) + for member in manifest.members + ) + # The ledger enforces the previous datum stage implicitly and deliberately + # rejects a duplicate explicit edge. Seal the complete effective set on + # the Phase 7 plan while storing only scope edges in Phase 4 accounting. + previous_stage = accounting.stages[-2] + stage_predecessors = tuple( + _TaskPredecessor(_TaskKey(previous_stage, task.subject), task) for task in application_tasks + ) + accounting = accounting.with_task_predecessors((*accounting.task_predecessors, *scope_predecessors)) + application_predecessors = (*scope_predecessors, *stage_predecessors) + candidate = _Phase7Plan(manifests, accounting, scope_tasks, application_tasks, application_predecessors) + snapshot = _phase7_plan_snapshot(candidate) + if snapshot is None: + return _Phase7Rejected(_Phase7AdmissionCode.INVALID_INPUT) + return _Phase7Plan( + manifests, + accounting, + scope_tasks, + application_tasks, + application_predecessors, + _Phase7PlanProof(_PHASE7_PLAN_SEAL, snapshot), + ) + + +def _is_admitted_phase7_plan(value: object) -> bool: + return ( + isinstance(value, _Phase7Plan) + and value._proof is not None + and value._proof.seal is _PHASE7_PLAN_SEAL + and all(_is_admitted_scope_manifest(manifest) for manifest in value.manifests) + and len(value.scope_tasks) == len(value.manifests) + and len(value.application_tasks) == len(value.accounting.datums) + and all( + task in value.accounting.tasks and isinstance(task.subject, _ScopeTaskSubject) for task in value.scope_tasks + ) + and all( + task in value.accounting.tasks and isinstance(task.subject, _DatumTaskSubject) + for task in value.application_tasks + ) + and _has_exact_application_predecessors(value) + and value._proof.snapshot == _phase7_plan_snapshot(value) + ) + + +def _has_exact_application_predecessors(plan: _Phase7Plan) -> bool: + application_by_datum = { + task.subject.datum_id: task for task in plan.application_tasks if isinstance(task.subject, _DatumTaskSubject) + } + expected_scope = tuple( + _TaskPredecessor(scope_task, application_by_datum[member]) + for manifest, scope_task in zip(plan.manifests, plan.scope_tasks, strict=True) + for member in manifest.members + if member in application_by_datum + ) + if len(plan.accounting.stages) < 2: + return False + previous_stage = plan.accounting.stages[-2] + expected_stage = tuple( + _TaskPredecessor(_TaskKey(previous_stage, task.subject), task) for task in plan.application_tasks + ) + phase7_tasks = {*plan.scope_tasks, *plan.application_tasks} + observed_explicit = { + predecessor + for predecessor in plan.accounting.task_predecessors + if predecessor.prerequisite in phase7_tasks or predecessor.dependent in phase7_tasks + } + return ( + len(application_by_datum) == len(plan.accounting.datums) + and set(application_by_datum) == {datum.id for datum in plan.accounting.datums} + and all(task.stage == plan.accounting.stages[-1] for task in plan.application_tasks) + and all(predecessor.prerequisite in plan.accounting.tasks for predecessor in expected_stage) + and plan.application_predecessors == (*expected_scope, *expected_stage) + and observed_explicit == set(expected_scope) + ) + + +def _valid_phase6_handoffs( + phase6: _Phase6Plan, + handoffs: tuple[object, ...], + contract: _Phase7StableSubstituteContract, +) -> bool: + if phase6.profile_version is not _Phase6ProfileVersion.SUBSTITUTE_V1: + return False + if len(handoffs) != len(phase6.components) or not all( + isinstance(handoff, _Phase6SubstituteHandoff) for handoff in handoffs + ): + return False + typed_handoffs = tuple(handoff for handoff in handoffs if isinstance(handoff, _Phase6SubstituteHandoff)) + expected_components = {component.key for component in phase6.components} + if ( + len({handoff.component for handoff in typed_handoffs}) != len(typed_handoffs) + or {handoff.component for handoff in typed_handoffs} != expected_components + ): + return False + contract_roles = {role.name for role in contract.roles} + expected_datums = {datum.id for datum in phase6.accounting.datums} + terminal_datums: list[_DatumId] = [] + for handoff in typed_handoffs: + if ( + not _is_admitted_substitute_handoff(handoff, phase6) + or handoff.result_version != contract.phase6_result_version + or handoff.policy_version != contract.phase6_policy_version + or handoff.policy_digest != contract.phase6_policy_digest + or any( + not isinstance(mention.role_result, _ClassifiedRole) + or mention.role_result.role.value not in contract_roles + for mention in handoff.resolved.mentions + ) + ): + return False + terminal_datums.extend(handoff.terminal_evidence.datum_ids) + return len(set(terminal_datums)) == len(terminal_datums) and set(terminal_datums) == expected_datums + + +def _materialize_scope_manifests( + scopes: tuple[_CoherenceScope, ...], + phase6: _Phase6Plan, + handoffs: tuple[object, ...], + relations: tuple[object, ...], + contract: _Phase7StableSubstituteContract, +) -> tuple[_ScopeManifest, ...] | _Phase7Rejected: + scope_members = _canonical_scope_members(scopes, phase6) + typed_handoffs = tuple(handoff for handoff in handoffs if isinstance(handoff, _Phase6SubstituteHandoff)) + pre_slots = _compile_pre_slots(scope_members, phase6, typed_handoffs, contract) + relation_result = _compile_pre_slot_relations(relations, pre_slots) + if isinstance(relation_result, _Phase7Rejected): + return relation_result + slots = _materialize_slots(pre_slots) + compiled_relations = _bind_relations(relation_result, pre_slots, slots) + return _seal_scope_manifests(scope_members, pre_slots, slots, compiled_relations, contract) + + +def _canonical_scope_members( + scopes: tuple[_CoherenceScope, ...], + phase6: _Phase6Plan, +) -> tuple[tuple[_Phase7ScopeId, tuple[_DatumId, ...]], ...]: + datum_by_value = {datum.id.value: datum.id for datum in phase6.accounting.datums} + position = {datum.id.value: index for index, datum in enumerate(phase6.accounting.datums)} + canonical = sorted( + (frozenset(member.value for member in scope.members) for scope in scopes), + key=lambda values: tuple(sorted(position[value] for value in values)), + ) + scope_members: list[tuple[_Phase7ScopeId, tuple[_DatumId, ...]]] = [] + for values in canonical: + members = tuple(datum_by_value[value] for value in sorted(values, key=position.__getitem__)) + scope_members.append((_Phase7ScopeId(), members)) + return tuple(scope_members) + + +def _materialize_slots(pre_slots: tuple[_PreSlot, ...]) -> tuple[_ReplacementSlot, ...]: + return tuple( + _ReplacementSlot( + _ReplacementSlotId(), + pre_slot.cluster_id, + pre_slot.role, + pre_slot.format, + pre_slot.mask, + pre_slot.mention_ids, + ) + for pre_slot in pre_slots + ) + + +def _bind_relations( + relations: tuple[_PreSlotRelation, ...], + pre_slots: tuple[_PreSlot, ...], + slots: tuple[_ReplacementSlot, ...], +) -> tuple[_CompiledRelation, ...]: + slot_by_pre_slot = dict(zip(pre_slots, slots, strict=True)) + compiled_relations = tuple( + _CompiledRelation( + relation.version, + tuple(slot_by_pre_slot[pre_slot].id for pre_slot in relation.upstream), + slot_by_pre_slot[relation.downstream].id, + ) + for relation in relations + ) + slot_position = {slot.id: index for index, slot in enumerate(slots)} + return tuple( + sorted( + compiled_relations, + key=lambda relation: ( + slot_position[relation.downstream], + tuple(slot_position[slot_id] for slot_id in relation.upstream), + ), + ) + ) + + +def _seal_scope_manifests( + scope_members: tuple[tuple[_Phase7ScopeId, tuple[_DatumId, ...]], ...], + pre_slots: tuple[_PreSlot, ...], + slots: tuple[_ReplacementSlot, ...], + relations: tuple[_CompiledRelation, ...], + contract: _Phase7StableSubstituteContract, +) -> tuple[_ScopeManifest, ...] | _Phase7Rejected: + manifests: list[_ScopeManifest] = [] + for scope_id, members in scope_members: + manifest = _seal_scope_manifest(scope_id, members, pre_slots, slots, relations, contract) + if isinstance(manifest, _Phase7Rejected): + return manifest + manifests.append(manifest) + return tuple(manifests) + + +def _seal_scope_manifest( + scope_id: _Phase7ScopeId, + members: tuple[_DatumId, ...], + pre_slots: tuple[_PreSlot, ...], + slots: tuple[_ReplacementSlot, ...], + relations: tuple[_CompiledRelation, ...], + contract: _Phase7StableSubstituteContract, +) -> _ScopeManifest | _Phase7Rejected: + scope_slots = tuple(slot for pre, slot in zip(pre_slots, slots, strict=True) if pre.scope_id is scope_id) + pairs = tuple( + _RequiredDistinctPair(left.id, right.id) + for index, left in enumerate(scope_slots) + for right in scope_slots[index + 1 :] + ) + slot_ids = {slot.id for slot in scope_slots} + scope_relations = tuple(relation for relation in relations if relation.downstream in slot_ids) + limits = dict(contract.count_limits) + if ( + len(scope_slots) > limits["max_slots_per_scope"] + or len(pairs) > limits["max_distinct_pairs_per_scope"] + or len({slot.cluster_id for slot in scope_slots}) > limits["max_clusters_per_scope"] + or sum(len(slot.mention_ids) for slot in scope_slots) > limits["max_mentions_per_scope"] + or len(scope_relations) > limits["max_relations_per_scope"] + ): + return _Phase7Rejected(_Phase7AdmissionCode.LIMIT_EXCEEDED) + candidate = _ScopeManifest(scope_id, members, scope_slots, pairs, scope_relations) + snapshot = _scope_manifest_snapshot(candidate) + if snapshot is None: + return _Phase7Rejected(_Phase7AdmissionCode.INVALID_INPUT) + proof = _ScopeManifestProof(_SCOPE_MANIFEST_SEAL, snapshot) + return _ScopeManifest(scope_id, members, scope_slots, pairs, scope_relations, proof) + + +def _compile_pre_slots( + scopes: tuple[tuple[_Phase7ScopeId, tuple[_DatumId, ...]], ...], + phase6: _Phase6Plan, + handoffs: tuple[_Phase6SubstituteHandoff, ...], + contract: _Phase7StableSubstituteContract, +) -> tuple[_PreSlot, ...]: + scope_by_datum = {member: scope_id for scope_id, members in scopes for member in members} + role_by_name = {role.name: role for role in contract.roles} + component_position = {component.key: index for index, component in enumerate(phase6.components)} + ordered_handoffs = sorted(handoffs, key=lambda handoff: component_position[handoff.component]) + keys: list[tuple[_Phase7ScopeId, _ClusterId, str]] = [] + mention_ids: dict[tuple[_Phase7ScopeId, _ClusterId, str], list[_MentionId]] = {} + for handoff in ordered_handoffs: + for resolved in handoff.resolved.mentions: + role_result = resolved.role_result + if not isinstance(role_result, _ClassifiedRole): + continue + key = ( + scope_by_datum[resolved.mention.target_datum_id], + resolved.cluster_id, + role_result.role.value, + ) + if key not in mention_ids: + keys.append(key) + mention_ids[key] = [] + mention_ids[key].append(resolved.mention.id) + return tuple( + _PreSlot( + scope_id, + cluster_id, + role, + role_by_name[role].format, + role_by_name[role].mask, + tuple(mention_ids[(scope_id, cluster_id, role)]), + ) + for scope_id, cluster_id, role in keys + ) + + +def _total_limits_exceeded( + phase6: _Phase6Plan, + handoffs: tuple[object, ...], + declarations: _Phase7Declarations, + contract: _Phase7StableSubstituteContract, +) -> bool: + counts = dict(contract.count_limits) + bytes_limits = dict(contract.byte_limits) + if ( + len(phase6.accounting.datums) > counts["max_datums_per_invocation"] + or len(declarations.coherence_scopes) > counts["max_scopes_per_invocation"] + or any( + isinstance(scope, _CoherenceScope) + and isinstance(scope.members, tuple) + and len(scope.members) > counts["max_scope_members"] + for scope in declarations.coherence_scopes + ) + ): + return True + if _context_limits_exceeded(phase6, declarations.coherence_scopes, counts, bytes_limits): + return True + return _handoff_limits_exceeded(handoffs, counts, bytes_limits) + + +def _handoff_limits_exceeded( + handoffs: tuple[object, ...], + counts: dict[str, int], + bytes_limits: dict[str, int], +) -> bool: + clusters: set[_ClusterId] = set() + slot_keys: set[tuple[_ClusterId, str]] = set() + mention_count = 0 + original_bytes = 0 + for handoff in handoffs: + if not isinstance(handoff, _Phase6SubstituteHandoff): + continue + clusters.update(cluster.id for cluster in handoff.resolved.clustered.clusters) + for resolved in handoff.resolved.mentions: + mention_count += 1 + source = resolved.mention.source_slice + detector_label = resolved.mention.detector_label + try: + source_bytes = len(source.encode("utf-8")) + detector_label_bytes = len(detector_label.encode("utf-8")) + except (AttributeError, UnicodeEncodeError): + continue + original_bytes += source_bytes + if ( + source_bytes > bytes_limits["max_original_value_bytes"] + or detector_label_bytes > bytes_limits["max_detector_label_bytes"] + ): + return True + if isinstance(resolved.role_result, _ClassifiedRole): + slot_keys.add((resolved.cluster_id, resolved.role_result.role.value)) + return ( + len(clusters) > counts["max_clusters_per_invocation"] + or len(slot_keys) > counts["max_slots_per_invocation"] + or mention_count > counts["max_mentions_per_invocation"] + or original_bytes > bytes_limits["max_all_original_value_bytes"] + ) + + +def _context_limits_exceeded( + phase6: _Phase6Plan, + scopes: tuple[_CoherenceScope, ...], + counts: dict[str, int], + bytes_limits: dict[str, int], +) -> bool: + if not all( + isinstance(scope, _CoherenceScope) + and isinstance(scope.members, tuple) + and all(isinstance(member, _DatumId) for member in scope.members) + for scope in scopes + ): + return False + projection_by_target = {projection.target_datum_id: projection for projection in phase6.context.projections} + datum_by_id = {datum.id: datum for datum in (*phase6.accounting.datums, *phase6.context.context_only_datums)} + target_by_value = {datum.id.value: datum.id for datum in phase6.accounting.datums} + for scope in scopes: + context_ids = _context_ids_for_scope(scope, target_by_value, projection_by_target) + if context_ids is None: + continue + if len(context_ids) > counts["max_context_fragments_per_scope"]: + return True + sizes = _context_sizes(context_ids, datum_by_id) + if ( + any(size > bytes_limits["max_context_fragment_bytes"] for size in sizes) + or sum(sizes) > bytes_limits["max_context_bytes_per_scope"] + ): + return True + return False + + +def _context_ids_for_scope( + scope: _CoherenceScope, + target_by_value: dict[str, _DatumId], + projection_by_target: dict[_DatumId, _CompiledContextProjection], +) -> set[_DatumId] | None: + target_ids = tuple(target_by_value.get(member.value) for member in scope.members) + if any(target_id is None for target_id in target_ids): + return None + return { + context_id + for target_id in target_ids + if target_id is not None and target_id in projection_by_target + for context_id in projection_by_target[target_id].context_datum_ids + } + + +def _context_sizes(context_ids: set[_DatumId], datum_by_id: dict[_DatumId, _TextDatum]) -> tuple[int, ...]: + sizes: list[int] = [] + for context_id in context_ids: + datum = datum_by_id.get(context_id) + if datum is None: + continue + try: + sizes.append(len(datum.text.encode("utf-8"))) + except UnicodeEncodeError: + continue + return tuple(sizes) + + +def _compile_pre_slot_relations( + declarations: tuple[object, ...], + candidates: tuple[_PreSlot, ...], +) -> tuple[_PreSlotRelation, ...] | _Phase7Rejected: + compiled: list[_PreSlotRelation] = [] + for declaration in declarations: + if ( + not isinstance(declaration, _RelationDeclaration) + or declaration.version != "email_from_name/v1" + or not isinstance(declaration.upstream, tuple) + or not 1 <= len(declaration.upstream) <= 2 + or not isinstance(declaration.downstream, _ClusterRoleSelector) + ): + return _Phase7Rejected(_Phase7AdmissionCode.UNSUPPORTED_RELATION) + if len(set(declaration.upstream)) != len(declaration.upstream): + return _Phase7Rejected(_Phase7AdmissionCode.SELECTOR_AMBIGUOUS) + resolved_upstream: list[_PreSlot] = [] + for selector in declaration.upstream: + resolved = _resolve_selector(selector, candidates) + if isinstance(resolved, _Phase7Rejected): + return resolved + resolved_upstream.append(resolved) + downstream = _resolve_selector(declaration.downstream, candidates) + if isinstance(downstream, _Phase7Rejected): + return downstream + selected = (*resolved_upstream, downstream) + if len({item.scope_id for item in selected}) != 1: + return _Phase7Rejected(_Phase7AdmissionCode.CROSS_SCOPE_RELATION) + if ( + any(item.role not in {"person_family_name", "person_given_name"} for item in resolved_upstream) + or downstream.role != "email_address" + ): + return _Phase7Rejected(_Phase7AdmissionCode.RELATION_ROLE_MISMATCH) + compiled.append(_PreSlotRelation(declaration.version, tuple(resolved_upstream), downstream)) + return tuple(compiled) + + +def _resolve_selector( + selector: object, + candidates: tuple[_PreSlot, ...], +) -> _PreSlot | _Phase7Rejected: + if not isinstance(selector, _ClusterRoleSelector) or selector.version != "cluster_role/v1": + return _Phase7Rejected(_Phase7AdmissionCode.UNSUPPORTED_SELECTOR) + if not isinstance(selector.cluster, _ClusterId) or not isinstance(selector.role, str): + return _Phase7Rejected(_Phase7AdmissionCode.SELECTOR_MISSING) + matches = tuple( + candidate + for candidate in candidates + if candidate.cluster_id is selector.cluster and candidate.role == selector.role + ) + if not matches: + return _Phase7Rejected(_Phase7AdmissionCode.SELECTOR_MISSING) + if len(matches) != 1: + return _Phase7Rejected(_Phase7AdmissionCode.SELECTOR_AMBIGUOUS) + return matches[0] + + +def _is_admitted_scope_manifest(value: object) -> bool: + return ( + isinstance(value, _ScopeManifest) + and value._proof is not None + and value._proof.seal is _SCOPE_MANIFEST_SEAL + and value._proof.snapshot == _scope_manifest_snapshot(value) + ) + + +def _scope_manifest_snapshot(manifest: _ScopeManifest) -> tuple[object, ...] | None: + try: + return ( + manifest.id, + tuple(member.value for member in manifest.members), + manifest.slots, + manifest.required_pairs, + manifest.relations, + ) + except (AttributeError, TypeError): + return None + + +def _phase7_plan_snapshot(plan: _Phase7Plan) -> tuple[object, ...] | None: + try: + return ( + tuple(manifest._proof for manifest in plan.manifests), + plan.accounting._proof, + # The accounting-plan proof alone authenticates its own source + # compilation, not this Phase 7 expansion. Seal the concrete + # Phase 6 prefix and appended scope-task sequence as well. + tuple(plan.accounting.tasks), + tuple(plan.scope_tasks), + tuple(plan.application_tasks), + tuple(plan.application_predecessors), + ) + except (AttributeError, TypeError): + return None + + +def _validate_partition( + scopes: tuple[_CoherenceScope, ...], + datums: tuple[_TextDatum, ...], +) -> _Phase7Rejected | None: + compiled = _compile_scope_semantics(scopes) + if isinstance(compiled, _Phase7Rejected): + return compiled + semantic_scopes = compiled + known = {datum.id.value for datum in datums} + if any( + not isinstance(member, _DatumId) or not isinstance(member.value, str) or member.value not in known + for scope in scopes + for member in scope.members + ): + return _Phase7Rejected(_Phase7AdmissionCode.UNKNOWN_SCOPE_DATUM) + covered = set().union(*semantic_scopes) if semantic_scopes else set() + if covered != known: + return _Phase7Rejected(_Phase7AdmissionCode.SCOPE_COVERAGE_GAP) + overlap = _unsupported_scope_relationship(semantic_scopes) + return overlap + + +def _compile_scope_semantics( + scopes: tuple[_CoherenceScope, ...], +) -> tuple[frozenset[object], ...] | _Phase7Rejected: + if not all(isinstance(scope, _CoherenceScope) and isinstance(scope.members, tuple) for scope in scopes): + return _Phase7Rejected(_Phase7AdmissionCode.INVALID_INPUT) + if any(not scope.members for scope in scopes): + return _Phase7Rejected(_Phase7AdmissionCode.EMPTY_SCOPE) + scope_values = tuple(tuple(getattr(member, "value", None) for member in scope.members) for scope in scopes) + semantic_scopes = tuple(frozenset(values) for values in scope_values) + if len(set(semantic_scopes)) != len(semantic_scopes): + return _Phase7Rejected(_Phase7AdmissionCode.DUPLICATE_SCOPE) + if any(len(set(values)) != len(values) for values in scope_values): + return _Phase7Rejected(_Phase7AdmissionCode.DUPLICATE_SCOPE_MEMBER) + return semantic_scopes + + +def _unsupported_scope_relationship( + semantic_scopes: tuple[frozenset[object], ...], +) -> _Phase7Rejected | None: + for index, left in enumerate(semantic_scopes): + for right in semantic_scopes[index + 1 :]: + if left & right and not (left < right or right < left): + return _Phase7Rejected(_Phase7AdmissionCode.SCOPE_OVERLAP) + if any( + left < right or right < left + for index, left in enumerate(semantic_scopes) + for right in semantic_scopes[index + 1 :] + ): + return _Phase7Rejected(_Phase7AdmissionCode.UNSUPPORTED_SCOPE_NESTING) + return None diff --git a/src/anonymizer/engine/execution/phase7_application.py b/src/anonymizer/engine/execution/phase7_application.py new file mode 100644 index 00000000..39280c19 --- /dev/null +++ b/src/anonymizer/engine/execution/phase7_application.py @@ -0,0 +1,270 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Anchored, non-cascading application for private Phase 7 Substitute scopes.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum + +from anonymizer.engine.execution.graph import _DatumId +from anonymizer.engine.execution.mention_admission import _MentionId, _MentionTargetToken +from anonymizer.engine.execution.phase7_validation import ( + _index_scope_sources, + _is_validated_bundle, + _ValidatedBundle, +) + + +class _PrivatePhase7ApplicationValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Phase 7 application values are not serializable") + + +class _ApplicationRejectionCode(str, Enum): + INVALID_APPLICATION = "invalid_application" + + +@dataclass(frozen=True, slots=True, repr=False) +class _SubstitutePatch(_PrivatePhase7ApplicationValue): + mention_id: _MentionId + target: _MentionTargetToken + start: int + end: int + source_slice: str + replacement: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _AppliedDatum(_PrivatePhase7ApplicationValue): + datum_id: _DatumId + output: str + applied: bool + + +@dataclass(frozen=True, slots=True, repr=False) +class _AppliedScopeProof(_PrivatePhase7ApplicationValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _AppliedScope(_PrivatePhase7ApplicationValue): + bundle: _ValidatedBundle + datums: tuple[_AppliedDatum, ...] + patches: tuple[_SubstitutePatch, ...] + _proof: _AppliedScopeProof | None = field(default=None, compare=False) + + +@dataclass(frozen=True, slots=True, repr=False) +class _ApplicationRejected(_PrivatePhase7ApplicationValue): + code: _ApplicationRejectionCode + + +_APPLIED_SCOPE_SEAL = object() + + +def _materialize_substitute_patches( + bundle: object, +) -> tuple[_SubstitutePatch, ...] | _ApplicationRejected: + """Bind every admitted mention token to its authoritative source interval.""" + if not isinstance(bundle, _ValidatedBundle) or not _is_validated_bundle(bundle): + return _rejected() + source_index = _index_scope_sources(bundle.manifest, bundle.handoffs) + if source_index is None: + return _rejected() + mention_by_id = dict(source_index.mentions) + target_by_datum = {target.datum_id: target for target in source_index.targets} + assignment_by_token = {assignment.token: assignment.value for assignment in bundle.assignments} + patches: list[_SubstitutePatch] = [] + for slot in bundle.manifest.slots: + replacement = assignment_by_token.get(slot.id) + if replacement is None: + return _rejected() + for mention_id in slot.mention_ids: + mention = mention_by_id.get(mention_id) + if mention is None: + return _rejected() + target = target_by_datum.get(mention.target_datum_id) + if target is None: + return _rejected() + patches.append( + _SubstitutePatch( + mention.id, + target.token, + mention.start, + mention.end, + mention.source_slice, + replacement, + ) + ) + return tuple(patches) + + +def _apply_substitute_patches( + bundle: object, + patches: object, +) -> _AppliedScope | _ApplicationRejected: + """Validate and apply one complete patch set over immutable source text.""" + if not isinstance(bundle, _ValidatedBundle) or not _is_validated_bundle(bundle): + return _rejected() + expected = _materialize_substitute_patches(bundle) + if isinstance(expected, _ApplicationRejected): + return expected + if not _patches_are_exact(expected, patches): + return _rejected() + datums = _reconstruct_scope(bundle, expected) + if isinstance(datums, _ApplicationRejected): + return datums + + values = (bundle, datums, expected) + candidate = _AppliedScope(*values) + snapshot = _applied_scope_snapshot(candidate) + if snapshot is None: + return _rejected() + return _AppliedScope(*values, _AppliedScopeProof(_APPLIED_SCOPE_SEAL, snapshot)) + + +def _patches_are_exact(expected: tuple[_SubstitutePatch, ...], patches: object) -> bool: + if not isinstance(patches, tuple) or not all(isinstance(patch, _SubstitutePatch) for patch in patches): + return False + typed_patches = tuple(patch for patch in patches if isinstance(patch, _SubstitutePatch)) + expected_by_mention = {patch.mention_id: patch for patch in expected} + observed_mentions = tuple(patch.mention_id for patch in typed_patches) + if len(set(observed_mentions)) != len(observed_mentions) or set(observed_mentions) != set(expected_by_mention): + return False + observed_by_mention = {patch.mention_id: patch for patch in typed_patches} + return all( + _is_exact_patch(observed_by_mention[expected_patch.mention_id], expected_patch) for expected_patch in expected + ) + + +def _apply_substitute_datum( + bundle: object, + patches: object, + datum_id: object, +) -> _AppliedDatum | _ApplicationRejected: + """Apply an exact planned bundle to one datum without widening a local fault.""" + if ( + not isinstance(bundle, _ValidatedBundle) + or not _is_validated_bundle(bundle) + or not isinstance(datum_id, _DatumId) + ): + return _rejected() + expected = _materialize_substitute_patches(bundle) + if isinstance(expected, _ApplicationRejected) or not _patches_are_exact(expected, patches): + return _rejected() + source_index = _index_scope_sources(bundle.manifest, bundle.handoffs) + if source_index is None: + return _rejected() + targets = tuple(target for target in source_index.targets if target.datum_id == datum_id) + if len(targets) != 1: + return _rejected() + target = targets[0] + target_patches = tuple(patch for patch in expected if patch.target is target.token) + output = _reconstruct_source(target.text, target_patches) + if output is None: + return _rejected() + return _AppliedDatum(datum_id, output, bool(target_patches)) + + +def _reconstruct_scope( + bundle: _ValidatedBundle, + patches: tuple[_SubstitutePatch, ...], +) -> tuple[_AppliedDatum, ...] | _ApplicationRejected: + source_index = _index_scope_sources(bundle.manifest, bundle.handoffs) + if source_index is None: + return _rejected() + datums: list[_AppliedDatum] = [] + for target in source_index.targets: + applied = _apply_substitute_datum(bundle, patches, target.datum_id) + if not isinstance(applied, _AppliedDatum): + return _rejected() + datums.append(applied) + return tuple(datums) + + +def _is_applied_scope(value: object) -> bool: + if ( + not isinstance(value, _AppliedScope) + or value._proof is None + or value._proof.seal is not _APPLIED_SCOPE_SEAL + or not _is_validated_bundle(value.bundle) + or value._proof.snapshot != _applied_scope_snapshot(value) + or tuple(datum.datum_id for datum in value.datums) != value.bundle.manifest.members + ): + return False + expected = _materialize_substitute_patches(value.bundle) + return isinstance(expected, tuple) and value.patches == expected + + +def _is_exact_patch(observed: _SubstitutePatch, expected: _SubstitutePatch) -> bool: + return ( + observed.mention_id is expected.mention_id + and observed.target is expected.target + and type(observed.start) is int + and observed.start == expected.start + and type(observed.end) is int + and observed.end == expected.end + and type(observed.source_slice) is str + and observed.source_slice == expected.source_slice + and type(observed.replacement) is str + and observed.replacement == expected.replacement + ) + + +def _reconstruct_source(text: object, patches: object) -> str | None: + if ( + type(text) is not str + or not isinstance(patches, tuple) + or not all(isinstance(patch, _SubstitutePatch) for patch in patches) + ): + return None + cursor = 0 + parts: list[str] = [] + for patch in sorted(patches, key=lambda item: (item.start, item.end)): + if ( + type(patch.start) is not int + or type(patch.end) is not int + or patch.start < cursor + or patch.start < 0 + or patch.end <= patch.start + or patch.end > len(text) + or type(patch.source_slice) is not str + or text[patch.start : patch.end] != patch.source_slice + or type(patch.replacement) is not str + ): + return None + parts.extend((text[cursor : patch.start], patch.replacement)) + cursor = patch.end + parts.append(text[cursor:]) + return "".join(parts) + + +def _applied_scope_snapshot(value: _AppliedScope) -> tuple[object, ...] | None: + try: + return ( + value.bundle, + tuple((datum.datum_id, datum.output, datum.applied) for datum in value.datums), + tuple( + ( + patch.mention_id, + patch.target, + patch.start, + patch.end, + patch.source_slice, + patch.replacement, + ) + for patch in value.patches + ), + ) + except (AttributeError, TypeError): + return None + + +def _rejected() -> _ApplicationRejected: + return _ApplicationRejected(_ApplicationRejectionCode.INVALID_APPLICATION) diff --git a/src/anonymizer/engine/execution/phase7_contract.py b/src/anonymizer/engine/execution/phase7_contract.py new file mode 100644 index 00000000..3f3af9a1 --- /dev/null +++ b/src/anonymizer/engine/execution/phase7_contract.py @@ -0,0 +1,338 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Frozen owner contract for the private Phase 7 stable-Substitute profile.""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass, field +from enum import Enum +from importlib.resources import files +from typing import TypeAlias, cast + +from anonymizer.engine.constants import DEFAULT_ENTITY_LABELS + +_FrozenJson: TypeAlias = None | bool | int | str | tuple["_FrozenJson", ...] | tuple[tuple[str, "_FrozenJson"], ...] + + +class _PrivatePhase7ContractValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Phase 7 contract values are not serializable") + + +class _Phase7ContractVersion(str, Enum): + V1 = "anonymizer-phase7-stable-substitute/v1" + + +class _Phase7ContractRejectionCode(str, Enum): + INVALID_CONTRACT = "contract_invalid" + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7Role(_PrivatePhase7ContractValue): + name: str + format: str + mask: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7ContractProof(_PrivatePhase7ContractValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7StableSubstituteContract(_PrivatePhase7ContractValue): + version: _Phase7ContractVersion + digest: str + phase6_result_version: str + phase6_policy_version: str + phase6_policy_digest: str + roles: tuple[_Phase7Role, ...] + selectors: tuple[str, ...] + relations: tuple[str, ...] + formats: tuple[str, ...] + masks: tuple[str, ...] + count_limits: tuple[tuple[str, int], ...] + byte_limits: tuple[tuple[str, int], ...] + corpus_version: str + corpus_case_count: int + corpus_digest: str + _source_snapshot: _FrozenJson = field(compare=False) + _proof: _Phase7ContractProof | None = field(default=None, compare=False) + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7ContractRejected(_PrivatePhase7ContractValue): + code: _Phase7ContractRejectionCode = _Phase7ContractRejectionCode.INVALID_CONTRACT + + +_PHASE7_CONTRACT_SEAL = object() +_PHASE7_CONTRACT_RESOURCE = "phase7_stable_substitute_contract.json" +_PHASE7_POLICY_RESOURCE = "phase6_substitute_role_policy.json" +_PHASE7_CORPUS_RESOURCE = "phase7_owner_contract_corpus.json" +_PHASE7_CONTRACT_DIGEST = "3755832ecc64fe6e9dbeccc136c40020e5158446e5c74d6dca2392efdbb006bb" +_PHASE7_POLICY_DIGEST = "c27580bd2cc4051bdd11b63a91391f8995bdef1ed2052534623cdd3160318ef8" +_PHASE7_CORPUS_DIGEST = "5ba1e69c7836a428b16e9d2ac7fc0cf3fbb445fd0ca72c1d0a8a049194115123" +_ENVELOPE_SCHEMA_VERSION = "anonymizer-phase7-owner-contract-envelope/v1" +_DIGEST_ALGORITHM = "sha256_of_UTF8_compact_sorted_key_JSON_of_contract_member_with_no_trailing_newline" + + +def _load_phase7_contract() -> _Phase7StableSubstituteContract | _Phase7ContractRejected: + """Load the exact owner-frozen contract and its companion resources.""" + try: + package = files("anonymizer.engine.execution") + envelope = _parse_json(package.joinpath(_PHASE7_CONTRACT_RESOURCE).read_text(encoding="utf-8")) + policy = _parse_json(package.joinpath(_PHASE7_POLICY_RESOURCE).read_text(encoding="utf-8")) + corpus = _parse_json(package.joinpath(_PHASE7_CORPUS_RESOURCE).read_text(encoding="utf-8")) + return _compile_phase7_contract(envelope, policy, corpus) + except (OSError, TypeError, UnicodeEncodeError, ValueError): + return _Phase7ContractRejected() + + +def _compile_phase7_contract( + envelope: object, + policy: object, + corpus: object, +) -> _Phase7StableSubstituteContract | _Phase7ContractRejected: + """Validate and seal only the exact contract approved by the owners.""" + try: + if type(envelope) is not dict or set(envelope) != { + "contract", + "digest", + "digest_algorithm", + "schema_version", + }: + raise TypeError + envelope_dict = cast(dict[str, object], envelope) + contract = envelope_dict["contract"] + digest = envelope_dict["digest"] + if ( + type(contract) is not dict + or type(digest) is not str + or envelope_dict["digest_algorithm"] != _DIGEST_ALGORITHM + or envelope_dict["schema_version"] != _ENVELOPE_SCHEMA_VERSION + or _canonical_digest(contract) != digest + or digest != _PHASE7_CONTRACT_DIGEST + ): + raise ValueError + contract_dict = cast(dict[str, object], contract) + if contract_dict["status"] != "frozen_owner_contract": + raise ValueError + version = _Phase7ContractVersion(contract_dict["version"]) + + roles_payload = _require_dict(contract_dict, "roles") + roles = tuple( + _Phase7Role( + name, + _require_string(_require_dict(roles_payload, name), "format"), + _require_string(_require_dict(roles_payload, name), "mask"), + ) + for name in sorted(roles_payload) + ) + formats = _closed_keys(contract_dict, "formats", excluded=()) + masks = _closed_keys(contract_dict, "masks", excluded=("unknown",)) + selectors = _closed_keys(contract_dict, "selectors", excluded=("unknown",)) + relations = _closed_keys(contract_dict, "relations", excluded=("unknown", "wildcard_constraints")) + count_limits = _positive_integer_items(_require_dict(_require_dict(contract_dict, "limits"), "counts")) + byte_limits = _positive_integer_items(_require_dict(_require_dict(contract_dict, "limits"), "bytes")) + + handoff = _require_dict(contract_dict, "phase6_handoff") + policy_contract = _require_dict(handoff, "substitute_policy") + _validate_policy(policy, policy_contract, roles) + corpus_contract = _require_dict(contract_dict, "oracle_contract_corpus") + _validate_corpus(corpus, corpus_contract) + + values = ( + version, + digest, + _require_string(handoff, "required_result_version"), + _require_string(policy_contract, "version"), + _require_string(policy_contract, "digest"), + roles, + selectors, + relations, + formats, + masks, + count_limits, + byte_limits, + _require_string(corpus_contract, "version"), + _require_integer(corpus_contract, "case_count"), + _require_string(corpus_contract, "digest"), + (_freeze_json(contract_dict), _freeze_json(policy), _freeze_json(corpus)), + ) + candidate = _Phase7StableSubstituteContract(*values) + snapshot = _phase7_contract_snapshot(candidate) + if snapshot is None: + raise TypeError + return _Phase7StableSubstituteContract( + *values, + _Phase7ContractProof(_PHASE7_CONTRACT_SEAL, snapshot), + ) + except (KeyError, TypeError, UnicodeEncodeError, ValueError): + return _Phase7ContractRejected() + + +def _is_admitted_phase7_contract(value: object) -> bool: + if not isinstance(value, _Phase7StableSubstituteContract) or value._proof is None: + return False + return ( + value._proof.seal is _PHASE7_CONTRACT_SEAL + and value.digest == _PHASE7_CONTRACT_DIGEST + and value._proof.snapshot == _phase7_contract_snapshot(value) + ) + + +def _validate_policy( + policy: object, + expected: dict[str, object], + roles: tuple[_Phase7Role, ...], +) -> None: + if type(policy) is not dict: + raise TypeError + policy_dict = cast(dict[str, object], policy) + if ( + set(policy_dict) != {"dispositions", "result_version", "version"} + or _canonical_digest(policy_dict) != _PHASE7_POLICY_DIGEST + or _require_string(expected, "digest") != _PHASE7_POLICY_DIGEST + or policy_dict["version"] != expected["version"] + or policy_dict["result_version"] != "phase6-role-result/v1" + ): + raise ValueError + dispositions = _require_dict(policy_dict, "dispositions") + if set(dispositions) != set(DEFAULT_ENTITY_LABELS) or len(dispositions) != len(DEFAULT_ENTITY_LABELS): + raise ValueError + role_names = {role.name for role in roles} + classified = {label: role for label, role in dispositions.items() if role is not None} + if ( + any(type(role) is not str or role not in role_names for role in classified.values()) + or set(classified.values()) != role_names + or sorted(classified) != expected["supported_detector_labels"] + or _require_integer(expected, "detector_label_count") != len(dispositions) + ): + raise ValueError + + +def _validate_corpus(corpus: object, expected: dict[str, object]) -> None: + if type(corpus) is not dict: + raise TypeError + corpus_dict = cast(dict[str, object], corpus) + if ( + set(corpus_dict) != {"cases", "version"} + or _canonical_digest(corpus_dict) != _PHASE7_CORPUS_DIGEST + or _require_string(expected, "digest") != _PHASE7_CORPUS_DIGEST + or corpus_dict["version"] != expected["version"] + ): + raise ValueError + cases = corpus_dict["cases"] + if type(cases) is not list or len(cases) != _require_integer(expected, "case_count"): + raise ValueError + case_list = cast(list[object], cases) + case_ids: set[str] = set() + for case in case_list: + if type(case) is not dict or set(case) != {"expected", "id"}: + raise TypeError + case_dict = cast(dict[str, object], case) + case_id = _require_string(case_dict, "id") + _require_string(case_dict, "expected") + if case_id in case_ids: + raise ValueError + case_ids.add(case_id) + + +def _parse_json(text: object) -> object: + if type(text) is not str: + raise TypeError + return json.loads(text, object_pairs_hook=_object_without_duplicates) + + +def _object_without_duplicates(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + if key in result: + raise ValueError("duplicate JSON key") + result[key] = value + return result + + +def _canonical_digest(value: object) -> str: + encoded = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def _closed_keys(container: dict[str, object], key: str, *, excluded: tuple[str, ...]) -> tuple[str, ...]: + values = _require_dict(container, key) + return tuple(sorted(name for name in values if name not in excluded)) + + +def _positive_integer_items(values: dict[str, object]) -> tuple[tuple[str, int], ...]: + result: list[tuple[str, int]] = [] + for key in sorted(values): + value = values[key] + if type(value) is not int or value <= 0: + raise TypeError + result.append((key, value)) + return tuple(result) + + +def _require_dict(container: dict[str, object], key: str) -> dict[str, object]: + value = container[key] + if type(value) is not dict: + raise TypeError + return cast(dict[str, object], value) + + +def _require_string(container: dict[str, object], key: str) -> str: + value = container[key] + if type(value) is not str: + raise TypeError + return value + + +def _require_integer(container: dict[str, object], key: str) -> int: + value = container[key] + if type(value) is not int: + raise TypeError + return value + + +def _freeze_json(value: object) -> _FrozenJson: + if value is None or type(value) in {bool, int, str}: + return cast(None | bool | int | str, value) + if type(value) is list: + return tuple(_freeze_json(item) for item in cast(list[object], value)) + if type(value) is dict: + mapping = cast(dict[str, object], value) + if any(type(key) is not str for key in mapping): + raise TypeError + return tuple((key, _freeze_json(mapping[key])) for key in sorted(mapping)) + raise TypeError + + +def _phase7_contract_snapshot(contract: _Phase7StableSubstituteContract) -> tuple[object, ...] | None: + try: + return ( + contract.version.value, + contract.digest, + contract.phase6_result_version, + contract.phase6_policy_version, + contract.phase6_policy_digest, + tuple((role.name, role.format, role.mask) for role in contract.roles), + contract.selectors, + contract.relations, + contract.formats, + contract.masks, + contract.count_limits, + contract.byte_limits, + contract.corpus_version, + contract.corpus_case_count, + contract.corpus_digest, + contract._source_snapshot, + ) + except (AttributeError, TypeError): + return None diff --git a/src/anonymizer/engine/execution/phase7_ndd_backend.py b/src/anonymizer/engine/execution/phase7_ndd_backend.py new file mode 100644 index 00000000..9cc589ca --- /dev/null +++ b/src/anonymizer/engine/execution/phase7_ndd_backend.py @@ -0,0 +1,650 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Bounded NDD candidate proposals for private Phase 7 Substitute scopes.""" + +from __future__ import annotations + +import json +import secrets +from collections.abc import Callable +from dataclasses import dataclass +from enum import Enum +from typing import TypeGuard, TypeVar + +import pandas as pd +from data_designer.config.column_configs import LLMStructuredColumnConfig +from pydantic import BaseModel, ConfigDict, StrictStr + +from anonymizer.engine.constants import ( + COL_ATTEMPT_ID, + COL_PHASE7_CANDIDATE_BUNDLE, + COL_PHASE7_CANDIDATE_REQUEST, + COL_PHASE7_INVOCATION_ID, + COL_TARGET_WORK_ID, + COL_TASK_ID, + _jinja, +) +from anonymizer.engine.execution.accounting_evidence import ( + _AttemptId, + _Dispatch, + _InvocationId, + _RowToken, +) +from anonymizer.engine.execution.accounting_plan import _ScopeTaskSubject, _TaskKey +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.execution.phase7_admission import ( + _is_admitted_scope_manifest, + _ReplacementSlotId, + _ScopeManifest, +) +from anonymizer.engine.execution.phase7_contract import ( + _is_admitted_phase7_contract, + _Phase7StableSubstituteContract, +) +from anonymizer.engine.execution.phase7_planner_ledger import ( + _PlannerLedger, + _PlannerSnapshot, + _PlannerState, + _Reservation, +) +from anonymizer.engine.execution.phase7_validation import ( + _CandidateAssignment, + _index_scope_sources, + _ScopeSourceIndex, + _validate_scope_bundle, + _ValidatedBundle, +) +from anonymizer.engine.ndd.adapter import ( + RECORD_ID_COLUMN, + FailedRecord, + NddAdapter, + WorkflowRunResult, + _FailedRowEvidence, +) +from anonymizer.engine.ndd.model_loader import resolve_model_alias + +T = TypeVar("T", bound=BaseModel) + + +class _PrivatePhase7NddValue: + __slots__ = () + + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Phase 7 NDD values are not serializable") + + +class _Phase7NddStatus(str, Enum): + NO_WORK = "no_work" + PENDING = "pending" + CANDIDATE = "candidate" + TASK_FAILED = "task_failed" + INVOCATION_INCONSISTENT = "invocation_inconsistent" + ABORTED = "aborted" + POISONED = "poisoned" + + +class _Phase7NddReason(str, Enum): + BACKEND_FAILED = "backend_failed" + EVIDENCE_UNATTRIBUTABLE = "evidence_unattributable" + LIMIT_EXCEEDED = "limit_exceeded" + CONTRACT_INVALID = "contract_invalid" + PHASE6_HANDOFF_MISMATCH = "phase6_handoff_mismatch" + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7NddResult(_PrivatePhase7NddValue): + status: _Phase7NddStatus + assignments: tuple[_CandidateAssignment, ...] = () + reason: _Phase7NddReason | None = None + # This is opaque evidence from an authority independent of the candidate + # backend. Runtime verifies both the receipt and its dispatch binding. + trusted_stop_receipt: object | None = None + + +class _WireAssignment(BaseModel): + model_config = ConfigDict(extra="forbid") + + slot_token: StrictStr + value: StrictStr + + +class _WireBundle(BaseModel): + model_config = ConfigDict(extra="forbid") + + assignments: list[_WireAssignment] + + +@dataclass(frozen=True, slots=True, repr=False) +class _SlotBinding(_PrivatePhase7NddValue): + token: str + slot_id: _ReplacementSlotId + + +@dataclass(frozen=True, slots=True, repr=False) +class _CandidateWorkframe(_PrivatePhase7NddValue): + dataframe: pd.DataFrame + dispatch: _Dispatch + task_token: str + slots: tuple[_SlotBinding, ...] + + +def _default_identity() -> str: + return secrets.token_hex(16) + + +class _Phase7NddBackend(_PrivatePhase7NddValue): + """Propose and structurally reconcile one complete scope bundle.""" + + __slots__ = ( + "_adapter", + "_invocation", + "_identity_factory", + "_planner", + "_barrier_callback", + "_crash_after_dispatch_callback", + ) + + def __init__( + self, + adapter: NddAdapter, + invocation: _CompiledInvocation, + *, + identity_factory: Callable[[], str] = _default_identity, + barrier: Callable[[str], None] | None = None, + crash_after_dispatch: Callable[[], None] | None = None, + ) -> None: + self._adapter = adapter + self._invocation = invocation + self._identity_factory = identity_factory + # Kept private to this invocation; it is intentionally unrelated to + # Phase 4 accounting and is never accepted from a caller. + self._planner = _PlannerLedger[_Phase7NddResult]() + self._barrier_callback = barrier + self._crash_after_dispatch_callback = crash_after_dispatch + + def close(self) -> None: + self._planner.close() + + def discard_values(self) -> None: + """Release accepted private bundle references after verified cleanup.""" + self._planner.discard_values() + + def cleanup_attestation(self, cleanup_identity: object) -> object: + """Return verifiable, content-free closure evidence for the runtime owner.""" + # The planner can only discard after close. Read its sealed state + # after retirement; do not fabricate a zero-reference assertion. + try: + self._planner.discard_values() + except RuntimeError: + return None + observation = self._planner.cleanup_observation() + if observation is None: + return None + active_reservations, provisional_references, values_observable = observation + from anonymizer.engine.execution.phase7_runtime import _Phase7CleanupAttestation + + return _Phase7CleanupAttestation( + "phase7-cleanup-attestation/v1", + active_reservations == 0 and provisional_references == 0 and not values_observable, + active_reservations, + 0, # Candidate workframes are stack-local and never retained. + True, + provisional_references, + values_observable, + cleanup_identity, + ) + + def cancel_scope(self, manifest: object, *, trusted_stop: bool = False) -> _Phase7NddResult | None: + """Private, bounded cancellation hook used by lifecycle owners/tests.""" + if not isinstance(manifest, _ScopeManifest) or not _is_admitted_scope_manifest(manifest): + return _inconsistent(_Phase7NddReason.CONTRACT_INVALID) + try: + snapshot = self._planner.cancel(manifest.id, trusted_stop=trusted_stop) + except RuntimeError: + return _poisoned() + return None if snapshot is None else _snapshot_result(snapshot) + + def propose_scope( + self, + manifest: object, + handoffs: object, + contract: object, + dispatch: object, + ) -> _Phase7NddResult: + prepared = _prepare_scope(manifest, handoffs, contract) + if isinstance(prepared, _Phase7NddResult): + return prepared + admitted_manifest, admitted_contract, sources = prepared + if not admitted_manifest.slots: + if dispatch is not None: + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + return _Phase7NddResult(_Phase7NddStatus.NO_WORK) + if not _valid_dispatch(dispatch): + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + if not isinstance(self._invocation, _CompiledInvocation): + return _inconsistent(_Phase7NddReason.CONTRACT_INVALID) + ledger = self._planner + try: + reservation, replay = ledger.reserve(admitted_manifest.id, _dispatch_evidence(dispatch)) + except RuntimeError: + return _poisoned() + if replay is not None: + return _snapshot_result(replay) + if reservation is None: + return _Phase7NddResult(_Phase7NddStatus.PENDING) + self._barrier("reserve") + replayed = self._after_barrier(ledger, admitted_manifest.id, reservation, dispatch) + if replayed is not None: + return replayed + workframe = _lower_candidate_workframe( + admitted_manifest, + sources, + admitted_contract, + dispatch, + identity_factory=self._identity_factory, + ) + if workframe is None: + return self._terminal( + ledger, admitted_manifest.id, reservation, _inconsistent(_Phase7NddReason.LIMIT_EXCEEDED) + ) + if not ledger.mark_dispatched(admitted_manifest.id, reservation): + return self._current(ledger, admitted_manifest.id, dispatch) + workflow = self._run_candidate_workflow(workframe) + # This barrier deliberately follows the adapter call. A reentrant + # cancellation here is post-dispatch and the received result below is + # therefore late evidence which must not be published. + self._barrier("dispatch") + replayed = self._after_barrier(ledger, admitted_manifest.id, reservation, dispatch) + if replayed is not None: + return replayed + self._barrier("receipt") + replayed = self._after_barrier(ledger, admitted_manifest.id, reservation, dispatch) + if replayed is not None: + return replayed + crash = self._crash_after_dispatch_callback + if crash is not None: + try: + crash() + except Exception: + return self._terminal(ledger, admitted_manifest.id, reservation, _poisoned()) + if isinstance(workflow, _Phase7NddResult): + self._barrier("validation") + replayed = self._after_barrier(ledger, admitted_manifest.id, reservation, dispatch) + if replayed is not None: + return replayed + self._barrier("transformation") + replayed = self._after_barrier(ledger, admitted_manifest.id, reservation, dispatch) + if replayed is not None: + return replayed + self._barrier("reconciliation") + replayed = self._after_barrier(ledger, admitted_manifest.id, reservation, dispatch) + if replayed is not None: + return replayed + return self._terminal(ledger, admitted_manifest.id, reservation, workflow) + result = _reconcile_candidate_result(workframe, workflow) + self._barrier("validation") + replayed = self._after_barrier(ledger, admitted_manifest.id, reservation, dispatch) + if replayed is not None: + return replayed + if result.status is _Phase7NddStatus.CANDIDATE: + validated = _validate_scope_bundle( + admitted_manifest, + handoffs, + result.assignments, + admitted_contract, + ) + if not isinstance(validated, _ValidatedBundle): + result = _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + self._barrier("transformation") + replayed = self._after_barrier(ledger, admitted_manifest.id, reservation, dispatch) + if replayed is not None: + return replayed + self._barrier("reconciliation") + replayed = self._after_barrier(ledger, admitted_manifest.id, reservation, dispatch) + if replayed is not None: + return replayed + self._barrier("publication") + replayed = self._after_barrier(ledger, admitted_manifest.id, reservation, dispatch) + if replayed is not None: + return replayed + return self._terminal(ledger, admitted_manifest.id, reservation, result) + + def _barrier(self, stage: str) -> None: + callback = self._barrier_callback + if callback is not None: + callback(stage) + + def _after_barrier( + self, + ledger: _PlannerLedger[_Phase7NddResult], + scope: object, + reservation: _Reservation, + dispatch: _Dispatch, + ) -> _Phase7NddResult | None: + try: + if ledger.owns(scope, reservation): + return None + except RuntimeError: + return _poisoned() + return self._current(ledger, scope, dispatch) + + def _current( + self, ledger: _PlannerLedger[_Phase7NddResult], scope: object, dispatch: _Dispatch | None = None + ) -> _Phase7NddResult: + try: + snapshot = ledger.current(scope) + except RuntimeError: + return _poisoned() + return _Phase7NddResult(_Phase7NddStatus.PENDING) if snapshot is None else _snapshot_result(snapshot) + + def _terminal( + self, + ledger: _PlannerLedger[_Phase7NddResult], + scope: object, + reservation: _Reservation, + result: _Phase7NddResult, + ) -> _Phase7NddResult: + state = ( + _PlannerState.PLANNED + if result.status is _Phase7NddStatus.CANDIDATE + else _PlannerState.POISONED + if result.status in {_Phase7NddStatus.INVOCATION_INCONSISTENT, _Phase7NddStatus.POISONED} + else _PlannerState.ABORTED + ) + if not ledger.terminal(scope, reservation, _PlannerSnapshot(state, result)): + return self._current(ledger, scope) + self._barrier("release") + released = self._current(ledger, scope) + self._barrier("cleanup") + cleaned = self._current(ledger, scope) + return cleaned if cleaned != released else released + + def _run_candidate_workflow( + self, + workframe: _CandidateWorkframe, + ) -> WorkflowRunResult | _Phase7NddResult: + try: + with self._adapter.private_execution(): + return self._adapter.run_workflow( + workframe.dataframe, + model_configs=list(self._invocation.model_configs), + columns=[_candidate_column(self._invocation)], + workflow_name="phase7-candidate-planning", + ) + except Exception: + # After dispatch, an exception leaves no trusted terminal run + # evidence. It is transport/process loss, never attributable + # FailedRecord task evidence. + return _poisoned() + + +def _prepare_scope( + manifest: object, + handoffs: object, + contract: object, +) -> tuple[_ScopeManifest, _Phase7StableSubstituteContract, _ScopeSourceIndex] | _Phase7NddResult: + if not isinstance(contract, _Phase7StableSubstituteContract) or not _is_admitted_phase7_contract(contract): + return _inconsistent(_Phase7NddReason.CONTRACT_INVALID) + if not isinstance(manifest, _ScopeManifest) or not _is_admitted_scope_manifest(manifest): + return _inconsistent(_Phase7NddReason.CONTRACT_INVALID) + if not isinstance(handoffs, tuple): + return _inconsistent(_Phase7NddReason.PHASE6_HANDOFF_MISMATCH) + sources = _index_scope_sources(manifest, handoffs) + if sources is None: + return _inconsistent(_Phase7NddReason.PHASE6_HANDOFF_MISMATCH) + return manifest, contract, sources + + +def _candidate_column(invocation: _CompiledInvocation) -> LLMStructuredColumnConfig: + return LLMStructuredColumnConfig( + name=COL_PHASE7_CANDIDATE_BUNDLE, + prompt="""Propose exactly one complete candidate assignment for this private scope. +Request: """ + + _jinja(COL_PHASE7_CANDIDATE_REQUEST) + + """ +Return every opaque slot_token exactly once. Values are proposals and will be validated separately.""", + model_alias=resolve_model_alias( + "replacement_generator", + invocation.selected_models.replace, + ), + output_format=_WireBundle, + ) + + +def _valid_dispatch(value: object) -> TypeGuard[_Dispatch]: + return ( + isinstance(value, _Dispatch) + and isinstance(value.invocation_id, _InvocationId) + and type(value.invocation_id.value) is str + and bool(value.invocation_id.value) + and isinstance(value.task, _TaskKey) + and isinstance(value.task.subject, _ScopeTaskSubject) + and isinstance(value.attempt_id, _AttemptId) + and type(value.attempt_id.value) is str + and bool(value.attempt_id.value) + and isinstance(value.row_token, _RowToken) + and type(value.row_token.value) is str + and bool(value.row_token.value) + ) + + +def _dispatch_evidence(dispatch: _Dispatch) -> tuple[str, _TaskKey, str, str]: + """Return the exact compiler-issued correlation evidence for one attempt.""" + return ( + dispatch.invocation_id.value, + dispatch.task, + dispatch.attempt_id.value, + dispatch.row_token.value, + ) + + +def _lower_candidate_workframe( + manifest: _ScopeManifest, + sources: _ScopeSourceIndex, + contract: _Phase7StableSubstituteContract, + dispatch: _Dispatch, + *, + identity_factory: Callable[[], str], +) -> _CandidateWorkframe | None: + try: + used = { + dispatch.invocation_id.value, + dispatch.attempt_id.value, + dispatch.row_token.value, + } + task_token = _claim_identity(identity_factory(), used) + bindings = tuple(_SlotBinding(_claim_identity(identity_factory(), used), slot.id) for slot in manifest.slots) + request = _candidate_request(manifest, sources, bindings) + encoded_request = json.dumps(request, ensure_ascii=False, separators=(",", ":")) + dataframe = _candidate_dataframe( + dispatch, + task_token, + encoded_request, + max_bytes=dict(contract.byte_limits)["max_workframe_bytes_per_scope"], + ) + if dataframe is None: + return None + except (KeyError, StopIteration, TypeError, UnicodeEncodeError, ValueError): + return None + return _CandidateWorkframe(dataframe, dispatch, task_token, bindings) + + +def _candidate_request( + manifest: _ScopeManifest, + sources: _ScopeSourceIndex, + bindings: tuple[_SlotBinding, ...], +) -> dict[str, object]: + wire_token = {binding.slot_id: binding.token for binding in bindings} + mention_by_id = dict(sources.mentions) + return { + "schema_version": "phase7-workframe/v1", + "slots": [ + { + "slot_token": binding.token, + "role": slot.role, + "format": slot.format, + "mask": slot.mask, + "source_values": [mention_by_id[mention_id].source_slice for mention_id in slot.mention_ids], + } + for slot, binding in zip(manifest.slots, bindings, strict=True) + ], + "required_distinct_pairs": [ + {"left": wire_token[pair.left], "right": wire_token[pair.right]} for pair in manifest.required_pairs + ], + "relations": [ + { + "version": relation.version, + "upstream": [wire_token[token] for token in relation.upstream], + "downstream": wire_token[relation.downstream], + } + for relation in manifest.relations + ], + } + + +def _candidate_dataframe( + dispatch: _Dispatch, + task_token: str, + request: str, + *, + max_bytes: int, +) -> pd.DataFrame | None: + row = { + COL_TARGET_WORK_ID: dispatch.row_token.value, + COL_PHASE7_INVOCATION_ID: dispatch.invocation_id.value, + COL_TASK_ID: task_token, + COL_ATTEMPT_ID: dispatch.attempt_id.value, + COL_PHASE7_CANDIDATE_REQUEST: request, + RECORD_ID_COLUMN: dispatch.row_token.value, + } + encoded = json.dumps(row, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + if len(encoded.encode("utf-8")) > max_bytes: + return None + return pd.DataFrame( + [row], + columns=pd.Index( + [ + COL_TARGET_WORK_ID, + COL_PHASE7_INVOCATION_ID, + COL_TASK_ID, + COL_ATTEMPT_ID, + COL_PHASE7_CANDIDATE_REQUEST, + RECORD_ID_COLUMN, + ] + ), + ) + + +def _claim_identity(value: object, used: set[str]) -> str: + if type(value) is not str or not value or value in used: + raise TypeError + used.add(value) + return value + + +def _reconcile_candidate_result( + workframe: _CandidateWorkframe, + result: object, +) -> _Phase7NddResult: + if not isinstance(result, WorkflowRunResult): + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + if not isinstance(result.failed_records, list) or not isinstance(result.failed_row_evidence, tuple): + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + if result.failed_records: + return _reconcile_failed_result(workframe, result) + if result.failed_row_evidence: + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + dataframe = result.dataframe + if not isinstance(dataframe, pd.DataFrame) or len(dataframe) != 1: + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + expected_columns = { + COL_TARGET_WORK_ID: workframe.dispatch.row_token.value, + COL_PHASE7_INVOCATION_ID: workframe.dispatch.invocation_id.value, + COL_TASK_ID: workframe.task_token, + COL_ATTEMPT_ID: workframe.dispatch.attempt_id.value, + } + if not _has_exact_correlations(dataframe, expected_columns): + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + if COL_PHASE7_CANDIDATE_BUNDLE not in dataframe.columns: + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + bundle = _coerce_model(dataframe.iloc[0][COL_PHASE7_CANDIDATE_BUNDLE], _WireBundle) + if bundle is None: + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + binding_by_token = {binding.token: binding.slot_id for binding in workframe.slots} + observed = tuple(item.slot_token for item in bundle.assignments) + if len(set(observed)) != len(observed) or set(observed) != set(binding_by_token): + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + value_by_token = {item.slot_token: item.value for item in bundle.assignments} + assignments = tuple( + _CandidateAssignment(binding.slot_id, value_by_token[binding.token]) for binding in workframe.slots + ) + return _Phase7NddResult(_Phase7NddStatus.CANDIDATE, assignments) + + +def _has_exact_correlations(dataframe: pd.DataFrame, expected: dict[str, str]) -> bool: + for column, expected_value in expected.items(): + if dataframe.columns.tolist().count(column) != 1: + return False + observed = dataframe.iloc[0][column] + if type(observed) is not str or observed != expected_value: + return False + return True + + +def _reconcile_failed_result( + workframe: _CandidateWorkframe, + result: WorkflowRunResult, +) -> _Phase7NddResult: + dataframe = result.dataframe + if not isinstance(dataframe, pd.DataFrame) or not dataframe.empty: + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + if len(result.failed_records) != 1 or len(result.failed_row_evidence) != 1: + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + evidence = result.failed_row_evidence[0] + record = result.failed_records[0] + if ( + not isinstance(evidence, _FailedRowEvidence) + or not isinstance(record, FailedRecord) + or type(evidence.row_token) is not str + or evidence.row_token != workframe.dispatch.row_token.value + or evidence.record is not record + ): + return _inconsistent(_Phase7NddReason.EVIDENCE_UNATTRIBUTABLE) + return _Phase7NddResult( + _Phase7NddStatus.TASK_FAILED, + reason=_Phase7NddReason.BACKEND_FAILED, + ) + + +def _coerce_model(raw: object, model: type[T]) -> T | None: + try: + if isinstance(raw, model): + return raw + if isinstance(raw, str): + raw = json.loads(raw) + return model.model_validate(raw) + except Exception as error: + del error + return None + + +def _inconsistent(reason: _Phase7NddReason) -> _Phase7NddResult: + return _Phase7NddResult(_Phase7NddStatus.INVOCATION_INCONSISTENT, reason=reason) + + +def _poisoned() -> _Phase7NddResult: + return _Phase7NddResult(_Phase7NddStatus.POISONED) + + +def _snapshot_result(snapshot: _PlannerSnapshot[_Phase7NddResult]) -> _Phase7NddResult: + if snapshot.value is not None: + return snapshot.value + if snapshot.state is _PlannerState.PENDING: + return _Phase7NddResult(_Phase7NddStatus.PENDING) + if snapshot.state is _PlannerState.ABORTED: + return _Phase7NddResult(_Phase7NddStatus.ABORTED) + return _poisoned() diff --git a/src/anonymizer/engine/execution/phase7_owner_contract_corpus.json b/src/anonymizer/engine/execution/phase7_owner_contract_corpus.json new file mode 100644 index 00000000..d69223b5 --- /dev/null +++ b/src/anonymizer/engine/execution/phase7_owner_contract_corpus.json @@ -0,0 +1,35 @@ +{ + "cases": [ + {"expected": "planned_empty", "id": "valid_empty_scope_zero_dispatch"}, + {"expected": "planned", "id": "valid_single_given_name"}, + {"expected": "planned", "id": "valid_given_family_email_relation"}, + {"expected": "planned", "id": "valid_phone_source_mask"}, + {"expected": "contract_invalid", "id": "unknown_contract_version"}, + {"expected": "digest_mismatch", "id": "contract_digest_mismatch"}, + {"expected": "detector_universe_incomplete", "id": "missing_detector_disposition"}, + {"expected": "unsupported_role", "id": "unknown_role"}, + {"expected": "unsupported_constraint", "id": "unknown_relation"}, + {"expected": "unsupported_mask", "id": "unknown_mask"}, + {"expected": "unsupported_label", "id": "unsupported_detector_label"}, + {"expected": "selector_missing", "id": "selector_resolves_zero_slots"}, + {"expected": "selector_ambiguous", "id": "selector_resolves_multiple_slots"}, + {"expected": "cross_scope_relation", "id": "relation_crosses_scopes"}, + {"expected": "relation_role_mismatch", "id": "email_relation_wrong_roles"}, + {"expected": "canonical_collision", "id": "distinct_slots_same_canonical_value"}, + {"expected": "candidate_matches_original", "id": "candidate_matches_own_original"}, + {"expected": "candidate_matches_original", "id": "candidate_matches_other_slot_original"}, + {"expected": "relation_failed", "id": "email_local_part_omits_name"}, + {"expected": "planned", "id": "count_limits_exact"}, + {"expected": "limit_exceeded", "id": "count_limits_one_over"}, + {"expected": "planned", "id": "byte_limits_exact"}, + {"expected": "limit_exceeded", "id": "byte_limits_one_over"}, + {"expected": "missing_capability", "id": "runtime_capability_missing"}, + {"expected": "failed", "id": "trusted_task_failure"}, + {"expected": "inconsistent_global_embargo", "id": "unattributable_failure"}, + {"expected": "release_eligible", "id": "cleanup_attestation_verified"}, + {"expected": "inconsistent_global_embargo", "id": "cleanup_attestation_missing"}, + {"expected": "inconsistent_global_embargo", "id": "cleanup_attestation_contradictory"}, + {"expected": "blocked_zero_effects", "id": "redact_policy_role_bearing_scope"} + ], + "version": "anonymizer-phase7-owner-contract-corpus/v1" +} diff --git a/src/anonymizer/engine/execution/phase7_planner_ledger.py b/src/anonymizer/engine/execution/phase7_planner_ledger.py new file mode 100644 index 00000000..dd169884 --- /dev/null +++ b/src/anonymizer/engine/execution/phase7_planner_ledger.py @@ -0,0 +1,180 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Invocation-private, one-shot ownership for Phase 7 scope planning. + +This deliberately is not an accounting ledger: Phase 4 remains the only +authority which releases work. The small state machine here only prevents a +planner invocation from producing more than one effect for an admitted scope. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from threading import RLock +from typing import Generic, TypeVar + +T = TypeVar("T") + + +class _PlannerState(str, Enum): + PENDING = "pending" + PLANNED = "planned" + ABORTED = "aborted" + POISONED = "poisoned" + + +@dataclass(frozen=True, slots=True, repr=False) +class _PlannerSnapshot(Generic[T]): + state: _PlannerState + value: T | None = None + trusted_stop: bool = False + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _Reservation: + """Identity-only capability; equality must never confer authority.""" + + +@dataclass(slots=True) +class _Entry(Generic[T]): + reservation: _Reservation | None = None + snapshot: _PlannerSnapshot[T] | None = None + evidence: object | None = None + dispatched: bool = False + + +class _PlannerLedger(Generic[T]): + """Closed, non-serializable per-backend ledger keyed by scope capability.""" + + __slots__ = ("__entries", "__closed", "__lock") + + def __init__(self) -> None: + self.__entries: dict[object, _Entry[T]] = {} + self.__closed = False + self.__lock = RLock() + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Phase 7 planner ledger is not serializable") + + def __copy__(self) -> _PlannerLedger[T]: + raise TypeError("private Phase 7 planner ledger cannot escape") + + __deepcopy__ = __copy__ + + def reserve( + self, scope: object, evidence: object | None = None + ) -> tuple[_Reservation | None, _PlannerSnapshot[T] | None]: + with self.__lock: + entry = self.__entries.get(scope) + if entry is None: + self.__active() + entry = _Entry() + self.__entries[scope] = entry + if entry.snapshot is not None: + if entry.evidence != evidence: + # Do not disclose a previously accepted private result to + # stale or foreign terminal evidence. + return None, _PlannerSnapshot(_PlannerState.POISONED) + return None, entry.snapshot + self.__active() + if entry.reservation is not None: + if entry.evidence != evidence: + # Conflicting evidence while a candidate is still + # unaccepted is ambiguous and closes this scope. + entry.snapshot = _PlannerSnapshot(_PlannerState.POISONED) + entry.reservation = None + return None, entry.snapshot + return None, _PlannerSnapshot(_PlannerState.PENDING) + reservation = _Reservation() + entry.reservation = reservation + entry.evidence = evidence + return reservation, None + + def current(self, scope: object) -> _PlannerSnapshot[T] | None: + with self.__lock: + entry = self.__entries.get(scope) + return None if entry is None else entry.snapshot + + def owns(self, scope: object, reservation: object) -> bool: + with self.__lock: + self.__active() + entry = self.__entries.get(scope) + return entry is not None and entry.reservation is reservation and entry.snapshot is None + + def mark_dispatched(self, scope: object, reservation: object) -> bool: + with self.__lock: + if self.__closed: + return False + entry = self.__entries.get(scope) + if entry is None or entry.reservation is not reservation or entry.snapshot is not None: + return False + entry.dispatched = True + return True + + def terminal(self, scope: object, reservation: object, snapshot: _PlannerSnapshot[T]) -> bool: + with self.__lock: + if self.__closed: + return False + entry = self.__entries.get(scope) + if entry is None or entry.reservation is not reservation or entry.snapshot is not None: + return False + entry.snapshot = snapshot + entry.reservation = None + return True + + def cancel(self, scope: object, *, trusted_stop: bool, value: T | None = None) -> _PlannerSnapshot[T] | None: + with self.__lock: + self.__active() + entry = self.__entries.get(scope) + if entry is None: + return None + if entry.snapshot is not None: + return entry.snapshot + if entry.reservation is None: + return None + state = _PlannerState.ABORTED if not entry.dispatched or trusted_stop else _PlannerState.POISONED + entry.snapshot = _PlannerSnapshot(state, value, entry.dispatched and trusted_stop) + entry.reservation = None + return entry.snapshot + + def close(self) -> None: + with self.__lock: + if self.__closed: + return + for entry in self.__entries.values(): + if entry.snapshot is None: + entry.snapshot = _PlannerSnapshot(_PlannerState.POISONED) + entry.reservation = None + self.__closed = True + + def discard_values(self) -> None: + """Drop every private bundle after lifecycle cleanup has been verified.""" + with self.__lock: + if not self.__closed: + raise RuntimeError("private Phase 7 planner ledger is still active") + for entry in self.__entries.values(): + if entry.snapshot is not None: + entry.snapshot = _PlannerSnapshot(entry.snapshot.state) + entry.evidence = None + + def cleanup_observation(self) -> tuple[int, int, bool] | None: + """Return content-free, post-retirement facts from this sealed ledger. + + ``None`` means that closure/retirement was not completed, so callers + must embargo rather than manufacture a zero-reference attestation. + """ + with self.__lock: + if not self.__closed: + return None + active = sum(entry.reservation is not None for entry in self.__entries.values()) + provisional = sum( + entry.snapshot is not None and entry.snapshot.value is not None for entry in self.__entries.values() + ) + observable = any(entry.evidence is not None for entry in self.__entries.values()) + return active, provisional, observable + + def __active(self) -> None: + if self.__closed: + raise RuntimeError("private Phase 7 planner ledger is closed") diff --git a/src/anonymizer/engine/execution/phase7_runtime.py b/src/anonymizer/engine/execution/phase7_runtime.py new file mode 100644 index 00000000..2fc3fcd9 --- /dev/null +++ b/src/anonymizer/engine/execution/phase7_runtime.py @@ -0,0 +1,454 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Private Phase 7 lifecycle coordinator and release-qualified projection.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from typing import Callable, Protocol, cast + +from anonymizer.engine.execution.accounting_ledger import _AccountingLedger, _EvidenceAcceptance +from anonymizer.engine.execution.accounting_outcomes import ( + _AccountingResult, + _CauseCode, + _GroupReleased, + _TaskBlocked, + _TaskCancelled, + _TaskFailed, + _TaskInconsistent, + _TaskLost, + _TaskSucceeded, +) +from anonymizer.engine.execution.accounting_plan import _DatumTaskSubject, _TaskKey +from anonymizer.engine.execution.phase6_plan import _is_admitted_phase6_plan, _Phase6Plan +from anonymizer.engine.execution.phase6_runtime import ( + _is_admitted_phase6_execution, + _is_admitted_substitute_handoff, + _Phase6Execution, +) +from anonymizer.engine.execution.phase7_admission import _is_admitted_phase7_plan, _Phase7Plan, _ScopeManifest +from anonymizer.engine.execution.phase7_application import ( + _AppliedDatum, + _apply_substitute_datum, + _materialize_substitute_patches, +) +from anonymizer.engine.execution.phase7_contract import _is_admitted_phase7_contract, _Phase7StableSubstituteContract +from anonymizer.engine.execution.phase7_ndd_backend import _Phase7NddResult, _Phase7NddStatus +from anonymizer.engine.execution.phase7_validation import _validate_scope_bundle, _ValidatedBundle + + +class _PrivatePhase7RuntimeValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Phase 7 runtime values are not serializable") + + +class _ScopePlanState(str, Enum): + PLANNED = "planned" + BLOCKED = "blocked" + FAILED = "failed" + CANCELLED = "cancelled" + LOST = "lost" + INCONSISTENT = "inconsistent" + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7PlanReceipt(_PrivatePhase7RuntimeValue): + """Content-free proof that an immutable bundle passed private validation.""" + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7ScopeOutcome(_PrivatePhase7RuntimeValue): + """Content-free terminal reduction for exactly one declared scope.""" + + state: _ScopePlanState + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7CleanupAttestation(_PrivatePhase7RuntimeValue): + """The frozen pre-release cleanup evidence required by the P7 contract.""" + + version: str + verified: bool + active_reservation_count: int + backend_workframe_reference_count: int + ledger_mutation_closed: bool + provisional_bundle_reference_count: int + provisional_values_observable: bool + # Issued by this runtime immediately before finalization. It is an + # invocation-private capability, not a printable/backend-derived ID. + cleanup_identity: object | None = None + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7Phase4Evidence(_PrivatePhase7RuntimeValue): + """Private evidence handoff; Phase 4 remains the release authority.""" + + scopes: tuple[_Phase7ScopeOutcome, ...] + accounting: _AccountingResult[object] + cleanup: _Phase7CleanupAttestation + global_embargo: bool + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7Execution(_PrivatePhase7RuntimeValue): + scopes: tuple[_Phase7ScopeOutcome, ...] + cleanup: _Phase7CleanupAttestation + phase4: _Phase7Phase4Evidence + released: tuple[_AppliedDatum, ...] = () + + +class _Phase7EffectBackend(Protocol): + def propose_scope( + self, manifest: object, handoffs: object, contract: object, dispatch: object + ) -> _Phase7NddResult: ... + + def close(self) -> None: ... + + def discard_values(self) -> None: ... + + def cleanup_attestation(self, cleanup_identity: object) -> object: ... + + +class _Phase7RuntimeAdmissionError(TypeError): + def __init__(self) -> None: + super().__init__("admitted private Phase 6 and Phase 7 plans and compatible backend required") + + def __repr__(self) -> str: + return "" + + +class _Phase7Runtime: + """Plan one immutable bundle per ready scope and attest cleanup before handoff.""" + + def __init__( + self, + backend: _Phase7EffectBackend, + *, + cancellation_requested: Callable[[], bool] | None = None, + trusted_stop_receipt_verified: Callable[[object, object], bool] | None = None, + ) -> None: + self._backend = backend + self._cancellation_requested = cancellation_requested + # The execution host owns this verifier. It is deliberately separate + # from the backend that can only report an aborted candidate attempt. + self._trusted_stop_receipt_verified = trusted_stop_receipt_verified + + def run( + self, + phase6: _Phase6Plan, + phase6_execution: _Phase6Execution, + plan: _Phase7Plan, + contract: _Phase7StableSubstituteContract, + ) -> _Phase7Execution: + self._preflight(phase6, phase6_execution, plan, contract) + # The compiler, not this coordinator, owns the single Phase 4 task + # plan. In particular scope capabilities are never reconstructed + # from a manifest or declaration order at execution time. + task_by_scope = dict(zip((manifest.id for manifest in plan.manifests), plan.scope_tasks, strict=True)) + if not _has_exact_phase6_prefix(phase6, phase6_execution, plan): + raise _Phase7RuntimeAdmissionError + ledger: _AccountingLedger[object] = _AccountingLedger(plan.accounting) + ledger.open() + # Phase 7 extends, rather than replaces, Phase 6's compiler-expanded + # Phase 4 plan. These exact terminal records are immutable input. + ledger.import_terminal_outcomes(phase6_execution.accounting.tasks) + planned_bundles: dict[object, _ValidatedBundle] = {} + try: + for manifest in plan.manifests: + task = task_by_scope[manifest.id] + # Cancellation is an invocation event, not backend status. + # Observe it before a scope becomes dispatchable so it cannot + # manufacture an attempt; observe it again after a synchronous + # backend return before accepting a candidate. + if self._cancelled(): + ledger.request_cancellation() + break + if not _scope_has_terminal_phase6_evidence(manifest, phase6, phase6_execution): + ledger.mark_task_blocked(task) + continue + bundle = self._plan_scope(ledger, task, manifest, phase6_execution.handoffs, contract) + if bundle is not None: + planned_bundles[manifest.id] = bundle + _apply_planned_bundles(ledger, plan, planned_bundles) + finally: + planned_bundles.clear() + ledger.seal_mutation() + cleanup = self._cleanup() + if not cleanup.verified: + ledger.record_cleanup_unconfirmed_after_seal() + accounting = ledger.finish() + frozen = tuple(_scope_outcome(accounting, task_by_scope[manifest.id]) for manifest in plan.manifests) + released_values = tuple( + (datum_id, candidate) + for group in accounting.groups + if isinstance(group, _GroupReleased) + for datum_id, candidate in group.outputs + ) + release_is_valid = len({datum_id for datum_id, _candidate in released_values}) == len(released_values) and all( + isinstance(candidate, _AppliedDatum) and candidate.datum_id == datum_id + for datum_id, candidate in released_values + ) + embargo = ( + not cleanup.verified + or any( + outcome.state in {_ScopePlanState.INCONSISTENT, _ScopePlanState.LOST, _ScopePlanState.CANCELLED} + for outcome in frozen + ) + or not release_is_valid + ) + phase4 = _Phase7Phase4Evidence(frozen, accounting, cleanup, embargo) + released_by_datum = { + datum_id: candidate + for datum_id, candidate in released_values + if isinstance(candidate, _AppliedDatum) and candidate.datum_id == datum_id + } + released = tuple( + released_by_datum[datum.id] for datum in plan.accounting.datums if datum.id in released_by_datum + ) + # Cleanup retired every provisional bundle before this result crosses + # the owner boundary. Only release-qualified protected text remains. + return _Phase7Execution(frozen, cleanup, phase4, released) + + def _preflight( + self, + phase6: object, + phase6_execution: object, + plan: object, + contract: object, + ) -> None: + if ( + not isinstance(phase6, _Phase6Plan) + or not _is_admitted_phase6_plan(phase6) + or not _is_admitted_phase6_execution(phase6_execution, phase6) + or not isinstance(plan, _Phase7Plan) + or not _is_admitted_phase7_plan(plan) + or not isinstance(contract, _Phase7StableSubstituteContract) + or not _is_admitted_phase7_contract(contract) + or not callable(getattr(self._backend, "propose_scope", None)) + or not callable(getattr(self._backend, "close", None)) + or not callable(getattr(self._backend, "discard_values", None)) + or not callable(getattr(self._backend, "cleanup_attestation", None)) + ): + raise _Phase7RuntimeAdmissionError + + def _plan_scope( + self, + ledger: _AccountingLedger[object], + task: _TaskKey, + manifest: _ScopeManifest, + handoffs: tuple[object, ...], + contract: _Phase7StableSubstituteContract, + ) -> _ValidatedBundle | None: + if not manifest.slots: + bundle = _validate_scope_bundle(manifest, handoffs, (), contract) + if isinstance(bundle, _ValidatedBundle): + ledger.mark_task_succeeded(task, _Phase7PlanReceipt()) + return bundle + ledger.mark_task_failed(task) + return None + dispatch = ledger.dispatch(task) + try: + result = self._backend.propose_scope(manifest, handoffs, contract, dispatch) + except Exception: + ledger.mark_transport_lost(dispatch) + return None + if self._cancelled(): + # Cancellation is only a request. Once dispatch occurred, a + # returned candidate does not independently prove that execution + # stopped, so it is stale and the attempt must remain lost rather + # than fabricating a trusted stop acknowledgement. + ledger.request_cancellation() + ledger.mark_transport_lost(dispatch) + return None + if not isinstance(result, _Phase7NddResult): + ledger.mark_inconsistent(_CauseCode.CONTRADICTORY) + return None + if result.status is _Phase7NddStatus.CANDIDATE: + bundle = _validate_scope_bundle(manifest, handoffs, result.assignments, contract) + if not isinstance(bundle, _ValidatedBundle): + ledger.accept_failure(dispatch) + return None + if ledger.accept_success(dispatch, _Phase7PlanReceipt()) is _EvidenceAcceptance.ACCEPTED: + return bundle + ledger.mark_inconsistent(_CauseCode.CONTRADICTORY) + return None + if result.status is _Phase7NddStatus.TASK_FAILED: + ledger.accept_failure(dispatch) + elif result.status is _Phase7NddStatus.ABORTED: + # An abort report and a dispatch echo are both backend assertions, + # not proof that work stopped. Only an independently verified + # receipt bound to this exact dispatch can acknowledge stop. + ledger.request_cancellation() + if self._trusted_stop_verified(result.trusted_stop_receipt, dispatch): + ledger.acknowledge_stop(dispatch) + else: + ledger.mark_transport_lost(dispatch) + elif result.status is _Phase7NddStatus.POISONED: + ledger.mark_transport_lost(dispatch) + else: + ledger.mark_inconsistent(_CauseCode.CONTRADICTORY) + return None + + def _cleanup(self) -> _Phase7CleanupAttestation: + """Perform and attest the lifecycle actions this owner actually observed. + + The backend supplies closure evidence only after the coordinator has + observed both mandatory retirement operations complete. + """ + try: + self._backend.close() + self._backend.discard_values() + except Exception: + return _Phase7CleanupAttestation("phase7-cleanup-attestation/v1", False, 0, 0, False, 0, False) + cleanup_identity = object() + try: + evidence = self._backend.cleanup_attestation(cleanup_identity) + except Exception: + evidence = None + if not _is_verified_cleanup_attestation(evidence, cleanup_identity): + return _Phase7CleanupAttestation("phase7-cleanup-attestation/v1", False, 0, 0, False, 0, False) + return cast(_Phase7CleanupAttestation, evidence) + + def _cancelled(self) -> bool: + callback = self._cancellation_requested + if callback is None: + return False + try: + return callback() is True + except Exception: + # An unreadable cancellation source is ambiguous lifecycle + # evidence, so fail closed through the same cancellation embargo. + return True + + def _trusted_stop_verified(self, receipt: object | None, dispatch: object) -> bool: + verifier = self._trusted_stop_receipt_verified + if receipt is None or verifier is None: + return False + try: + return verifier(receipt, dispatch) is True + except Exception: + return False + + +def _is_verified_cleanup_attestation(value: object, cleanup_identity: object) -> bool: + """Reject missing or contradictory backend closure evidence before release.""" + return ( + isinstance(value, _Phase7CleanupAttestation) + and value.version == "phase7-cleanup-attestation/v1" + and value.verified + and value.active_reservation_count == 0 + and value.backend_workframe_reference_count == 0 + and value.ledger_mutation_closed + and value.provisional_bundle_reference_count == 0 + and not value.provisional_values_observable + and value.cleanup_identity is cleanup_identity + ) + + +def _apply_planned_bundles( + ledger: _AccountingLedger[object], + plan: _Phase7Plan, + planned_bundles: dict[object, _ValidatedBundle], +) -> None: + """Apply each planned scope once through its compiler-owned datum tasks.""" + application_by_datum = { + task.subject.datum_id: task for task in plan.application_tasks if isinstance(task.subject, _DatumTaskSubject) + } + candidates: dict[object, _AppliedDatum] = {} + for manifest in plan.manifests: + bundle = planned_bundles.get(manifest.id) + if bundle is None: + continue + try: + patches = _materialize_substitute_patches(bundle) + except Exception as cause: + del cause + continue + if not isinstance(patches, tuple): + continue + for datum_id in manifest.members: + try: + applied = _apply_substitute_datum(bundle, patches, datum_id) + except Exception as cause: + del cause + continue + if isinstance(applied, _AppliedDatum) and applied.datum_id == datum_id: + candidates[datum_id] = applied + + for datum_id in plan.accounting.topological_datums: + task = application_by_datum[datum_id] + if task not in ledger.ready_tasks(): + continue + candidate = candidates.get(datum_id) + if candidate is None: + ledger.mark_task_failed(task) + else: + ledger.mark_task_succeeded(task, candidate) + + +def _scope_has_terminal_phase6_evidence( + manifest: _ScopeManifest, + phase6: _Phase6Plan, + execution: _Phase6Execution, +) -> bool: + handoffs = execution.handoffs + if not isinstance(handoffs, tuple) or not all(_is_admitted_substitute_handoff(item, phase6) for item in handoffs): + return False + task_by_key = {outcome.task: outcome for outcome in execution.accounting.tasks} + expected = tuple( + task for task in phase6.accounting.tasks if getattr(task.subject, "datum_id", None) in manifest.members + ) + # A non-success Phase 6 terminal record is a local prerequisite failure; + # it never becomes a new planner attempt. Missing records are equally + # non-admissible and remain withheld by the Phase 4 reduction. + if len(task_by_key) != len(execution.accounting.tasks) or any( + not isinstance(task_by_key.get(task), _TaskSucceeded) for task in expected + ): + return False + covered = {datum_id for handoff in handoffs for datum_id in handoff.terminal_evidence.datum_ids} + return set(manifest.members) <= covered + + +def _has_exact_phase6_prefix( + phase6: _Phase6Plan, + execution: _Phase6Execution, + plan: _Phase7Plan, +) -> bool: + """Bind imported terminals to the compiler-expanded Phase 4 prefix. + + Task-key membership is insufficient: a forged/reordered expansion could + otherwise import a plausible subset of Phase 6 while retaining Phase 7 + scope capabilities. The admitted execution is already sealed to + ``phase6``; this check binds that exact terminal sequence to the prefix of + the later compiler-issued plan before the ledger is opened. + """ + outcomes = execution.accounting.tasks + prefix = plan.accounting.tasks[: len(outcomes)] + return ( + len(outcomes) == len(phase6.accounting.tasks) + and tuple(outcome.task for outcome in outcomes) == phase6.accounting.tasks + and prefix == phase6.accounting.tasks + and plan.accounting.tasks[len(outcomes) :] == (*plan.scope_tasks, *plan.application_tasks) + ) + + +def _scope_outcome(accounting: _AccountingResult[object], task: _TaskKey) -> _Phase7ScopeOutcome: + outcome = next(item for item in accounting.tasks if item.task == task) + if isinstance(outcome, _TaskSucceeded): + return _Phase7ScopeOutcome(_ScopePlanState.PLANNED) + if isinstance(outcome, _TaskBlocked): + return _Phase7ScopeOutcome(_ScopePlanState.BLOCKED) + if isinstance(outcome, _TaskFailed): + return _Phase7ScopeOutcome(_ScopePlanState.FAILED) + if isinstance(outcome, _TaskCancelled): + return _Phase7ScopeOutcome(_ScopePlanState.CANCELLED) + if isinstance(outcome, _TaskLost): + return _Phase7ScopeOutcome(_ScopePlanState.LOST) + if isinstance(outcome, _TaskInconsistent): + return _Phase7ScopeOutcome(_ScopePlanState.INCONSISTENT) + return _Phase7ScopeOutcome(_ScopePlanState.INCONSISTENT) diff --git a/src/anonymizer/engine/execution/phase7_stable_substitute_contract.json b/src/anonymizer/engine/execution/phase7_stable_substitute_contract.json new file mode 100644 index 00000000..c060bbf4 --- /dev/null +++ b/src/anonymizer/engine/execution/phase7_stable_substitute_contract.json @@ -0,0 +1,319 @@ +{ + "contract": { + "canonicalization": { + "algorithm": "unicode_alphanumeric_skeleton/v1", + "steps": [ + "normalize_NFKC", + "strip_leading_and_trailing_unicode_whitespace", + "casefold", + "retain_only_codepoints_whose_unicode_general_category_starts_with_L_or_N" + ], + "empty_result": "reject", + "use": [ + "candidate_vs_every_protected_original_in_scope", + "all_pairs_slot_distinctness", + "same_slot_original_distinctness", + "relation_predicates" + ] + }, + "cleanup": { + "attestation": { + "artifact_class": "phase7_planner_cleanup", + "cardinality": "exactly_one_per_invocation_before_phase4_release", + "required_fields": { + "active_reservation_count": 0, + "backend_workframe_reference_count": 0, + "ledger_mutation_closed": true, + "provisional_bundle_reference_count": 0, + "provisional_values_observable": false, + "status": "verified" + }, + "trusted_identity": [ + "invocation_private_invocation_token", + "invocation_private_ledger_token" + ], + "version": "phase7-cleanup-attestation/v1" + }, + "covered_terminal_paths": [ + "planned", + "blocked", + "failed_validation", + "failed_backend", + "cancelled", + "lost", + "inconsistent", + "transformation_failed", + "publication_failed" + ], + "failure_effect": "invocation_inconsistent_and_global_release_embargo", + "host_teardown_after_accepted_result": "reported_by_host_and_cannot_rewrite_accepted_result", + "physical_zeroization_claim": false + }, + "diagnostics": { + "allowed_count_dimensions": [ + "scopes", + "clusters", + "slots", + "mentions", + "relations", + "attempts", + "failed_records" + ], + "allowed_reason_codes": [ + "backend_failed", + "candidate_matches_original", + "canonical_collision", + "cleanup_contradictory", + "cleanup_unconfirmed", + "contract_invalid", + "cross_scope_relation", + "detector_universe_incomplete", + "digest_mismatch", + "evidence_unattributable", + "invalid_application", + "limit_exceeded", + "missing_capability", + "phase6_handoff_mismatch", + "prerequisite_blocked", + "relation_failed", + "relation_role_mismatch", + "selector_ambiguous", + "selector_missing", + "unsupported_constraint", + "unsupported_label", + "unsupported_mask", + "unsupported_role" + ], + "forbidden": [ + "original_values", + "synthetic_values", + "prompts", + "candidate_bundles", + "content_derived_hashes", + "source_ids", + "public_graph_ids", + "backend_record_ids" + ], + "opaque_test_identity_allowed": true + }, + "distinctness": { + "comparison": "canonicalization.algorithm", + "cross_scope_guarantee": "none", + "pair_compilation": "all_unordered_pairs_of_distinct_slots_in_each_scope", + "rule": "all_pairs_exact", + "shared_literal_assignment": "must_be_one_shared_slot_not_two_equal_slots" + }, + "execution": { + "backend": { + "adapter_boundary": "NddAdapter.run_workflow", + "allowed_physical_rows_per_scope": 1, + "calls_for_empty_scope": 0, + "calls_per_nonempty_scope": 1, + "candidate_authority": "proposal_only", + "direct_DataDesigner_create_or_preview": "forbidden" + }, + "capability": { + "context_retention": "retention_disabled", + "profile": "phase7-stable-substitute/v1", + "required_artifacts": [ + "phase7_candidate_request", + "phase7_planner_cleanup" + ], + "schema_version": "phase7-workframe/v1" + }, + "failure_attribution": { + "ambiguous_or_unattributable": "invocation_inconsistent_and_global_release_embargo", + "backend_record_id_authoritative": false, + "content_authoritative": false, + "single_task_failure_without_slot_attribution": "task_failed_and_complete_scope_invalid", + "trusted_exactly_one_of": [ + "current_invocation_private_task_token", + "current_invocation_private_attempt_token", + "trusted_batch_or_call_token_mapping_to_exactly_one_task" + ] + }, + "lifecycle": { + "automatic_regeneration": false, + "automatic_retry": false, + "durable_state": false, + "ledger_created": "after_pure_compilation", + "ledger_lifetime": "one_invocation", + "ledger_serializable": false, + "max_active_planners_per_scope": 1, + "max_attempts_per_scope": 1, + "max_concurrent_planners_per_invocation": 2, + "max_invocation_planner_wall_seconds": 240, + "max_scope_planner_wall_seconds": 120, + "planned_bundle_mutability": "immutable", + "publication_authority": "phase4_only" + } + }, + "formats": { + "email_addr_spec_ascii/v1": { + "domain": "one_or_more_dot_separated_ASCII_labels_of_1_to_63_alnum_or_hyphen_codepoints_with_no_leading_or_trailing_hyphen_and_a_final_2_to_63_ASCII_letter_label", + "local": "1_to_64_ASCII_codepoints_from_ALNUM_or_.!#$%&'*+/=?^_`{|}~-_with_no_leading_trailing_or_consecutive_dot", + "total_utf8_bytes": [3, 254] + }, + "telephone_ascii/v1": { + "allowed": "ASCII_digits_space_parentheses_plus_hyphen_dot", + "digit_count": [7, 15], + "plus": "zero_or_one_and_only_at_codepoint_zero" + }, + "unicode_person_name/v1": { + "allowed": "unicode_general_categories_L_M_Zs_and_literals_apostrophe_hyphen_period", + "must_contain": "at_least_one_unicode_L_codepoint", + "normalized_form": "NFKC", + "utf8_bytes": [1, 128] + }, + "username_ascii/v1": { + "allowed": "ASCII_alphanumeric_period_underscore_hyphen", + "boundary": "first_and_last_codepoint_must_be_ASCII_alphanumeric", + "utf8_bytes": [1, 64] + } + }, + "limits": { + "bytes": { + "max_all_original_value_bytes": 1536, + "max_candidate_bundle_bytes_per_scope": 2048, + "max_candidate_value_bytes": 256, + "max_context_bytes_per_scope": 8192, + "max_context_fragment_bytes": 4096, + "max_detector_label_bytes": 64, + "max_ledger_bytes_per_invocation": 32768, + "max_original_value_bytes": 256, + "max_workframe_bytes_per_scope": 16384 + }, + "counts": { + "max_clusters_per_invocation": 3, + "max_clusters_per_scope": 3, + "max_context_fragments_per_scope": 4, + "max_datums_per_invocation": 4, + "max_distinct_pairs_per_scope": 6, + "max_mentions_per_invocation": 6, + "max_mentions_per_scope": 6, + "max_relations_per_scope": 4, + "max_scope_members": 4, + "max_scopes_per_invocation": 2, + "max_slots_per_invocation": 4, + "max_slots_per_scope": 4 + }, + "enforcement": "reject_exact_plus_one_before_ledger_workframe_telemetry_adapter_transform_or_publication_effect" + }, + "masks": { + "digit_literal/v1": "NFKC_source_and_candidate_must_have_equal_codepoint_length; each_ASCII_digit_source_position_requires_an_ASCII_digit_candidate; every_non_digit_codepoint_must_match_exactly", + "none/v1": "no_source_shape_predicate", + "unknown": "reject" + }, + "oracle_contract_corpus": { + "canonicalization": "UTF8_compact_sorted_key_JSON_of_the_complete_corpus_document_with_no_trailing_newline", + "case_count": 30, + "digest": "5ba1e69c7836a428b16e9d2ac7fc0cf3fbb445fd0ca72c1d0a8a049194115123", + "file": "anonymizer-phase-7-owner-contract-corpus-v1.json", + "version": "anonymizer-phase7-owner-contract-corpus/v1" + }, + "phase6_handoff": { + "current_redact_policy_digest": "e11a29db1af26c9572e1b4dec9e0a91e80966c6de1d813a378b64210a3bdfc40", + "current_redact_policy_version": "phase6-role-result/v1", + "required_result_version": "phase6-role-result/v1", + "substitute_policy": { + "canonicalization": "UTF8_compact_sorted_key_JSON_of_the_complete_policy_payload_with_no_trailing_newline", + "detector_label_count": 65, + "digest": "c27580bd2cc4051bdd11b63a91391f8995bdef1ed2052534623cdd3160318ef8", + "file": "anonymizer-phase6-substitute-role-policy-v1.json", + "supported_detector_labels": [ + "email", + "fax_number", + "first_name", + "last_name", + "phone_number", + "user_name" + ], + "unsupported_disposition": "explicit_null_for_every_other_built_in_detector_label_and_all_custom_labels", + "version": "phase6-substitute-role-policy/v1" + } + }, + "reference_model": { + "canonical_trace_max_exogenous_observations": 16, + "generated_manifest": "P4_must_separately_freeze_exact_graph_count_trace_count_and_digest_before_P4_promotion", + "independence": "must_not_import_or_accept_verdicts_from_production_phase7_pandas_DataDesigner_or_ledger_code", + "version": "phase7-reference-model/v1" + }, + "relations": { + "email_from_name/v1": { + "bundle_predicate": "the_canonicalized_candidate_value_of_at_least_one_upstream_name_slot_is_a_nonempty_substring_of_the_canonicalized_email_local_part", + "classification": [ + "ValueInput", + "bundle_predicate" + ], + "downstream_role": "email_address", + "scope": "all_selectors_must_resolve_in_one_scope", + "task_or_release_edge": false, + "upstream_cardinality": [1, 2], + "upstream_roles": [ + "person_family_name", + "person_given_name" + ] + }, + "unknown": "reject", + "wildcard_constraints": "none_supported" + }, + "roles": { + "email_address": {"format": "email_addr_spec_ascii/v1", "mask": "none/v1"}, + "fax_number": {"format": "telephone_ascii/v1", "mask": "digit_literal/v1"}, + "person_family_name": {"format": "unicode_person_name/v1", "mask": "none/v1"}, + "person_given_name": {"format": "unicode_person_name/v1", "mask": "none/v1"}, + "user_name": {"format": "username_ascii/v1", "mask": "none/v1"}, + "voice_phone_number": {"format": "telephone_ascii/v1", "mask": "digit_literal/v1"} + }, + "scope_and_slot_semantics": { + "cluster_cross_scope": "reject", + "coherence_scope": "flat_exact_partition_of_all_target_datums", + "cross_scope_assignment_sharing_or_uniqueness": "none", + "empty_scope_manifest": "planned_empty_bundle_verified_no_work_zero_dispatch", + "identity_sources_forbidden": [ + "text", + "detector_label_alone", + "row_order", + "dataframe_index", + "source_id", + "backend_record_id", + "content_hash" + ], + "slot_derivation": "exactly_one_compiler_issued_opaque_slot_per_scope_cluster_role_tuple_with_at_least_one_supported_mention", + "slot_identity": "compiler_issued_opaque_nonserializable_capability", + "unsupported_scope_shapes": [ + "coverage_gap", + "implicit_singleton_completion", + "nesting", + "partial_overlap", + "duplicate_member", + "duplicate_scope", + "empty_scope" + ] + }, + "selectors": { + "cluster_role/v1": { + "cluster_ref": "compiler_issued_typed_cluster_capability_from_the_exact_phase6_handoff", + "resolution": "must_resolve_exactly_one_pre_slot_scope_cluster_role_tuple_before_slot_identity_is_issued", + "role": "one_closed_supported_role", + "serialized_or_authored_opaque_ids": "forbidden" + }, + "unknown": "reject" + }, + "status": "frozen_owner_contract", + "task_ownership": { + "scope_plan_stage": "exactly_one_scope_owned_task_per_declared_scope_including_an_empty_manifest_scope", + "scope_subject": "compiler_issued_opaque_scope_capability", + "scope_tasks_in_datum_dependency_or_release_reduction": false, + "subject_sum": [ + "datum_owned", + "scope_owned" + ], + "representative_nullable_sentinel_or_encoded_datum": "forbidden" + }, + "version": "anonymizer-phase7-stable-substitute/v1" + }, + "digest": "3755832ecc64fe6e9dbeccc136c40020e5158446e5c74d6dca2392efdbb006bb", + "digest_algorithm": "sha256_of_UTF8_compact_sorted_key_JSON_of_contract_member_with_no_trailing_newline", + "schema_version": "anonymizer-phase7-owner-contract-envelope/v1" +} diff --git a/src/anonymizer/engine/execution/phase7_validation.py b/src/anonymizer/engine/execution/phase7_validation.py new file mode 100644 index 00000000..2e89930a --- /dev/null +++ b/src/anonymizer/engine/execution/phase7_validation.py @@ -0,0 +1,446 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Deterministic candidate validation for private Phase 7 Substitute scopes.""" + +from __future__ import annotations + +import re +import unicodedata +from dataclasses import dataclass, field +from enum import Enum + +from anonymizer.engine.execution.mention_admission import _AnchoredMention, _MentionId, _MentionTarget +from anonymizer.engine.execution.phase6_runtime import ( + _PHASE6_HANDOFF_SEAL, + _handoff_snapshot, + _Phase6SubstituteHandoff, +) +from anonymizer.engine.execution.phase7_admission import ( + _is_admitted_scope_manifest, + _ReplacementSlot, + _ReplacementSlotId, + _scope_manifest_snapshot, + _ScopeManifest, +) +from anonymizer.engine.execution.phase7_contract import ( + _is_admitted_phase7_contract, + _Phase7Role, + _Phase7StableSubstituteContract, +) +from anonymizer.engine.execution.role_policy import ( + _RESOLVED_GRAPH_SEAL, + _resolved_graph_snapshot, + _ResolvedGraph, +) + + +class _PrivatePhase7ValidationValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Phase 7 validation values are not serializable") + + +class _BundleRejectionCode(str, Enum): + INVALID_INPUT = "invalid_input" + DUPLICATE_SLOT = "duplicate_slot" + FOREIGN_SLOT = "foreign_slot" + PARTIAL_BUNDLE = "partial_bundle" + CANDIDATE_MATCHES_ORIGINAL = "candidate_matches_original" + LIMIT_EXCEEDED = "limit_exceeded" + UNSUPPORTED_ROLE = "unsupported_role" + CANONICAL_COLLISION = "canonical_collision" + UNSUPPORTED_CONSTRAINT = "unsupported_constraint" + RELATION_FAILED = "relation_failed" + + +@dataclass(frozen=True, slots=True, repr=False) +class _CandidateAssignment(_PrivatePhase7ValidationValue): + token: _ReplacementSlotId + value: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _ValidatedAssignment(_PrivatePhase7ValidationValue): + token: _ReplacementSlotId + value: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _ValidatedBundleProof(_PrivatePhase7ValidationValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _ValidatedBundle(_PrivatePhase7ValidationValue): + manifest: _ScopeManifest + handoffs: tuple[_Phase6SubstituteHandoff, ...] + assignments: tuple[_ValidatedAssignment, ...] + _proof: _ValidatedBundleProof | None = field(default=None, compare=False) + + +@dataclass(frozen=True, slots=True, repr=False) +class _BundleRejected(_PrivatePhase7ValidationValue): + code: _BundleRejectionCode + + +@dataclass(frozen=True, slots=True, repr=False) +class _ScopeSourceIndex(_PrivatePhase7ValidationValue): + mentions: tuple[tuple[_MentionId, _AnchoredMention], ...] + targets: tuple[_MentionTarget, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _PreparedBundle(_PrivatePhase7ValidationValue): + manifest: _ScopeManifest + handoffs: tuple[_Phase6SubstituteHandoff, ...] + assignments: tuple[_CandidateAssignment, ...] + sources: _ScopeSourceIndex + + +_VALIDATED_BUNDLE_SEAL = object() +_USERNAME_PATTERN = re.compile(r"[A-Za-z0-9](?:[A-Za-z0-9._-]{0,62}[A-Za-z0-9])?") +_TELEPHONE_PATTERN = re.compile(r"[0-9 ()+.-]+") +_EMAIL_LOCAL_PATTERN = re.compile(r"[A-Za-z0-9.!#$%&'*+/=?^_`{|}~-]+") +_EMAIL_LABEL_PATTERN = re.compile(r"[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?") + + +def _validate_scope_bundle( + manifest: object, + handoffs: object, + assignments: object, + contract: object, +) -> _ValidatedBundle | _BundleRejected: + """Validate one complete scope atomically against frozen source evidence.""" + prepared = _prepare_bundle(manifest, handoffs, assignments, contract) + if isinstance(prepared, _BundleRejected): + return prepared + if not isinstance(contract, _Phase7StableSubstituteContract): + return _BundleRejected(_BundleRejectionCode.INVALID_INPUT) + rejected = _validate_bundle_values(prepared, contract) + if rejected is not None: + return rejected + + assignment_by_token = {item.token: item.value for item in prepared.assignments} + expected_tokens = tuple(slot.id for slot in prepared.manifest.slots) + validated_assignments = tuple(_ValidatedAssignment(token, assignment_by_token[token]) for token in expected_tokens) + values = (prepared.manifest, prepared.handoffs, validated_assignments) + candidate = _ValidatedBundle(*values) + snapshot = _validated_bundle_snapshot(candidate) + if snapshot is None: + return _BundleRejected(_BundleRejectionCode.INVALID_INPUT) + return _ValidatedBundle(*values, _ValidatedBundleProof(_VALIDATED_BUNDLE_SEAL, snapshot)) + + +def _prepare_bundle( + manifest: object, + handoffs: object, + assignments: object, + contract: object, +) -> _PreparedBundle | _BundleRejected: + if ( + not isinstance(manifest, _ScopeManifest) + or not _is_admitted_scope_manifest(manifest) + or not isinstance(handoffs, tuple) + or not isinstance(assignments, tuple) + or not isinstance(contract, _Phase7StableSubstituteContract) + or not _is_admitted_phase7_contract(contract) + ): + return _BundleRejected(_BundleRejectionCode.INVALID_INPUT) + source_index = _index_scope_sources(manifest, handoffs) + if source_index is None: + return _BundleRejected(_BundleRejectionCode.INVALID_INPUT) + validated_assignments = _validate_assignment_set(manifest, assignments) + if isinstance(validated_assignments, _BundleRejected): + return validated_assignments + typed_handoffs = tuple(item for item in handoffs if isinstance(item, _Phase6SubstituteHandoff)) + return _PreparedBundle(manifest, typed_handoffs, validated_assignments, source_index) + + +def _validate_assignment_set( + manifest: _ScopeManifest, + assignments: tuple[object, ...], +) -> tuple[_CandidateAssignment, ...] | _BundleRejected: + if not all(isinstance(item, _CandidateAssignment) for item in assignments): + return _BundleRejected(_BundleRejectionCode.INVALID_INPUT) + typed_assignments = tuple(item for item in assignments if isinstance(item, _CandidateAssignment)) + if any(not isinstance(item.token, _ReplacementSlotId) for item in typed_assignments): + return _BundleRejected(_BundleRejectionCode.INVALID_INPUT) + observed_tokens = tuple(item.token for item in typed_assignments) + if len(set(observed_tokens)) != len(observed_tokens): + return _BundleRejected(_BundleRejectionCode.DUPLICATE_SLOT) + expected_tokens = tuple(slot.id for slot in manifest.slots) + expected_set = set(expected_tokens) + observed_set = set(observed_tokens) + if observed_set - expected_set: + return _BundleRejected(_BundleRejectionCode.FOREIGN_SLOT) + if observed_set != expected_set: + return _BundleRejected(_BundleRejectionCode.PARTIAL_BUNDLE) + if any(type(item.value) is not str for item in typed_assignments): + return _BundleRejected(_BundleRejectionCode.INVALID_INPUT) + return typed_assignments + + +def _validate_bundle_values( + prepared: _PreparedBundle, + contract: _Phase7StableSubstituteContract, +) -> _BundleRejected | None: + assignment_by_token = {item.token: item.value for item in prepared.assignments} + mention_by_id = dict(prepared.sources.mentions) + originals = tuple(mention.source_slice for _mention_id, mention in prepared.sources.mentions) + original_skeletons = {_canonicalize_value(original) for original in originals} + canonical_by_token: dict[_ReplacementSlotId, str] = {} + byte_limits = dict(contract.byte_limits) + bundle_bytes = 0 + roles = {role.name: role for role in contract.roles} + for slot in prepared.manifest.slots: + value = assignment_by_token[slot.id] + validated = _validate_slot_value(slot, value, mention_by_id, original_skeletons, roles, byte_limits) + if isinstance(validated, _BundleRejected): + return validated + skeleton, value_bytes = validated + bundle_bytes += value_bytes + if bundle_bytes > byte_limits["max_candidate_bundle_bytes_per_scope"]: + return _BundleRejected(_BundleRejectionCode.LIMIT_EXCEEDED) + canonical_by_token[slot.id] = skeleton + + return _validate_bundle_constraints(prepared.manifest, assignment_by_token, canonical_by_token) + + +def _validate_bundle_constraints( + manifest: _ScopeManifest, + assignment_by_token: dict[_ReplacementSlotId, str], + canonical_by_token: dict[_ReplacementSlotId, str], +) -> _BundleRejected | None: + for pair in manifest.required_pairs: + if canonical_by_token[pair.left] == canonical_by_token[pair.right]: + return _BundleRejected(_BundleRejectionCode.CANONICAL_COLLISION) + for relation in manifest.relations: + if relation.version != "email_from_name/v1": + return _BundleRejected(_BundleRejectionCode.UNSUPPORTED_CONSTRAINT) + if not _matches_email_from_name_relation( + relation.upstream, relation.downstream, assignment_by_token, canonical_by_token + ): + return _BundleRejected(_BundleRejectionCode.RELATION_FAILED) + return None + + +def _validate_slot_value( + slot: _ReplacementSlot, + value: str, + mention_by_id: dict[_MentionId, _AnchoredMention], + original_skeletons: set[str | None], + roles: dict[str, _Phase7Role], + byte_limits: dict[str, int], +) -> tuple[str, int] | _BundleRejected: + skeleton = _canonicalize_value(value) + if skeleton is None or skeleton in original_skeletons: + return _BundleRejected(_BundleRejectionCode.CANDIDATE_MATCHES_ORIGINAL) + try: + value_bytes = len(value.encode("utf-8")) + except UnicodeEncodeError: + return _BundleRejected(_BundleRejectionCode.UNSUPPORTED_ROLE) + if value_bytes > byte_limits["max_candidate_value_bytes"]: + return _BundleRejected(_BundleRejectionCode.LIMIT_EXCEEDED) + role = roles.get(slot.role) + if role is None or role.format != slot.format or role.mask != slot.mask or not _matches_format(slot.format, value): + return _BundleRejected(_BundleRejectionCode.UNSUPPORTED_ROLE) + if any( + mention_id not in mention_by_id or not _matches_mask(slot.mask, mention_by_id[mention_id].source_slice, value) + for mention_id in slot.mention_ids + ): + return _BundleRejected(_BundleRejectionCode.RELATION_FAILED) + return skeleton, value_bytes + + +def _canonicalize_value(value: object) -> str | None: + if type(value) is not str: + return None + try: + normalized = unicodedata.normalize("NFKC", value).strip().casefold() + except (TypeError, ValueError): + return None + canonical = "".join(character for character in normalized if unicodedata.category(character)[:1] in {"L", "N"}) + return canonical or None + + +def _matches_format(format_name: object, value: object) -> bool: + if type(format_name) is not str or type(value) is not str: + return False + try: + if format_name == "unicode_person_name/v1": + normalized = unicodedata.normalize("NFKC", value) + return ( + 1 <= len(normalized.encode("utf-8")) <= 128 + and any(unicodedata.category(character).startswith("L") for character in normalized) + and all( + unicodedata.category(character)[:1] in {"L", "M"} + or unicodedata.category(character) == "Zs" + or character in "'.-" + for character in normalized + ) + ) + if format_name == "username_ascii/v1": + return value.isascii() and _USERNAME_PATTERN.fullmatch(value) is not None + if format_name == "telephone_ascii/v1": + return ( + _TELEPHONE_PATTERN.fullmatch(value) is not None + and 7 <= sum(character.isascii() and character.isdigit() for character in value) <= 15 + and value.count("+") <= 1 + and ("+" not in value or value.startswith("+")) + ) + if format_name == "email_addr_spec_ascii/v1": + return _matches_email_format(value) + except (TypeError, UnicodeEncodeError, ValueError): + return False + return False + + +def _matches_mask(mask_name: object, source: object, candidate: object) -> bool: + if type(mask_name) is not str or type(source) is not str or type(candidate) is not str: + return False + if mask_name == "none/v1": + return True + if mask_name != "digit_literal/v1": + return False + try: + normalized_source = unicodedata.normalize("NFKC", source) + normalized_candidate = unicodedata.normalize("NFKC", candidate) + except (TypeError, ValueError): + return False + return len(normalized_source) == len(normalized_candidate) and all( + (candidate_character.isascii() and candidate_character.isdigit()) + if source_character.isascii() and source_character.isdigit() + else source_character == candidate_character + for source_character, candidate_character in zip(normalized_source, normalized_candidate, strict=True) + ) + + +def _is_validated_bundle(value: object) -> bool: + return ( + isinstance(value, _ValidatedBundle) + and value._proof is not None + and value._proof.seal is _VALIDATED_BUNDLE_SEAL + and value._proof.snapshot == _validated_bundle_snapshot(value) + and _is_admitted_scope_manifest(value.manifest) + and all(_is_sealed_handoff(handoff) for handoff in value.handoffs) + and _index_scope_sources(value.manifest, value.handoffs) is not None + and tuple(assignment.token for assignment in value.assignments) + == tuple(slot.id for slot in value.manifest.slots) + and all(type(assignment.value) is str for assignment in value.assignments) + ) + + +def _index_scope_sources( + manifest: _ScopeManifest, + handoffs: tuple[object, ...], +) -> _ScopeSourceIndex | None: + if not all(_is_sealed_handoff(handoff) for handoff in handoffs): + return None + members = set(manifest.members) + targets: list[_MentionTarget] = [] + mentions: list[_AnchoredMention] = [] + for handoff in handoffs: + if not isinstance(handoff, _Phase6SubstituteHandoff): + return None + targets.extend(target for target in handoff.resolved.clustered.detected.targets if target.datum_id in members) + mentions.extend(item.mention for item in handoff.resolved.mentions if item.mention.target_datum_id in members) + if ( + len({target.datum_id for target in targets}) != len(targets) + or set(target.datum_id for target in targets) != members + or len({mention.id for mention in mentions}) != len(mentions) + ): + return None + expected_mentions = tuple(mention_id for slot in manifest.slots for mention_id in slot.mention_ids) + if len(set(expected_mentions)) != len(expected_mentions) or set(expected_mentions) != { + mention.id for mention in mentions + }: + return None + position = {mention_id: index for index, mention_id in enumerate(expected_mentions)} + mentions.sort(key=lambda mention: position[mention.id]) + target_position = {member: index for index, member in enumerate(manifest.members)} + targets.sort(key=lambda target: target_position[target.datum_id]) + return _ScopeSourceIndex(tuple((mention.id, mention) for mention in mentions), tuple(targets)) + + +def _is_sealed_handoff(value: object) -> bool: + return ( + isinstance(value, _Phase6SubstituteHandoff) + and value._proof is not None + and value._proof.seal is _PHASE6_HANDOFF_SEAL + and value._proof.snapshot == _handoff_snapshot(value) + and _is_sealed_resolved_graph(value.resolved) + ) + + +def _is_sealed_resolved_graph(value: object) -> bool: + return ( + isinstance(value, _ResolvedGraph) + and value._proof is not None + and value._proof.seal is _RESOLVED_GRAPH_SEAL + and value._proof.snapshot + == _resolved_graph_snapshot( + value.clustered, + value.mentions, + value.policy_version, + value.policy_digest, + value.source_policy_version, + ) + ) + + +def _matches_email_format(value: str) -> bool: + if not value.isascii() or not 3 <= len(value.encode("utf-8")) <= 254 or value.count("@") != 1: + return False + local, domain = value.split("@") + labels = domain.split(".") + return ( + 1 <= len(local) <= 64 + and not local.startswith(".") + and not local.endswith(".") + and ".." not in local + and _EMAIL_LOCAL_PATTERN.fullmatch(local) is not None + and len(labels) >= 2 + and all(1 <= len(label) <= 63 and _EMAIL_LABEL_PATTERN.fullmatch(label) is not None for label in labels) + and 2 <= len(labels[-1]) <= 63 + and labels[-1].isalpha() + ) + + +def _matches_email_from_name_relation( + upstream: tuple[_ReplacementSlotId, ...], + downstream: _ReplacementSlotId, + assignment_by_token: dict[_ReplacementSlotId, str], + canonical_by_token: dict[_ReplacementSlotId, str], +) -> bool: + if downstream not in assignment_by_token or any(token not in canonical_by_token for token in upstream): + return False + local = assignment_by_token[downstream].split("@", maxsplit=1)[0] + local_skeleton = _canonicalize_value(local) + return local_skeleton is not None and any(canonical_by_token[token] in local_skeleton for token in upstream) + + +def _validated_bundle_snapshot(bundle: _ValidatedBundle) -> tuple[object, ...] | None: + try: + return ( + _scope_manifest_snapshot(bundle.manifest), + tuple( + ( + _handoff_snapshot(handoff), + _resolved_graph_snapshot( + handoff.resolved.clustered, + handoff.resolved.mentions, + handoff.resolved.policy_version, + handoff.resolved.policy_digest, + handoff.resolved.source_policy_version, + ), + ) + for handoff in bundle.handoffs + ), + tuple((assignment.token, assignment.value) for assignment in bundle.assignments), + ) + except (AttributeError, TypeError): + return None diff --git a/src/anonymizer/engine/execution/protection_service.py b/src/anonymizer/engine/execution/protection_service.py new file mode 100644 index 00000000..63481bf2 --- /dev/null +++ b/src/anonymizer/engine/execution/protection_service.py @@ -0,0 +1,505 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Redact verification and compatibility projection over terminal accounting.""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass +from typing import Protocol, assert_never + +import pandas as pd + +from anonymizer.engine.constants import COL_FINAL_ENTITIES, COL_REPLACED_TEXT, COL_REPLACEMENT_APPLICATION +from anonymizer.engine.execution.accounting_admission import ( + _AccountingAdmissionResult, + _compile_accounting_plan, +) +from anonymizer.engine.execution.accounting_outcomes import ( + _CauseCode, + _DatumBlocked, + _DatumCancelled, + _DatumFailed, + _DatumInconsistent, + _DatumLost, + _DatumOutcome, + _DatumQualified, + _GroupReleased, + _InvocationCompleted, +) +from anonymizer.engine.execution.accounting_plan import _AccountingLimits, _AccountingPlan +from anonymizer.engine.execution.context_admission import ( + _compile_context_plan, + _ContextAdmissionResult, + _ContextPlan, +) +from anonymizer.engine.execution.context_contract import _ContextExecutionContract, _snapshot_context_capability +from anonymizer.engine.execution.context_observations import _observe_context_boundary +from anonymizer.engine.execution.graph import _DatumId, _DatumPurpose, _TextDatum +from anonymizer.engine.execution.graph_runtime import _AccountingGraphExecution +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.execution.mention_admission import _MentionLimits +from anonymizer.engine.execution.phase6_plan import ( + _compile_phase6_plan, + _Phase6Plan, + _Phase6PlanRejectionCode, + _Phase6ProfileVersion, + _Phase6Rejected, +) +from anonymizer.engine.execution.phase6_runtime import ( + _Phase6Candidate, + _Phase6EffectBackend, + _Phase6Execution, + _Phase6Runtime, +) +from anonymizer.engine.execution.phase7_admission import ( + _compile_phase7_plan, + _Phase7Declarations, + _Phase7Plan, +) +from anonymizer.engine.execution.phase7_contract import _Phase7StableSubstituteContract +from anonymizer.engine.execution.phase7_runtime import ( + _Phase7EffectBackend, + _Phase7Execution, + _Phase7Runtime, +) +from anonymizer.engine.execution.redact_patches import _VerifiedDatum + + +class _PrivateProtectionValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private protection results are not serializable") + + +@dataclass(frozen=True, slots=True, repr=False) +class _RedactCandidate(_PrivateProtectionValue): + output: str + applied: bool + release_qualified: bool + + +@dataclass(frozen=True, slots=True, repr=False) +class _GraphProtectionSucceeded(_PrivateProtectionValue): + datum_id: _DatumId + output: str + applied: bool + + +@dataclass(frozen=True, slots=True, repr=False) +class _GraphProtectionFailed(_PrivateProtectionValue): + datum_id: _DatumId + stage: str + scope: str + + +_GraphProtectionOutcome = _GraphProtectionSucceeded | _GraphProtectionFailed + + +@dataclass(frozen=True, slots=True, repr=False) +class _GraphProtectionResult(_PrivateProtectionValue): + outcomes: tuple[_GraphProtectionOutcome, ...] + + +class _GraphRuntimeBackend(Protocol): + def run( + self, + plan: _AccountingPlan | _ContextPlan, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + hydrate: Callable[[_TextDatum, pd.Series], _RedactCandidate], + datum_release_predicate: Callable[[_DatumId, _RedactCandidate], bool], + ) -> _AccountingGraphExecution[_RedactCandidate]: ... + + +class _RedactProtectionService: + """Protect text datums and project accounting into the compatibility result.""" + + def __init__(self, runtime: _GraphRuntimeBackend) -> None: + self._runtime = runtime + + @staticmethod + def admit( + graph: object, + *, + limits: _AccountingLimits, + ) -> _AccountingAdmissionResult: + """Compile the complete graph before any execution context opens.""" + return _compile_accounting_plan(graph, limits=limits) + + def admit_context( + self, + graph: object, + *, + accounting_limits: _AccountingLimits, + contract: _ContextExecutionContract, + ) -> _ContextAdmissionResult: + """Compile context framing against the selected backend's preflight snapshot.""" + target_count, context_count = _preflight_observation_counts(graph) + with _observe_context_boundary( + "preflight", + target_count=target_count, + context_count=context_count, + ) as observation: + capability = _snapshot_context_capability(self._runtime) + result = _compile_context_plan( + graph, + accounting_limits=accounting_limits, + contract=contract, + capability=capability, + ) + if isinstance(result, _ContextPlan): + observation.outcome = "admitted" + else: + observation.outcome = "rejected" + observation.reason = result.code.value + return result + + def protect( + self, + plan: _AccountingPlan | _ContextPlan, + *, + invocation: _CompiledInvocation, + ) -> _GraphProtectionResult: + execution = self._runtime.run( + plan, + invocation=invocation, + data_summary=None, + preview_num_records=None, + hydrate=_hydrate_redact_candidate, + datum_release_predicate=lambda _datum_id, candidate: candidate.release_qualified, + ) + return _materialize(execution) + + +class _Phase6RedactProtectionService: + """Compile and execute the selected private Phase 6 Redact profile.""" + + def __init__(self, backend: _Phase6EffectBackend, *, mention_limits: _MentionLimits) -> None: + self._backend = backend + self._mention_limits = mention_limits + + def admit_context( + self, + graph: object, + *, + accounting_limits: _AccountingLimits, + contract: _ContextExecutionContract, + ) -> _Phase6Plan | _Phase6Rejected: + target_count, context_count = _preflight_observation_counts(graph) + with _observe_context_boundary( + "preflight", + target_count=target_count, + context_count=context_count, + ) as observation: + capability = _snapshot_context_capability(self._backend) + if capability is None: + result: _Phase6Plan | _Phase6Rejected = _Phase6Rejected(_Phase6PlanRejectionCode.INVALID_PROFILE) + else: + result = _compile_phase6_plan( + graph, + accounting_limits=accounting_limits, + context_contract=contract, + capability=capability, + mention_limits=self._mention_limits, + ) + if isinstance(result, _Phase6Plan): + observation.outcome = "admitted" + else: + observation.outcome = "rejected" + observation.reason = result.code.value + return result + + def protect( + self, + plan: _Phase6Plan, + *, + invocation: _CompiledInvocation, + ) -> _GraphProtectionResult: + del invocation + return _materialize_phase6(plan, _Phase6Runtime(self._backend).run(plan)) + + +class _Phase7SubstituteProtectionService: + """Compile, plan, apply, and qualify the private stable-Substitute profile.""" + + def __init__( + self, + phase6_backend: _Phase6EffectBackend, + phase7_backend_factory: Callable[[], _Phase7EffectBackend], + *, + mention_limits: _MentionLimits, + ) -> None: + self._phase6_backend = phase6_backend + self._phase7_backend_factory = phase7_backend_factory + self._mention_limits = mention_limits + + def admit_context( + self, + graph: object, + *, + accounting_limits: _AccountingLimits, + contract: _ContextExecutionContract, + ) -> _Phase6Plan | _Phase6Rejected: + target_count, context_count = _preflight_observation_counts(graph) + with _observe_context_boundary( + "preflight", + target_count=target_count, + context_count=context_count, + ) as observation: + capability = _snapshot_context_capability(self._phase6_backend) + if capability is None: + result: _Phase6Plan | _Phase6Rejected = _Phase6Rejected(_Phase6PlanRejectionCode.INVALID_PROFILE) + else: + result = _compile_phase6_plan( + graph, + accounting_limits=accounting_limits, + context_contract=contract, + capability=capability, + mention_limits=self._mention_limits, + profile_version=_Phase6ProfileVersion.SUBSTITUTE_V1, + ) + if isinstance(result, _Phase6Plan): + observation.outcome = "admitted" + else: + observation.outcome = "rejected" + observation.reason = result.code.value + return result + + def protect( + self, + plan: _Phase6Plan, + *, + contract: _Phase7StableSubstituteContract, + ) -> _GraphProtectionResult: + phase6 = _Phase6Runtime(self._phase6_backend).run(plan) + phase7 = _compile_phase7_plan( + plan, + phase6.handoffs, + _Phase7Declarations(plan.coherence_scopes), + contract, + ) + if not isinstance(phase7, _Phase7Plan): + return _fail_all(tuple(datum.id for datum in plan.accounting.datums)) + execution = _Phase7Runtime(self._phase7_backend_factory()).run(plan, phase6, phase7, contract) + return _materialize_phase7(plan, execution) + + +def _preflight_observation_counts(graph: object) -> tuple[int, int]: + """Best-effort total counts for telemetry, kept outside admission semantics.""" + try: + datums = getattr(graph, "datums", ()) + scopes = getattr(graph, "context_scopes", ()) + except BaseException: + return 0, 0 + target_count = 0 + context_count = 0 + if isinstance(datums, tuple): + for datum in datums: + try: + target_count += getattr(datum, "purpose", None) is _DatumPurpose.TARGET + except BaseException: + continue + if isinstance(scopes, tuple): + for scope in scopes: + try: + context = getattr(scope, "context", ()) + except BaseException: + continue + if isinstance(context, tuple): + context_count += len(context) + return target_count, context_count + + +def _hydrate_redact_candidate(datum: _TextDatum, row: pd.Series) -> _RedactCandidate: + output = row[COL_REPLACED_TEXT] + if not isinstance(output, str): + raise TypeError("private redact result is malformed") + valid_entities, has_detections = _accepted_detection_state(row[COL_FINAL_ENTITIES]) + release_qualified = ( + valid_entities + and _redact_release_passed( + row[COL_FINAL_ENTITIES], + row[COL_REPLACEMENT_APPLICATION], + datum.text, + output, + ) + and (has_detections or output == datum.text) + ) + return _RedactCandidate(output, has_detections, release_qualified) + + +def _materialize(execution: _AccountingGraphExecution[_RedactCandidate]) -> _GraphProtectionResult: + if not isinstance(execution.accounting.invocation, _InvocationCompleted): + return _fail_all(tuple(datum.id for datum in execution.plan.datums)) + released = { + datum_id: candidate + for group in execution.accounting.groups + if isinstance(group, _GroupReleased) + for datum_id, candidate in group.outputs + } + datum_outcomes = {outcome.datum_id: outcome for outcome in execution.accounting.datums} + return _GraphProtectionResult( + tuple( + _materialize_datum(datum.id, released.get(datum.id), datum_outcomes[datum.id]) + for datum in execution.plan.datums + ) + ) + + +def _materialize_phase6(plan: _Phase6Plan, execution: _Phase6Execution) -> _GraphProtectionResult: + datum_ids = tuple(datum.id for datum in plan.accounting.datums) + if not isinstance(execution.accounting.invocation, _InvocationCompleted): + return _fail_all(datum_ids) + released = {datum.datum_id: datum for datum in execution.released} + datum_outcomes = {outcome.datum_id: outcome for outcome in execution.accounting.datums} + return _GraphProtectionResult( + tuple( + _materialize_phase6_datum(datum_id, released.get(datum_id), datum_outcomes[datum_id]) + for datum_id in datum_ids + ) + ) + + +def _materialize_phase7(plan: _Phase6Plan, execution: _Phase7Execution) -> _GraphProtectionResult: + datum_ids = tuple(datum.id for datum in plan.accounting.datums) + if not isinstance(execution.phase4.accounting.invocation, _InvocationCompleted) or execution.phase4.global_embargo: + return _fail_all(datum_ids) + released = {datum.datum_id: datum for datum in execution.released} + if len(released) != len(execution.released): + return _fail_all(datum_ids) + return _GraphProtectionResult( + tuple( + _GraphProtectionSucceeded(datum_id, released[datum_id].output, released[datum_id].applied) + if datum_id in released + else _GraphProtectionFailed(datum_id, "planning", "datum") + for datum_id in datum_ids + ) + ) + + +def _materialize_phase6_datum( + datum_id: _DatumId, + candidate: _VerifiedDatum | None, + outcome: _DatumOutcome[_Phase6Candidate], +) -> _GraphProtectionOutcome: + if candidate is not None: + return _GraphProtectionSucceeded(datum_id, candidate.output, candidate.applied) + match outcome: + case _DatumFailed(causes=causes) if any(cause.code is _CauseCode.RELEASE_PREDICATE_FAILED for cause in causes): + return _GraphProtectionFailed(datum_id, "release", "datum") + case _DatumQualified(): + return _GraphProtectionFailed(datum_id, "release", "group") + case _DatumFailed() | _DatumBlocked() | _DatumCancelled() | _DatumLost() | _DatumInconsistent(): + return _GraphProtectionFailed(datum_id, "pipeline", "datum") + case unreachable: + assert_never(unreachable) + + +def _materialize_datum( + datum_id: _DatumId, + candidate: _RedactCandidate | None, + outcome: _DatumOutcome[_RedactCandidate], +) -> _GraphProtectionOutcome: + if candidate is not None: + return _GraphProtectionSucceeded(datum_id, candidate.output, candidate.applied) + match outcome: + case _DatumFailed(causes=causes) if any(cause.code is _CauseCode.RELEASE_PREDICATE_FAILED for cause in causes): + return _GraphProtectionFailed(datum_id, "release", "datum") + case _DatumQualified(): + return _GraphProtectionFailed(datum_id, "release", "group") + case _DatumFailed() | _DatumBlocked() | _DatumCancelled() | _DatumLost() | _DatumInconsistent(): + return _GraphProtectionFailed(datum_id, "pipeline", "datum") + case unreachable: + assert_never(unreachable) + + +def _fail_all(datum_ids: tuple[_DatumId, ...]) -> _GraphProtectionResult: + return _GraphProtectionResult( + tuple(_GraphProtectionFailed(datum_id, "pipeline", "invocation") for datum_id in datum_ids) + ) + + +def _accepted_detection_state(value: object) -> tuple[bool, bool]: + if isinstance(value, dict): + if "entities" not in value: + return False, False + entities = value["entities"] + else: + entities = getattr(value, "entities", None) + if not isinstance(entities, (list, tuple)): + return False, False + return True, bool(entities) + + +def _redact_release_passed(value: object, application: object, input_text: str, output: str) -> bool: + """Require complete Redact accounting and removal of authoritative source spans.""" + if isinstance(value, dict): + entities = value.get("entities", []) + else: + entities = getattr(value, "entities", []) + if not isinstance(entities, (list, tuple)) or not _replacement_application_passed(application, len(entities)): + return False + spans: list[tuple[int, int, str]] = [] + for entity in entities: + raw = entity.get("value") if isinstance(entity, dict) else getattr(entity, "value", None) + label = entity.get("label") if isinstance(entity, dict) else getattr(entity, "label", None) + start = entity.get("start_position") if isinstance(entity, dict) else getattr(entity, "start_position", None) + end = entity.get("end_position") if isinstance(entity, dict) else getattr(entity, "end_position", None) + if ( + not isinstance(raw, str) + or not isinstance(label, str) + or not label + or type(start) is not int + or type(end) is not int + or start < 0 + or end <= start + or end > len(input_text) + or input_text[start:end] != raw + ): + return False + spans.append((start, end, input_text[start:end])) + previous_end = 0 + for start, end, source_slice in sorted(spans): + if start < previous_end or source_slice in output: + return False + previous_end = end + return True + + +def _replacement_application_passed(value: object, entity_count: int) -> bool: + if not isinstance(value, dict): + return False + expected_keys = { + "targeted_span_count", + "applied_span_count", + "skipped_span_count", + "skipped_span_label_counts", + } + if set(value) != expected_keys: + return False + targeted = value["targeted_span_count"] + applied = value["applied_span_count"] + skipped = value["skipped_span_count"] + skipped_by_label = value["skipped_span_label_counts"] + if ( + type(targeted) is not int + or type(applied) is not int + or type(skipped) is not int + or targeted < 0 + or applied < 0 + or skipped < 0 + or not isinstance(skipped_by_label, dict) + ): + return False + if any( + not isinstance(label, str) or not label or type(count) is not int or count <= 0 + for label, count in skipped_by_label.items() + ): + return False + return targeted == entity_count and applied == targeted and skipped == 0 and not skipped_by_label diff --git a/src/anonymizer/engine/execution/redact_patches.py b/src/anonymizer/engine/execution/redact_patches.py new file mode 100644 index 00000000..c14cfe88 --- /dev/null +++ b/src/anonymizer/engine/execution/redact_patches.py @@ -0,0 +1,324 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Mention-keyed local Redact patch construction and exact verification.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum + +from anonymizer.engine.execution.graph import _DatumId +from anonymizer.engine.execution.mention_admission import _MentionId, _MentionTargetToken +from anonymizer.engine.execution.role_policy import _ResolvedGraph + + +class _PrivatePatchValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private Redact patch values are not serializable") + + +class _RedactProfileVersion(str, Enum): + V1 = "phase6-redact/v1" + + +@dataclass(frozen=True, slots=True, repr=False, eq=False) +class _PatchToken(_PrivatePatchValue): + """Invocation-private identity bound to one expected mention patch.""" + + +@dataclass(frozen=True, slots=True, repr=False) +class _PatchManifestEntry(_PrivatePatchValue): + mention_id: _MentionId + + +@dataclass(frozen=True, slots=True, repr=False) +class _PatchManifestProof(_PrivatePatchValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _PatchManifest(_PrivatePatchValue): + resolved: _ResolvedGraph + entries: tuple[_PatchManifestEntry, ...] + profile_version: _RedactProfileVersion + _proof: _PatchManifestProof | None = field(default=None, compare=False) + + +@dataclass(frozen=True, slots=True, repr=False) +class _BoundPatchEntry(_PrivatePatchValue): + token: _PatchToken + mention_id: _MentionId + + +@dataclass(frozen=True, slots=True, repr=False) +class _BoundPatchProof(_PrivatePatchValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _BoundPatchManifest(_PrivatePatchValue): + manifest: _PatchManifest + entries: tuple[_BoundPatchEntry, ...] + _proof: _BoundPatchProof | None = field(default=None, compare=False) + + +@dataclass(frozen=True, slots=True, repr=False) +class _RedactPatch(_PrivatePatchValue): + token: _PatchToken + target: _MentionTargetToken + start: int + end: int + replacement: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _ReturnedRedact(_PrivatePatchValue): + target: _MentionTargetToken + output: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _VerifiedDatum(_PrivatePatchValue): + datum_id: _DatumId + output: str + applied: bool + + +@dataclass(frozen=True, slots=True, repr=False) +class _VerifiedGraph(_PrivatePatchValue): + resolved: _ResolvedGraph + datums: tuple[_VerifiedDatum, ...] + patches: tuple[_RedactPatch, ...] + profile_version: _RedactProfileVersion + + +class _PatchRejectionCode(str, Enum): + FOREIGN_TOKEN = "foreign_token" + STALE_TOKEN = "stale_token" + INVALID_PATCH = "invalid_patch" + RELEASE_PREDICATE_FAILED = "release_predicate_failed" + + +@dataclass(frozen=True, slots=True, repr=False) +class _PatchRejected(_PrivatePatchValue): + code: _PatchRejectionCode + owner: _MentionTargetToken | None = None + + +_PATCH_MANIFEST_SEAL = object() +_BOUND_PATCH_SEAL = object() +_REDACT_REPLACEMENT = "[REDACTED]" + + +def _build_patch_manifest(resolved: _ResolvedGraph) -> _PatchManifest | _PatchRejected: + if not isinstance(resolved, _ResolvedGraph): + return _PatchRejected(_PatchRejectionCode.STALE_TOKEN) + detected_ids = tuple(mention.id for mention in resolved.clustered.detected.mentions) + resolved_ids = tuple(item.mention.id for item in resolved.mentions) + if len(set(detected_ids)) != len(detected_ids) or resolved_ids != detected_ids: + return _PatchRejected(_PatchRejectionCode.INVALID_PATCH) + entries = tuple(_PatchManifestEntry(mention_id) for mention_id in resolved_ids) + snapshot = (resolved, tuple(entry.mention_id for entry in entries), _RedactProfileVersion.V1) + return _PatchManifest( + resolved, + entries, + _RedactProfileVersion.V1, + _PatchManifestProof(_PATCH_MANIFEST_SEAL, snapshot), + ) + + +def _bind_patch_manifest(manifest: _PatchManifest) -> _BoundPatchManifest | _PatchRejected: + if not _is_manifest(manifest): + return _PatchRejected(_PatchRejectionCode.STALE_TOKEN) + entries = tuple(_BoundPatchEntry(_PatchToken(), entry.mention_id) for entry in manifest.entries) + snapshot = (manifest, tuple((entry.token, entry.mention_id) for entry in entries)) + return _BoundPatchManifest(manifest, entries, _BoundPatchProof(_BOUND_PATCH_SEAL, snapshot)) + + +def _materialize_redact_patches(bound: _BoundPatchManifest) -> tuple[_RedactPatch, ...] | _PatchRejected: + if not _is_bound(bound): + return _PatchRejected(_PatchRejectionCode.STALE_TOKEN) + resolved = bound.manifest.resolved + mention_by_id = {item.mention.id: item.mention for item in resolved.mentions} + target_by_datum = {target.datum_id: target.token for target in resolved.clustered.detected.targets} + patches: list[_RedactPatch] = [] + for entry in bound.entries: + mention = mention_by_id.get(entry.mention_id) + if mention is None or mention.target_datum_id not in target_by_datum: + return _PatchRejected(_PatchRejectionCode.STALE_TOKEN) + if mention.source_slice in _REDACT_REPLACEMENT: + return _PatchRejected(_PatchRejectionCode.INVALID_PATCH, target_by_datum[mention.target_datum_id]) + patches.append( + _RedactPatch( + entry.token, + target_by_datum[mention.target_datum_id], + mention.start, + mention.end, + _REDACT_REPLACEMENT, + ) + ) + return tuple(patches) + + +def _apply_redact_patches( + resolved: _ResolvedGraph, + patches: tuple[_RedactPatch, ...], +) -> tuple[_ReturnedRedact, ...] | _PatchRejected: + if not isinstance(resolved, _ResolvedGraph) or not isinstance(patches, tuple): + return _PatchRejected(_PatchRejectionCode.INVALID_PATCH) + by_target: dict[_MentionTargetToken, list[_RedactPatch]] = { + target.token: [] for target in resolved.clustered.detected.targets + } + for patch in patches: + if not isinstance(patch, _RedactPatch) or patch.target not in by_target: + return _PatchRejected(_PatchRejectionCode.INVALID_PATCH) + by_target[patch.target].append(patch) + returned: list[_ReturnedRedact] = [] + for target in resolved.clustered.detected.targets: + output = _apply_target_patches(target.text, by_target[target.token]) + if output is None: + return _PatchRejected(_PatchRejectionCode.INVALID_PATCH, target.token) + returned.append(_ReturnedRedact(target.token, output)) + return tuple(returned) + + +def _apply_target_patches(text: str, patches: list[_RedactPatch]) -> str | None: + cursor = 0 + parts: list[str] = [] + for patch in sorted(patches, key=lambda item: (item.start, item.end)): + if patch.start < cursor or patch.end > len(text): + return None + parts.extend((text[cursor : patch.start], patch.replacement)) + cursor = patch.end + parts.append(text[cursor:]) + return "".join(parts) + + +def _verify_redact_patches( + bound: _BoundPatchManifest, + patches: tuple[_RedactPatch, ...], + returned: tuple[_ReturnedRedact, ...], +) -> _VerifiedGraph | _PatchRejected: + if not _is_bound(bound): + return _PatchRejected(_PatchRejectionCode.STALE_TOKEN) + validated = _validate_patches(bound, patches) + if isinstance(validated, _PatchRejected): + return validated + expected = _reconstruct_outputs(bound, validated) + if isinstance(expected, _PatchRejected): + return expected + returned_by_target = _validate_returned(bound, returned) + if isinstance(returned_by_target, _PatchRejected): + return returned_by_target + targets = bound.manifest.resolved.clustered.detected.targets + for target in targets: + if returned_by_target[target.token] != expected[target.token]: + return _PatchRejected(_PatchRejectionCode.RELEASE_PREDICATE_FAILED, target.token) + mention_datums = {item.mention.target_datum_id for item in bound.manifest.resolved.mentions} + datums = tuple( + _VerifiedDatum(target.datum_id, expected[target.token], target.datum_id in mention_datums) for target in targets + ) + return _VerifiedGraph(bound.manifest.resolved, datums, validated, bound.manifest.profile_version) + + +def _is_manifest(manifest: object) -> bool: + if not isinstance(manifest, _PatchManifest) or manifest._proof is None: + return False + snapshot = ( + manifest.resolved, + tuple(entry.mention_id for entry in manifest.entries), + manifest.profile_version, + ) + return manifest._proof.seal is _PATCH_MANIFEST_SEAL and manifest._proof.snapshot == snapshot + + +def _is_bound(bound: object) -> bool: + if not isinstance(bound, _BoundPatchManifest) or bound._proof is None or not _is_manifest(bound.manifest): + return False + snapshot = (bound.manifest, tuple((entry.token, entry.mention_id) for entry in bound.entries)) + return ( + bound._proof.seal is _BOUND_PATCH_SEAL + and bound._proof.snapshot == snapshot + and len({entry.token for entry in bound.entries}) == len(bound.entries) + and tuple(entry.mention_id for entry in bound.entries) + == tuple(entry.mention_id for entry in bound.manifest.entries) + ) + + +def _validate_patches( + bound: _BoundPatchManifest, + patches: object, +) -> tuple[_RedactPatch, ...] | _PatchRejected: + if not isinstance(patches, tuple) or not all(isinstance(patch, _RedactPatch) for patch in patches): + return _PatchRejected(_PatchRejectionCode.INVALID_PATCH) + expected_by_token = {entry.token: entry.mention_id for entry in bound.entries} + observed_tokens = tuple(patch.token for patch in patches) + if any(token not in expected_by_token for token in observed_tokens): + return _PatchRejected(_PatchRejectionCode.FOREIGN_TOKEN) + if len(set(observed_tokens)) != len(observed_tokens) or set(observed_tokens) != set(expected_by_token): + return _PatchRejected(_PatchRejectionCode.INVALID_PATCH) + mention_by_id = {item.mention.id: item.mention for item in bound.manifest.resolved.mentions} + target_by_datum = {target.datum_id: target.token for target in bound.manifest.resolved.clustered.detected.targets} + patch_by_token = {patch.token: patch for patch in patches} + for entry in bound.entries: + patch = patch_by_token[entry.token] + mention = mention_by_id[entry.mention_id] + if ( + patch.target is not target_by_datum[mention.target_datum_id] + or type(patch.start) is not int + or type(patch.end) is not int + or patch.start != mention.start + or patch.end != mention.end + or patch.replacement != _REDACT_REPLACEMENT + or mention.source_slice in patch.replacement + ): + return _PatchRejected(_PatchRejectionCode.INVALID_PATCH, target_by_datum[mention.target_datum_id]) + return tuple(patch_by_token[entry.token] for entry in bound.entries) + + +def _reconstruct_outputs( + bound: _BoundPatchManifest, + patches: tuple[_RedactPatch, ...], +) -> dict[_MentionTargetToken, str] | _PatchRejected: + resolved = bound.manifest.resolved + target_by_token = {target.token: target for target in resolved.clustered.detected.targets} + by_target: dict[_MentionTargetToken, list[_RedactPatch]] = {token: [] for token in target_by_token} + for patch in patches: + if patch.target not in by_target: + return _PatchRejected(_PatchRejectionCode.FOREIGN_TOKEN) + by_target[patch.target].append(patch) + outputs: dict[_MentionTargetToken, str] = {} + for token, target in target_by_token.items(): + cursor = 0 + parts: list[str] = [] + for patch in sorted(by_target[token], key=lambda item: (item.start, item.end)): + if patch.start < cursor or patch.end > len(target.text): + return _PatchRejected(_PatchRejectionCode.INVALID_PATCH, token) + parts.extend((target.text[cursor : patch.start], patch.replacement)) + cursor = patch.end + parts.append(target.text[cursor:]) + outputs[token] = "".join(parts) + return outputs + + +def _validate_returned( + bound: _BoundPatchManifest, + returned: object, +) -> dict[_MentionTargetToken, str] | _PatchRejected: + if not isinstance(returned, tuple) or not all(isinstance(item, _ReturnedRedact) for item in returned): + return _PatchRejected(_PatchRejectionCode.RELEASE_PREDICATE_FAILED) + known = frozenset(target.token for target in bound.manifest.resolved.clustered.detected.targets) + targets = tuple(item.target for item in returned) + if any(target not in known for target in targets): + return _PatchRejected(_PatchRejectionCode.FOREIGN_TOKEN) + if len(set(targets)) != len(targets) or set(targets) != set(known): + return _PatchRejected(_PatchRejectionCode.RELEASE_PREDICATE_FAILED) + if not all(isinstance(item.output, str) for item in returned): + return _PatchRejected(_PatchRejectionCode.RELEASE_PREDICATE_FAILED) + return {item.target: item.output for item in returned} diff --git a/src/anonymizer/engine/execution/role_policy.py b/src/anonymizer/engine/execution/role_policy.py new file mode 100644 index 00000000..c87c82ff --- /dev/null +++ b/src/anonymizer/engine/execution/role_policy.py @@ -0,0 +1,428 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Versioned structural replacement-role results for the private graph profile.""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import dataclass, field +from enum import Enum +from importlib.resources import files +from typing import TypeAlias, cast + +from anonymizer.engine.constants import DEFAULT_ENTITY_LABELS +from anonymizer.engine.execution.mention_admission import _AnchoredMention, _MentionId +from anonymizer.engine.execution.mention_resolution import _ClusteredGraph, _ClusterId +from anonymizer.engine.execution.phase7_contract import ( + _canonical_digest, + _load_phase7_contract, + _Phase7ContractRejected, + _Phase7StableSubstituteContract, +) + + +class _PrivateRoleValue: + def __repr__(self) -> str: + return f"" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private role values are not serializable") + + +class _RolePolicyVersion(str, Enum): + V1 = "phase6-role-result/v1" + + +@dataclass(frozen=True, slots=True, repr=False) +class _ReplacementRole(_PrivateRoleValue): + value: str + + +class _UnsupportedRoleReason(str, Enum): + UNSUPPORTED_ROLE = "unsupported_role" + + +@dataclass(frozen=True, slots=True, repr=False) +class _ClassifiedRole(_PrivateRoleValue): + role: _ReplacementRole + policy_version: _RolePolicyVersion + + +@dataclass(frozen=True, slots=True, repr=False) +class _UnsupportedRole(_PrivateRoleValue): + reason: _UnsupportedRoleReason + policy_version: _RolePolicyVersion + + +_RoleResult: TypeAlias = _ClassifiedRole | _UnsupportedRole + + +@dataclass(frozen=True, slots=True, repr=False) +class _RolePolicyProof(_PrivateRoleValue): + seal: object = field(compare=False) + snapshot: tuple[object, ...] + + +@dataclass(frozen=True, slots=True, repr=False) +class _RolePolicy(_PrivateRoleValue): + version: _RolePolicyVersion + mappings: tuple[tuple[str, _ReplacementRole], ...] + digest: str + policy_version: str = "" + dispositions: tuple[tuple[str, str | None], ...] = () + _proof: _RolePolicyProof | None = field(default=None, compare=False) + + @property + def result_version(self) -> _RolePolicyVersion: + return self.version + + +@dataclass(frozen=True, slots=True, repr=False) +class _ResolvedMention(_PrivateRoleValue): + mention: _AnchoredMention + cluster_id: _ClusterId + role_result: _RoleResult + + +@dataclass(frozen=True, slots=True, repr=False) +class _ResolvedGraph(_PrivateRoleValue): + clustered: _ClusteredGraph + mentions: tuple[_ResolvedMention, ...] + policy_version: _RolePolicyVersion + policy_digest: str + source_policy_version: str + _proof: _RolePolicyProof | None = field(default=None, compare=False) + + +class _RolePolicyRejectionCode(str, Enum): + UNSUPPORTED_ROLE = "unsupported_role" + + +@dataclass(frozen=True, slots=True, repr=False) +class _RolePolicyRejected(_PrivateRoleValue): + code: _RolePolicyRejectionCode + mention_id: _MentionId | None = None + + +_ROLE_POLICY_SEAL = object() +_RESOLVED_GRAPH_SEAL = object() +_REDACT_ROLE_POLICY_RESOURCE = "phase6_redact_role_policy.json" +_SUBSTITUTE_ROLE_POLICY_RESOURCE = "phase6_substitute_role_policy.json" + + +def _load_redact_role_policy() -> _RolePolicy | _RolePolicyRejected: + try: + payload = json.loads( + files("anonymizer.engine.execution").joinpath(_REDACT_ROLE_POLICY_RESOURCE).read_text(encoding="utf-8") + ) + if type(payload) is not dict or set(payload) != {"digest", "mappings", "version"}: + raise TypeError + digest = payload["digest"] + mappings = payload["mappings"] + version = payload["version"] + if type(digest) is not str or type(mappings) is not list or type(version) is not str: + raise TypeError + if mappings: + raise ValueError + parsed_mappings: list[tuple[str, str]] = [] + for mapping in mappings: + if type(mapping) is not list or len(mapping) != 2 or any(type(value) is not str for value in mapping): + raise TypeError + parsed_mappings.append((mapping[0], mapping[1])) + policy = _compile_role_policy(_RolePolicyVersion(version), tuple(parsed_mappings)) + if isinstance(policy, _RolePolicyRejected) or digest != policy.digest: + raise ValueError + return policy + except (OSError, TypeError, ValueError): + return _RolePolicyRejected(_RolePolicyRejectionCode.UNSUPPORTED_ROLE) + + +def _load_substitute_role_policy() -> _RolePolicy | _RolePolicyRejected: + """Load only the owner-frozen P0 Substitute disposition resource.""" + try: + contract = _load_phase7_contract() + if isinstance(contract, _Phase7ContractRejected): + raise ValueError + text = ( + files("anonymizer.engine.execution").joinpath(_SUBSTITUTE_ROLE_POLICY_RESOURCE).read_text(encoding="utf-8") + ) + payload = json.loads(text, object_pairs_hook=_object_without_duplicates) + return _compile_substitute_policy(payload, contract) + except (KeyError, OSError, TypeError, UnicodeEncodeError, ValueError): + return _RolePolicyRejected(_RolePolicyRejectionCode.UNSUPPORTED_ROLE) + + +def _compile_substitute_policy( + payload: object, + contract: _Phase7StableSubstituteContract, +) -> _RolePolicy: + if type(payload) is not dict or set(payload) != {"dispositions", "result_version", "version"}: + raise TypeError + policy_payload = cast(dict[str, object], payload) + result_version = policy_payload["result_version"] + policy_version = policy_payload["version"] + dispositions = _compile_substitute_dispositions(policy_payload["dispositions"], contract) + if ( + type(result_version) is not str + or type(policy_version) is not str + or result_version != contract.phase6_result_version + or policy_version != contract.phase6_policy_version + or _canonical_digest(policy_payload) != contract.phase6_policy_digest + ): + raise ValueError + mappings = tuple((label, role) for label, role in dispositions if role is not None) + policy = _compile_role_policy( + _RolePolicyVersion(result_version), + mappings, + policy_version=policy_version, + dispositions=dispositions, + ) + if isinstance(policy, _RolePolicyRejected) or policy.digest != contract.phase6_policy_digest: + raise ValueError + return policy + + +def _compile_substitute_dispositions( + payload: object, + contract: _Phase7StableSubstituteContract, +) -> tuple[tuple[str, str | None], ...]: + if type(payload) is not dict: + raise TypeError + dispositions = cast(dict[str, object], payload) + if set(dispositions) != set(DEFAULT_ENTITY_LABELS) or len(dispositions) != len(DEFAULT_ENTITY_LABELS): + raise ValueError + roles = {role.name for role in contract.roles} + result: list[tuple[str, str | None]] = [] + for label in sorted(dispositions): + role = dispositions[label] + if role is not None and (type(role) is not str or role not in roles): + raise ValueError + result.append((label, role)) + return tuple(result) + + +def _compile_role_policy( + version: _RolePolicyVersion, + mappings: tuple[tuple[str, str], ...], + *, + policy_version: str | None = None, + dispositions: tuple[tuple[str, str | None], ...] = (), +) -> _RolePolicy | _RolePolicyRejected: + selected_version = ( + version.value if policy_version is None and isinstance(version, _RolePolicyVersion) else policy_version + ) + if ( + version is not _RolePolicyVersion.V1 + or not isinstance(mappings, tuple) + or not _valid_text(selected_version) + or not isinstance(dispositions, tuple) + ): + return _RolePolicyRejected(_RolePolicyRejectionCode.UNSUPPORTED_ROLE) + validated: list[tuple[str, _ReplacementRole]] = [] + labels: set[str] = set() + for mapping in mappings: + if not isinstance(mapping, tuple) or len(mapping) != 2: + return _RolePolicyRejected(_RolePolicyRejectionCode.UNSUPPORTED_ROLE) + label, role = mapping + if not _valid_text(label) or not _valid_text(role) or label in labels: + return _RolePolicyRejected(_RolePolicyRejectionCode.UNSUPPORTED_ROLE) + labels.add(label) + validated.append((label, _ReplacementRole(role))) + ordered = tuple(sorted(validated, key=lambda item: item[0])) + if dispositions and not _valid_dispositions(dispositions, ordered): + return _RolePolicyRejected(_RolePolicyRejectionCode.UNSUPPORTED_ROLE) + canonical_dispositions = tuple(sorted(dispositions)) + digest = _policy_digest(version, ordered, selected_version, canonical_dispositions) + values = (version, ordered, digest, selected_version, canonical_dispositions) + snapshot = _role_policy_snapshot(*values) + return _RolePolicy(*values, _RolePolicyProof(_ROLE_POLICY_SEAL, snapshot)) + + +def _classify_roles( + clustered: _ClusteredGraph, + policy: _RolePolicy, +) -> _ResolvedGraph | _RolePolicyRejected: + if not _is_admitted_policy(policy) or not isinstance(clustered, _ClusteredGraph): + return _RolePolicyRejected(_RolePolicyRejectionCode.UNSUPPORTED_ROLE) + cluster_by_mention: dict[_MentionId, _ClusterId] = {} + for cluster in clustered.clusters: + for mention_id in cluster.ordered_mention_ids: + if mention_id in cluster_by_mention: + return _RolePolicyRejected(_RolePolicyRejectionCode.UNSUPPORTED_ROLE, mention_id) + cluster_by_mention[mention_id] = cluster.id + expected = {mention.id for mention in clustered.detected.mentions} + if set(cluster_by_mention) != expected: + return _RolePolicyRejected(_RolePolicyRejectionCode.UNSUPPORTED_ROLE) + mapping = dict(policy.mappings) + resolved = tuple( + _ResolvedMention( + mention, + cluster_by_mention[mention.id], + _ClassifiedRole(mapping[mention.detector_label], policy.version) + if mention.detector_label in mapping + else _UnsupportedRole(_UnsupportedRoleReason.UNSUPPORTED_ROLE, policy.version), + ) + for mention in clustered.detected.mentions + ) + values = (clustered, resolved, policy.version, policy.digest, policy.policy_version) + snapshot = _resolved_graph_snapshot(*values) + return _ResolvedGraph(*values, _RolePolicyProof(_RESOLVED_GRAPH_SEAL, snapshot)) + + +def _is_admitted_policy(policy: object) -> bool: + if not isinstance(policy, _RolePolicy) or policy._proof is None: + return False + snapshot = _role_policy_snapshot( + policy.version, + policy.mappings, + policy.digest, + policy.policy_version, + policy.dispositions, + ) + return ( + policy._proof.seal is _ROLE_POLICY_SEAL + and policy._proof.snapshot == snapshot + and policy.digest == _policy_digest(policy.version, policy.mappings, policy.policy_version, policy.dispositions) + ) + + +def _is_admitted_resolved_graph(value: object, policy: _RolePolicy) -> bool: + if not isinstance(value, _ResolvedGraph) or value._proof is None or not _is_admitted_policy(policy): + return False + if ( + value.policy_version is not policy.version + or value.source_policy_version != policy.policy_version + or value.policy_digest != policy.digest + or value._proof.seal is not _RESOLVED_GRAPH_SEAL + or value._proof.snapshot + != _resolved_graph_snapshot( + value.clustered, + value.mentions, + value.policy_version, + value.policy_digest, + value.source_policy_version, + ) + ): + return False + expected_mentions = value.clustered.detected.mentions + if len(value.mentions) != len(expected_mentions) or any( + result.mention is not mention for result, mention in zip(value.mentions, expected_mentions, strict=True) + ): + return False + mapping = dict(policy.mappings) + cluster_ids = { + mention_id: cluster.id for cluster in value.clustered.clusters for mention_id in cluster.ordered_mention_ids + } + for result in value.mentions: + if result.cluster_id is not cluster_ids.get(result.mention.id): + return False + expected_role = mapping.get(result.mention.detector_label) + if expected_role is None: + if not isinstance(result.role_result, _UnsupportedRole): + return False + elif not isinstance(result.role_result, _ClassifiedRole) or result.role_result.role != expected_role: + return False + if result.role_result.policy_version is not policy.version: + return False + return True + + +def _policy_digest( + version: _RolePolicyVersion, + mappings: tuple[tuple[str, _ReplacementRole], ...], + policy_version: str, + dispositions: tuple[tuple[str, str | None], ...], +) -> str: + if dispositions: + payload = { + "dispositions": dict(dispositions), + "result_version": version.value, + "version": policy_version, + } + else: + payload = { + "mappings": [[label, role.value] for label, role in mappings], + "version": version.value, + } + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def _role_policy_snapshot( + version: _RolePolicyVersion, + mappings: tuple[tuple[str, _ReplacementRole], ...], + digest: str, + policy_version: str, + dispositions: tuple[tuple[str, str | None], ...], +) -> tuple[object, ...]: + return ( + version.value, + tuple((label, role.value) for label, role in mappings), + digest, + policy_version, + dispositions, + ) + + +def _resolved_graph_snapshot( + clustered: _ClusteredGraph, + mentions: tuple[_ResolvedMention, ...], + result_version: _RolePolicyVersion, + policy_digest: str, + policy_version: str, +) -> tuple[object, ...]: + return ( + clustered, + tuple( + ( + result.mention, + result.cluster_id, + type(result.role_result), + result.role_result.role.value if isinstance(result.role_result, _ClassifiedRole) else None, + result.role_result.policy_version.value, + ) + for result in mentions + ), + result_version.value, + policy_digest, + policy_version, + ) + + +def _valid_dispositions( + dispositions: tuple[tuple[str, str | None], ...], + mappings: tuple[tuple[str, _ReplacementRole], ...], +) -> bool: + if any( + not isinstance(item, tuple) + or len(item) != 2 + or not _valid_text(item[0]) + or (item[1] is not None and not _valid_text(item[1])) + for item in dispositions + ): + return False + labels = tuple(label for label, _role in dispositions) + return len(set(labels)) == len(labels) and {(label, role) for label, role in dispositions if role is not None} == { + (label, role.value) for label, role in mappings + } + + +def _object_without_duplicates(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + if key in result: + raise ValueError + result[key] = value + return result + + +def _valid_text(value: object) -> bool: + if not isinstance(value, str) or not value: + return False + try: + value.encode("utf-8") + except UnicodeEncodeError: + return False + return True diff --git a/src/anonymizer/engine/ndd/adapter.py b/src/anonymizer/engine/ndd/adapter.py index dd6c8001..07e167a5 100644 --- a/src/anonymizer/engine/ndd/adapter.py +++ b/src/anonymizer/engine/ndd/adapter.py @@ -29,8 +29,11 @@ from data_designer.config.utils.constants import TRACE_COLUMN_POSTFIX from data_designer.config.utils.trace_type import TraceType +from anonymizer.engine.execution.context_observations import _private_context_observation_session +from anonymizer.engine.private_row_verification import PRIVATE_CORRELATION_COLUMN from anonymizer.interface.errors import AnonymizerWorkflowError from anonymizer.measurement import current_collector, record_ndd_workflow +from anonymizer.measurement.session import suppress_measurement if TYPE_CHECKING: import pandas as pd @@ -42,6 +45,7 @@ _TRACEABLE_LLM_COLUMN_TYPES = (LLMTextColumnConfig, LLMStructuredColumnConfig) _MODEL_TRACE_COLUMN: ContextVar[str | None] = ContextVar("anonymizer_dd_model_trace_column", default=None) _MODEL_TRACE_PURPOSE: ContextVar[str | None] = ContextVar("anonymizer_dd_model_trace_purpose", default=None) +_PRIVATE_EXECUTION: ContextVar[bool] = ContextVar("anonymizer_private_ndd_execution", default=False) @dataclass(frozen=True) @@ -53,12 +57,54 @@ class FailedRecord: reason: str -@dataclass(frozen=True) +@dataclass(frozen=True, slots=True, repr=False) +class _FailedRowEvidence: + """Private binding between one opaque row token and one public failure.""" + + row_token: str + record: FailedRecord + + def __repr__(self) -> str: + return "" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private failed row evidence is not serializable") + + +@dataclass(frozen=True, repr=False) class WorkflowRunResult: """Result of a single NDD workflow execution.""" dataframe: pd.DataFrame failed_records: list[FailedRecord] + failed_row_evidence: tuple[_FailedRowEvidence, ...] = () + + @property + def failed_row_tokens(self) -> tuple[str, ...]: + return tuple(evidence.row_token for evidence in self.failed_row_evidence) + + def __repr__(self) -> str: + return "" + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private workflow results are not serializable") + + +def _missing_private_row_tokens(input_df: pd.DataFrame, output_df: pd.DataFrame) -> tuple[str, ...]: + """Return missing opaque tokens without consulting public record identifiers.""" + if PRIVATE_CORRELATION_COLUMN not in input_df.columns or PRIVATE_CORRELATION_COLUMN not in output_df.columns: + return () + expected = tuple(input_df[PRIVATE_CORRELATION_COLUMN].tolist()) + observed = tuple(output_df[PRIVATE_CORRELATION_COLUMN].tolist()) + if ( + not all(isinstance(token, str) and token for token in (*expected, *observed)) + or len(set(expected)) != len(expected) + or len(set(observed)) != len(observed) + or not set(observed).issubset(expected) + ): + return () + observed_set = set(observed) + return tuple(token for token in expected if token not in observed_set) @dataclass(frozen=True) @@ -292,6 +338,16 @@ def consume_input_tokens(self) -> int: self._cumulative_input_tokens = 0 return tokens + @contextmanager + def private_execution(self) -> Iterator[None]: + """Use invocation-ephemeral artifacts and suppress ambient collection.""" + token = _PRIVATE_EXECUTION.set(True) + try: + with _private_context_observation_session(), suppress_measurement(): + yield + finally: + _PRIVATE_EXECUTION.reset(token) + def _add_input_tokens(self, model_usage: dict[str, Any] | None) -> None: input_tokens = 0 for usage in (model_usage or {}).values(): @@ -349,7 +405,11 @@ def run_workflow( else len(workflow_input_df) ) started = time.perf_counter() - collector = current_collector() + private_execution = _PRIVATE_EXECUTION.get() + collector = None if private_execution else current_collector() + task_trace_enabled = ( + False if private_execution else True if collector and collector.dd_task_trace_enabled else None + ) trace_plan = _DDMessageTracePlan.from_columns( columns=columns, model_configs=model_configs, @@ -358,7 +418,7 @@ def run_workflow( columns = trace_plan.columns usage_probe = _DataDesignerUsageProbe( self._data_designer, - enabled=True, + enabled=not private_execution, collector=collector, workflow_name=workflow_name, private_trace_columns=trace_plan.private_columns, @@ -375,13 +435,19 @@ def run_workflow( config_builder.add_column(column) task_traces: list[_TaskTrace] = [] + workflow_error: AnonymizerWorkflowError | None = None try: - with self._run_lock, usage_probe, _temporary_dd_task_trace(self._data_designer, collector=collector): + with ( + self._run_lock, + usage_probe, + _temporary_dd_task_trace(self._data_designer, enabled=task_trace_enabled), + ): if preview_num_records is None: run_results = self._data_designer.create( config_builder, num_records=len(workflow_input_df), dataset_name=workflow_name, + **({"artifact_path": tmp_dir} if private_execution else {}), ) task_traces = _task_traces_from_result(run_results) output_df = run_results.load_dataset() @@ -396,17 +462,7 @@ def run_workflow( else: output_df = preview_results.dataset except Exception as exc: - logger.warning( - "Workflow failed for %d input record(s) on model(s) %s: %s", - record_count, - available_model_aliases, - exc, - ) - logger.debug( - "Workflow '%s' failure context: columns=%s", - workflow_name, - col_names, - ) + logger.warning("Workflow failed for %d input record(s).", record_count) try: usage_probe.flush_private_trace_records() except Exception: @@ -414,21 +470,26 @@ def run_workflow( _error_model_usage = usage_probe.model_usage() with self._run_lock: self._add_input_tokens(_error_model_usage) - record_ndd_workflow( - workflow_name=workflow_name, - model_aliases=model_aliases, - input_row_count=record_count, - seed_row_count=len(workflow_input_df), - output_row_count=None, - failed_record_count=None, - elapsed_sec=time.perf_counter() - started, - status="error", - preview_num_records=preview_num_records, - column_count=len(col_names), - column_names=col_names, - model_usage=_error_model_usage, - ) - raise AnonymizerWorkflowError(f"Workflow failed: {exc}") from exc + if not private_execution: + record_ndd_workflow( + workflow_name=workflow_name, + model_aliases=model_aliases, + input_row_count=record_count, + seed_row_count=len(workflow_input_df), + output_row_count=None, + failed_record_count=None, + elapsed_sec=time.perf_counter() - started, + status="error", + preview_num_records=preview_num_records, + column_count=len(col_names), + column_names=col_names, + model_usage=_error_model_usage, + ) + workflow_error = AnonymizerWorkflowError("Workflow failed") + del exc + + if workflow_error is not None: + raise workflow_error from None output_df = trace_plan.record_and_strip_native_traces( output_df=output_df, @@ -443,32 +504,46 @@ def run_workflow( usage_probe.flush_private_trace_records() logger.debug("NDD workflow '%s' returned %d records", workflow_name, len(output_df)) + accounted_input_df = ( + workflow_input_df.iloc[:preview_num_records].copy() + if preview_num_records is not None + else workflow_input_df + ) failed_records = self._detect_missing_records( workflow_name=workflow_name, - input_df=( - workflow_input_df.iloc[:preview_num_records].copy() - if preview_num_records is not None - else workflow_input_df - ), + input_df=accounted_input_df, output_df=output_df, ) _success_model_usage = usage_probe.model_usage() with self._run_lock: self._add_input_tokens(_success_model_usage) - record_ndd_workflow( - workflow_name=workflow_name, - model_aliases=model_aliases, - input_row_count=record_count, - seed_row_count=len(workflow_input_df), - output_row_count=len(output_df), - failed_record_count=len(failed_records), - elapsed_sec=time.perf_counter() - started, - preview_num_records=preview_num_records, - column_count=len(col_names), - column_names=col_names, - model_usage=_success_model_usage, + if not private_execution: + record_ndd_workflow( + workflow_name=workflow_name, + model_aliases=model_aliases, + input_row_count=record_count, + seed_row_count=len(workflow_input_df), + output_row_count=len(output_df), + failed_record_count=len(failed_records), + elapsed_sec=time.perf_counter() - started, + preview_num_records=preview_num_records, + column_count=len(col_names), + column_names=col_names, + model_usage=_success_model_usage, + ) + missing_tokens = _missing_private_row_tokens(accounted_input_df, output_df) + failed_row_evidence = ( + tuple( + _FailedRowEvidence(token, record) for token, record in zip(missing_tokens, failed_records, strict=True) + ) + if len(missing_tokens) == len(failed_records) + else () + ) + return WorkflowRunResult( + dataframe=output_df, + failed_records=failed_records, + failed_row_evidence=failed_row_evidence, ) - return WorkflowRunResult(dataframe=output_df, failed_records=failed_records) def build_config( self, @@ -1023,8 +1098,8 @@ def _model_trace_usage(response: Any) -> Any: @contextmanager -def _temporary_dd_task_trace(data_designer: DataDesigner, *, collector: Any | None) -> Iterator[None]: - if collector is None or not collector.dd_task_trace_enabled: +def _temporary_dd_task_trace(data_designer: DataDesigner, *, enabled: bool | None) -> Iterator[None]: + if enabled is None: yield return @@ -1034,7 +1109,7 @@ def _temporary_dd_task_trace(data_designer: DataDesigner, *, collector: Any | No yield return - traced_run_config = _run_config_with_async_trace(original_run_config) + traced_run_config = _run_config_with_async_trace(original_run_config, enabled=enabled) set_run_config(traced_run_config) try: yield @@ -1042,12 +1117,12 @@ def _temporary_dd_task_trace(data_designer: DataDesigner, *, collector: Any | No set_run_config(original_run_config) -def _run_config_with_async_trace(run_config: Any) -> Any: +def _run_config_with_async_trace(run_config: Any, *, enabled: bool) -> Any: model_copy = getattr(run_config, "model_copy", None) if callable(model_copy): - return model_copy(update={"async_trace": True}) + return model_copy(update={"async_trace": enabled}) if isinstance(run_config, RunConfig): - return run_config.model_copy(update={"async_trace": True}) + return run_config.model_copy(update={"async_trace": enabled}) return run_config diff --git a/src/anonymizer/engine/private_row_verification.py b/src/anonymizer/engine/private_row_verification.py new file mode 100644 index 00000000..24114a22 --- /dev/null +++ b/src/anonymizer/engine/private_row_verification.py @@ -0,0 +1,256 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Invocation-scoped row accounting and accepted-detection verification. + +This module is intentionally private: it is an engine invariant, not a result, +trace, or dataframe API. Correlations and frozen entity values never leave an +invocation, and the verifier is invalidated before a result is returned. +""" + +from __future__ import annotations + +import hashlib +import json +import uuid +from dataclasses import dataclass +from enum import Enum +from typing import TYPE_CHECKING + +from anonymizer.engine.constants import COL_FINAL_ENTITIES, COL_TARGET_WORK_ID, COL_TEXT + +if TYPE_CHECKING: + import pandas as pd + + +PRIVATE_CORRELATION_COLUMN = COL_TARGET_WORK_ID + + +class _TerminalOutcome(str, Enum): + SUCCESS = "success" + FAILED = "failed" + CANCELLED = "cancelled" + + +@dataclass(frozen=True) +class _SafeFailure: + code: str + stage: str + scope: str + retry_owner: str = "anonymizer" + message: str = "private row verification failed" + + +class PrivateRowVerificationError(RuntimeError): + """Sanitized private-engine error; deliberately carries no causal exception.""" + + def __init__(self, failure: _SafeFailure) -> None: + self.failure = failure + super().__init__( + f"private_row_verification code={failure.code} stage={failure.stage} " + f"scope={failure.scope} retry_owner={failure.retry_owner}: {failure.message}" + ) + + +class _InvocationRowVerifier: + """One-shot verifier for one private engine invocation.""" + + def __init__(self, dataframe: pd.DataFrame, *, correlations: tuple[str, ...] | None = None) -> None: + if PRIVATE_CORRELATION_COLUMN in dataframe.columns: + raise PrivateRowVerificationError( + _SafeFailure("private_column_collision", "accept", "invocation", message="reserved private column") + ) + accepted = correlations if correlations is not None else tuple(uuid.uuid4().hex for _ in range(len(dataframe))) + if ( + len(accepted) != len(dataframe) + or len(set(accepted)) != len(accepted) + or not all(isinstance(value, str) and value for value in accepted) + ): + raise PrivateRowVerificationError( + _SafeFailure("correlation_invalid", "accept", "invocation", message="invalid private correlations") + ) + self._active = True + self._accepted = accepted + self._legacy_input_order = tuple(_stable_digest(value) for value in dataframe[COL_TEXT]) + self._legacy_identity_is_unambiguous = len(set(self._legacy_input_order)) == len(self._legacy_input_order) + self._frozen: dict[str, str] = {} + self._outcomes: dict[str, _TerminalOutcome] = {} + self._result_order: tuple[str, ...] = () + self._result_fingerprints: dict[str, str] = {} + + def bind(self, dataframe: pd.DataFrame) -> pd.DataFrame: + self._require_active() + bound = dataframe.copy() + bound[PRIVATE_CORRELATION_COLUMN] = list(self._accepted) + return bound + + def bind_complete_stage_output(self, dataframe: pd.DataFrame) -> pd.DataFrame: + """Require a stage to preserve correlation or prove exact legacy order. + + Legacy in-process doubles may reconstruct a complete frame without + unknown passthrough columns. They are accepted only if every accepted + text fingerprint is unique and the complete fingerprint sequence is + identical to the accepted sequence. Duplicate texts make a reordered + legacy frame indistinguishable from the original, so they require the + private correlation column. Real stages carry that column. + """ + self._require_active() + if PRIVATE_CORRELATION_COLUMN in dataframe.columns: + self._validate_correlations(dataframe, stage="stage_boundary") + return dataframe + if ( + COL_TEXT not in dataframe.columns + or len(dataframe) != len(self._accepted) + or not self._legacy_identity_is_unambiguous + or tuple(_stable_digest(value) for value in dataframe[COL_TEXT]) != self._legacy_input_order + ): + raise self._error("correlation_missing", "stage_boundary", "invocation") + bound = dataframe.copy() + bound[PRIVATE_CORRELATION_COLUMN] = self._accepted + return bound + + def freeze_accepted_detections(self, dataframe: pd.DataFrame) -> None: + self._require_active() + correlations = self._validate_correlations(dataframe, stage="detection") + if COL_FINAL_ENTITIES not in dataframe.columns: + raise self._error("accepted_detection_missing", "detection", "invocation") + self._frozen = { + correlation: _stable_digest(value) + for correlation, value in zip(correlations, dataframe[COL_FINAL_ENTITIES], strict=True) + } + self._mark_absent_as_failed(correlations) + + def finish(self, dataframe: pd.DataFrame, *, cancelled: bool = False) -> pd.DataFrame: + """Verify terminal cardinality and accepted detections, then remove state.""" + self._require_active() + try: + correlations = self._validate_correlations(dataframe, stage="result") + self._mark_absent_as_failed(correlations) + expected_successes = set(self._accepted) - set(self._outcomes) + if set(correlations) != expected_successes: + raise self._error("terminal_row_mismatch", "result", "row") + if COL_FINAL_ENTITIES not in dataframe.columns: + raise self._error("accepted_detection_missing", "result", "invocation") + for correlation, value in zip(correlations, dataframe[COL_FINAL_ENTITIES], strict=True): + if self._frozen.get(correlation) != _stable_digest(value): + raise self._error("accepted_detection_tampered", "result", "row") + self._outcomes.update({correlation: _TerminalOutcome.SUCCESS for correlation in correlations}) + self._result_order = tuple(correlations) + public_result = dataframe.drop(columns=[PRIVATE_CORRELATION_COLUMN], errors="ignore") + self._result_fingerprints = { + correlation: _stable_digest(row.to_dict()) + for correlation, (_index, row) in zip( + correlations, + public_result.iterrows(), + strict=True, + ) + } + return public_result + except BaseException: + # A verifier rejection is an invocation failure for every row that + # did not already receive a terminal state. Do this before the + # verifier is invalidated so the outer sanitizer cannot lose row + # accounting while translating the error. + self._complete_remaining(_TerminalOutcome.FAILED) + raise + finally: + self._active = False + self._frozen.clear() + + def abort(self, *, cancelled: bool) -> None: + """Close an interrupted invocation with one terminal outcome per accepted row.""" + if not self._active: + return + self._complete_remaining(_TerminalOutcome.CANCELLED if cancelled else _TerminalOutcome.FAILED) + self._active = False + self._frozen.clear() + + def abort_with_failure(self, *, stage: str, cause: BaseException) -> PrivateRowVerificationError: + """Close an invocation and return a failure that cannot expose ``cause``. + + The caller raises the returned error after leaving its exception handler. + This prevents Python from retaining the original exception through the + otherwise-accessible ``__context__`` attribute. + """ + del cause + self.abort(cancelled=False) + return self._error("invocation_failed", stage, "invocation") + + def take_terminal_outcomes(self) -> tuple[tuple[str, _TerminalOutcome], ...]: + """Consume the invocation-private correlation accounting exactly once.""" + if self._active: + raise self._error("invocation_active", "lifecycle", "invocation") + outcomes = tuple((correlation, self._outcomes[correlation]) for correlation in self._accepted) + self._accepted = () + self._outcomes.clear() + return outcomes + + def take_result_order(self) -> tuple[str, ...]: + """Consume verified successful-row correlations in dataframe order.""" + if self._active: + raise self._error("invocation_active", "lifecycle", "invocation") + result_order = self._result_order + self._result_order = () + return result_order + + def verify_returned_rows(self, dataframe: pd.DataFrame, result_order: tuple[str, ...]) -> None: + """Verify token-to-row binding after the backend returns from ``finish``.""" + if self._active: + raise self._error("invocation_active", "lifecycle", "invocation") + try: + if len(result_order) != len(dataframe) or set(result_order) != set(self._result_fingerprints): + raise self._error("returned_row_mismatch", "return", "invocation") + for token, (_index, row) in zip(result_order, dataframe.iterrows(), strict=True): + if self._result_fingerprints.get(token) != _stable_digest(row.to_dict()): + raise self._error("returned_row_tampered", "return", "row") + finally: + self._result_fingerprints.clear() + + def _validate_correlations(self, dataframe: pd.DataFrame, *, stage: str) -> list[str]: + if PRIVATE_CORRELATION_COLUMN not in dataframe.columns: + raise self._error("correlation_missing", stage, "invocation") + correlations = dataframe[PRIVATE_CORRELATION_COLUMN].tolist() + if not all(isinstance(value, str) and value for value in correlations): + raise self._error("correlation_invalid", stage, "row") + if len(set(correlations)) != len(correlations): + raise self._error("correlation_duplicate", stage, "row") + unknown = set(correlations) - set(self._accepted) + if unknown: + raise self._error("correlation_unknown", stage, "row") + return correlations + + def _mark_absent_as_failed(self, correlations: list[str]) -> None: + """Record dropped accepted rows before any invocation-wide terminal state. + + Row failure has precedence over later cancellation or invocation failure. + A surviving row is never inferred from its contents; only its private + correlation proves provenance. + """ + for correlation in set(self._accepted) - set(correlations): + self._outcomes.setdefault(correlation, _TerminalOutcome.FAILED) + + def _complete_remaining(self, terminal: _TerminalOutcome) -> None: + for correlation in self._accepted: + self._outcomes.setdefault(correlation, terminal) + + def _error(self, code: str, stage: str, scope: str) -> PrivateRowVerificationError: + return PrivateRowVerificationError(_SafeFailure(code, stage, scope)) + + def _require_active(self) -> None: + if not self._active: + raise PrivateRowVerificationError(_SafeFailure("invocation_closed", "lifecycle", "invocation")) + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private invocation verifier is not serializable") + + def __repr__(self) -> str: + return "" + + +def _stable_digest(value: object) -> str: + """Hash private detection data without serializing it into a public artifact.""" + try: + encoded = json.dumps(value, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8") + except (TypeError, ValueError): + encoded = repr(value).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() diff --git a/src/anonymizer/engine/replace/llm_replace_workflow.py b/src/anonymizer/engine/replace/llm_replace_workflow.py index 0a24c582..03711b64 100644 --- a/src/anonymizer/engine/replace/llm_replace_workflow.py +++ b/src/anonymizer/engine/replace/llm_replace_workflow.py @@ -6,7 +6,7 @@ import json import logging from collections import Counter -from dataclasses import dataclass +from dataclasses import dataclass, field import pandas as pd from data_designer.config.column_configs import LLMStructuredColumnConfig @@ -23,7 +23,7 @@ COL_REPLACEMENT_MAP_SOURCE, ENTITY_LABEL_EXAMPLES, ) -from anonymizer.engine.ndd.adapter import FailedRecord, NddAdapter +from anonymizer.engine.ndd.adapter import FailedRecord, NddAdapter, _FailedRowEvidence from anonymizer.engine.ndd.model_loader import resolve_model_alias from anonymizer.engine.prompt_utils import substitute_placeholders from anonymizer.engine.row_partitioning import merge_and_reorder, split_rows @@ -43,6 +43,7 @@ class LlmReplaceResult: dataframe: pd.DataFrame failed_records: list[FailedRecord] + failed_row_evidence: tuple[_FailedRowEvidence, ...] = field(default=(), repr=False) class LlmReplaceWorkflow: @@ -120,6 +121,7 @@ def generate_map_only( return LlmReplaceResult( dataframe=combined.drop(columns=list(_INTERNAL_COLUMNS), errors="ignore"), failed_records=run_result.failed_records, + failed_row_evidence=run_result.failed_row_evidence, ) diff --git a/src/anonymizer/engine/replace/replace_runner.py b/src/anonymizer/engine/replace/replace_runner.py index 8da79241..7615022d 100644 --- a/src/anonymizer/engine/replace/replace_runner.py +++ b/src/anonymizer/engine/replace/replace_runner.py @@ -4,7 +4,7 @@ from __future__ import annotations import logging -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import cast import pandas as pd @@ -28,7 +28,7 @@ from anonymizer.engine.evaluation.replace.attribute_fidelity_judge import AttributeFidelityJudgeWorkflow from anonymizer.engine.evaluation.replace.relational_consistency_judge import RelationalConsistencyJudgeWorkflow from anonymizer.engine.evaluation.replace.type_fidelity_judge import TypeFidelityJudgeWorkflow -from anonymizer.engine.ndd.adapter import RECORD_ID_COLUMN, FailedRecord, NddAdapter +from anonymizer.engine.ndd.adapter import RECORD_ID_COLUMN, FailedRecord, NddAdapter, _FailedRowEvidence from anonymizer.engine.replace.llm_replace_workflow import LlmReplaceWorkflow from anonymizer.engine.replace.strategies import apply_local_replace_strategy, apply_replacement_map from anonymizer.measurement import stage_timer @@ -42,6 +42,7 @@ class ReplacementResult: dataframe: pd.DataFrame failed_records: list[FailedRecord] + failed_row_evidence: tuple[_FailedRowEvidence, ...] = field(default=(), repr=False) class ReplacementWorkflow: @@ -88,6 +89,7 @@ def run( if isinstance(replace_method, (Annotate, Redact, Hash)): local_df = apply_local_replace_strategy(dataframe, strategy=replace_method) failed_records: list[FailedRecord] = [] + failed_row_evidence: tuple[_FailedRowEvidence, ...] = () elif isinstance(replace_method, Substitute): if self._llm_workflow is None: raise ValueError("Substitute requires an llm_workflow, but none was provided.") @@ -100,10 +102,15 @@ def run( ) local_df = apply_replacement_map(map_result.dataframe) failed_records = list(map_result.failed_records) + failed_row_evidence = map_result.failed_row_evidence else: raise ValueError(f"Unsupported replace method: {type(replace_method).__name__}") - result = ReplacementResult(dataframe=local_df, failed_records=failed_records) + result = ReplacementResult( + dataframe=local_df, + failed_records=failed_records, + failed_row_evidence=failed_row_evidence, + ) measurement.update( output_row_count=len(result.dataframe), failed_record_count=len(result.failed_records), diff --git a/src/anonymizer/engine/rewrite/combined_rewrite_workflow.py b/src/anonymizer/engine/rewrite/combined_rewrite_workflow.py index 73e39f11..bfb306c5 100644 --- a/src/anonymizer/engine/rewrite/combined_rewrite_workflow.py +++ b/src/anonymizer/engine/rewrite/combined_rewrite_workflow.py @@ -526,6 +526,7 @@ def run( result = RewriteResult( dataframe=merge_and_reorder(entity_rows, passthrough_rows), failed_records=run_result.failed_records, + failed_row_evidence=run_result.failed_row_evidence, ) measurement.update( output_row_count=len(result.dataframe), diff --git a/src/anonymizer/engine/rewrite/rewrite_workflow.py b/src/anonymizer/engine/rewrite/rewrite_workflow.py index 5b7fad74..693c2590 100644 --- a/src/anonymizer/engine/rewrite/rewrite_workflow.py +++ b/src/anonymizer/engine/rewrite/rewrite_workflow.py @@ -4,7 +4,7 @@ from __future__ import annotations import logging -from dataclasses import dataclass +from dataclasses import dataclass, field import pandas as pd from data_designer.config.models import ModelConfig @@ -32,7 +32,7 @@ COL_WEIGHTED_LEAKAGE_RATE, ) from anonymizer.engine.evaluation.detection_judge import DetectionJudgeWorkflow -from anonymizer.engine.ndd.adapter import RECORD_ID_COLUMN, FailedRecord, NddAdapter +from anonymizer.engine.ndd.adapter import RECORD_ID_COLUMN, FailedRecord, NddAdapter, _FailedRowEvidence from anonymizer.engine.replace.llm_replace_workflow import LlmReplaceWorkflow from anonymizer.engine.rewrite.domain_classification import DomainClassificationWorkflow from anonymizer.engine.rewrite.evaluate import EvaluateWorkflow @@ -228,6 +228,7 @@ def _apply_passthrough_defaults( class RewriteResult: dataframe: pd.DataFrame failed_records: list[FailedRecord] + failed_row_evidence: tuple[_FailedRowEvidence, ...] = field(default=(), repr=False) # --------------------------------------------------------------------------- @@ -270,6 +271,7 @@ def run( ) -> RewriteResult: with stage_timer("RewriteWorkflow.run", input_row_count=len(dataframe)) as measurement: all_failed: list[FailedRecord] = [] + all_failed_row_evidence: list[_FailedRowEvidence] = [] entity_rows, passthrough_rows = split_rows(dataframe, column=COL_ENTITIES_BY_VALUE, predicate=_has_entities) measurement.update( @@ -297,6 +299,7 @@ def run( ) entity_rows = _join_new_columns(entity_rows, replace_result.dataframe) all_failed.extend(replace_result.failed_records) + all_failed_row_evidence.extend(replace_result.failed_row_evidence) # --- Step 2: domain, disposition, QA, rewrite (single adapter call) --- pipeline_columns = [ @@ -325,9 +328,10 @@ def run( ) entity_rows = _join_new_columns(entity_rows, pipeline_result.dataframe) all_failed.extend(pipeline_result.failed_records) + all_failed_row_evidence.extend(pipeline_result.failed_row_evidence) # --- Step 5: evaluate-repair loop --- - entity_rows, eval_repair_failed = self._run_evaluate_repair_loop( + entity_rows, eval_repair_failed, eval_repair_failed_evidence = self._run_evaluate_repair_loop( entity_rows, model_configs=model_configs, selected_models=selected_models, @@ -336,11 +340,16 @@ def run( preview_num_records=preview_num_records, ) all_failed.extend(eval_repair_failed) + all_failed_row_evidence.extend(eval_repair_failed_evidence) # --- Merge and return --- _apply_passthrough_defaults(passthrough_rows) combined = merge_and_reorder(entity_rows, passthrough_rows) - result = RewriteResult(dataframe=combined, failed_records=all_failed) + result = RewriteResult( + dataframe=combined, + failed_records=all_failed, + failed_row_evidence=tuple(all_failed_row_evidence), + ) measurement.update( output_row_count=len(result.dataframe), failed_record_count=len(result.failed_records), @@ -360,8 +369,9 @@ def _run_evaluate_repair_loop( privacy_goal: PrivacyGoal, evaluation: EvaluationCriteria, preview_num_records: int | None, - ) -> tuple[pd.DataFrame, list[FailedRecord]]: + ) -> tuple[pd.DataFrame, list[FailedRecord], tuple[_FailedRowEvidence, ...]]: all_failed: list[FailedRecord] = [] + all_failed_row_evidence: list[_FailedRowEvidence] = [] if COL_REPAIR_ITERATIONS not in df.columns: df[COL_REPAIR_ITERATIONS] = 0 @@ -384,6 +394,7 @@ def _run_evaluate_repair_loop( df = _join_new_columns(df, eval_result.dataframe, overwrite=True, seed_cols=eval_seed_cols) _normalize_evaluation_payloads(df) all_failed.extend(eval_result.failed_records) + all_failed_row_evidence.extend(eval_result.failed_row_evidence) replacement_unavailable = ( ~df[COL_REWRITE_REPLACEMENT_READY].apply(bool) @@ -425,6 +436,7 @@ def _run_evaluate_repair_loop( preview_num_records=preview_num_records, ) all_failed.extend(repair_result.failed_records) + all_failed_row_evidence.extend(repair_result.failed_row_evidence) repaired = repair_result.dataframe failing_rows = _join_new_columns(failing_rows, repaired) @@ -447,6 +459,7 @@ def _run_evaluate_repair_loop( ) _normalize_evaluation_payloads(failing_rows) all_failed.extend(eval_result.failed_records) + all_failed_row_evidence.extend(eval_result.failed_row_evidence) df = pd.concat([passing_rows, failing_rows], ignore_index=True) replacement_unavailable = ( @@ -465,7 +478,7 @@ def _run_evaluate_repair_loop( needs_review = needs_review | (df[COL_LEAKAGE_MASS].apply(float) > evaluation.flag_leakage_above) df[COL_NEEDS_HUMAN_REVIEW] = needs_review - return df, all_failed + return df, all_failed, tuple(all_failed_row_evidence) def _run_final_judge( self, diff --git a/src/anonymizer/engine/rewrite/workflow_utils.py b/src/anonymizer/engine/rewrite/workflow_utils.py index 43adc5f0..ece862a5 100644 --- a/src/anonymizer/engine/rewrite/workflow_utils.py +++ b/src/anonymizer/engine/rewrite/workflow_utils.py @@ -10,6 +10,7 @@ from data_designer.config.column_types import ColumnConfigT from anonymizer.engine.ndd.adapter import RECORD_ID_COLUMN +from anonymizer.engine.private_row_verification import PRIVATE_CORRELATION_COLUMN def derive_seed_columns( @@ -32,7 +33,7 @@ def derive_seed_columns( for col in columns: required.update(col.required_columns) - external = (required - produced) | {RECORD_ID_COLUMN} + external = (required - produced) | {RECORD_ID_COLUMN, PRIVATE_CORRELATION_COLUMN} return [c for c in df.columns if c in external] diff --git a/src/anonymizer/interface/_protection.py b/src/anonymizer/interface/_protection.py new file mode 100644 index 00000000..dac6bf23 --- /dev/null +++ b/src/anonymizer/interface/_protection.py @@ -0,0 +1,565 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Private graph-native protection domain and synchronous lifecycle boundary.""" + +from __future__ import annotations + +import hashlib +import json +import secrets +import threading +from dataclasses import dataclass +from enum import Enum +from typing import TYPE_CHECKING + +from data_designer.config.models import ModelConfig + +from anonymizer.config.anonymizer_config import AnonymizerConfig +from anonymizer.config.models import ModelSelection +from anonymizer.config.replace_strategies import Annotate, Redact, Substitute +from anonymizer.engine.execution.accounting_admission import _AccountingRejected +from anonymizer.engine.execution.accounting_plan import _AccountingLimits +from anonymizer.engine.execution.context_admission import _ContextRejected +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextExecutionContract, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, +) +from anonymizer.engine.execution.graph import _DatumId, _TextDatum, _trivial_graph +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.execution.mention_admission import _MentionLimits +from anonymizer.engine.execution.phase6_ndd_backend import _Phase6NddBackend +from anonymizer.engine.execution.phase6_plan import _Phase6Rejected +from anonymizer.engine.execution.phase6_runtime import _Phase6EffectBackend +from anonymizer.engine.execution.phase7_contract import ( + _is_admitted_phase7_contract, + _load_phase7_contract, + _Phase7StableSubstituteContract, +) +from anonymizer.engine.execution.phase7_ndd_backend import _Phase7NddBackend +from anonymizer.engine.execution.phase7_runtime import _Phase7EffectBackend +from anonymizer.engine.execution.protection_service import ( + _GraphProtectionFailed, + _GraphProtectionResult, + _GraphProtectionSucceeded, + _Phase6RedactProtectionService, + _Phase7SubstituteProtectionService, +) + +if TYPE_CHECKING: + from anonymizer.interface.anonymizer import Anonymizer + +_MAX_REF_BYTES = 256 +_MAX_SEGMENT_BYTES = 65_536 +_MAX_RECORD_BYTES = 32_768 +_MAX_BATCH_BYTES = 1_048_576 +_MAX_RECORDS = 128 +_PHASE6_MENTION_LIMITS = _MentionLimits(128, 64, 256, _MAX_RECORD_BYTES) +_PHASE6_MAX_EXPANDED_FRAME_BYTES = 65_536 +_CONTRACT_VERSION = "private-protection-v1" +_REDACT_PROFILE = "redact-release-v1" +_SUBSTITUTE_PROFILE = "stable-substitute-v1" +_IMPLEMENTATION_VERSION = "pandas-runtime-v1" + + +class _SafeRepr: + def __repr__(self) -> str: + return f"" + + +@dataclass(frozen=True, slots=True, repr=False) +class _RecordRef(_SafeRepr): + value: str + + def __post_init__(self) -> None: + if not isinstance(self.value, str) or not self.value or len(self.value.encode("utf-8")) > _MAX_REF_BYTES: + raise ValueError("record reference is invalid") + + +@dataclass(frozen=True, slots=True, repr=False) +class _TextSegment(_SafeRepr): + text: str + + def __post_init__(self) -> None: + if not isinstance(self.text, str) or len(self.text.encode("utf-8")) > _MAX_SEGMENT_BYTES: + raise ValueError("text segment is invalid") + + +@dataclass(frozen=True, slots=True, repr=False) +class _ProtectionRecord(_SafeRepr): + ref: _RecordRef + segments: tuple[_TextSegment, ...] + + +class _CompileCode(str, Enum): + INVALID = "invalid" + UNSUPPORTED = "unsupported" + REJECTED = "rejected" + + +@dataclass(frozen=True, slots=True, repr=False) +class _PlanInvalid(_SafeRepr): + code: _CompileCode = _CompileCode.INVALID + + +@dataclass(frozen=True, slots=True, repr=False) +class _PlanUnsupported(_SafeRepr): + code: _CompileCode = _CompileCode.UNSUPPORTED + + +@dataclass(frozen=True, slots=True, repr=False) +class _PlanRejected(_SafeRepr): + code: _CompileCode = _CompileCode.REJECTED + + +@dataclass(frozen=True, slots=True, repr=False) +class _ProtectionPlan(_SafeRepr): + profile: str + digest: str + invocation: _CompiledInvocation + phase7_contract: _Phase7StableSubstituteContract | None = None + max_records: int = _MAX_RECORDS + max_record_bytes: int = _MAX_RECORD_BYTES + max_batch_bytes: int = _MAX_BATCH_BYTES + + def __reduce__(self) -> str | tuple[object, ...]: + raise TypeError("private protection plan is not serializable") + + +_CompileResult = _ProtectionPlan | _PlanInvalid | _PlanUnsupported | _PlanRejected + + +class _BatchFailureCode(str, Enum): + MALFORMED_BATCH = "malformed_batch" + DUPLICATE_REF = "duplicate_ref" + RECORD_TOO_LARGE = "record_too_large" + BATCH_TOO_LARGE = "batch_too_large" + TOO_MANY_RECORDS = "too_many_records" + UNSUPPORTED_CARDINALITY = "unsupported_cardinality" + + +class _ProtectionBatchError(ValueError): + def __init__(self, code: _BatchFailureCode) -> None: + self.code = code + super().__init__("private protection batch rejected") + + def __repr__(self) -> str: + return "" + + +class _FailureCode(str, Enum): + BUSY = "busy" + CLOSED = "closed" + CANCELLED_BEFORE_ADMISSION = "cancelled_before_admission" + INVOCATION_FAILED = "invocation_failed" + ROW_FAILED = "row_failed" + + +class _RetrySafety(str, Enum): + UNKNOWN = "unknown" + + +class _RetryOwner(str, Enum): + UNASSIGNED = "unassigned" + + +@dataclass(frozen=True, slots=True, repr=False) +class _SafeFailure(_SafeRepr): + code: _FailureCode + stage: str + scope: str + retry_safety: _RetrySafety = _RetrySafety.UNKNOWN + retry_owner: _RetryOwner = _RetryOwner.UNASSIGNED + + +@dataclass(frozen=True, slots=True, repr=False) +class _ProtectionApplied(_SafeRepr): + pass + + +@dataclass(frozen=True, slots=True, repr=False) +class _NoAcceptedDetections(_SafeRepr): + pass + + +_SuccessDisposition = _ProtectionApplied | _NoAcceptedDetections + + +@dataclass(frozen=True, slots=True, repr=False) +class _ProtectionReceipt(_SafeRepr): + contract_version: str + profile: str + implementation_version: str + terminal_accounting_verified: bool + accepted_detections_verified: bool + plan_digest: str + attempt_id: str + + +@dataclass(frozen=True, slots=True, repr=False) +class _Phase7ProtectionReceipt(_SafeRepr): + contract_version: str + profile: str + implementation_version: str + terminal_accounting_verified: bool + accepted_detections_verified: bool + + +@dataclass(frozen=True, slots=True, repr=False) +class _ProtectionSucceeded(_SafeRepr): + ref: _RecordRef + output: str + disposition: _SuccessDisposition + receipt: _ProtectionReceipt | _Phase7ProtectionReceipt + + +@dataclass(frozen=True, slots=True, repr=False) +class _Rejected(_SafeRepr): + ref: _RecordRef + failure: _SafeFailure + + +@dataclass(frozen=True, slots=True, repr=False) +class _Failed(_SafeRepr): + ref: _RecordRef + failure: _SafeFailure + + +@dataclass(frozen=True, slots=True, repr=False) +class _Cancelled(_SafeRepr): + ref: _RecordRef + failure: _SafeFailure + + +_RecordOutcome = _ProtectionSucceeded | _Rejected | _Failed | _Cancelled + + +@dataclass(frozen=True, slots=True, repr=False) +class _ProtectionRunRecord(_SafeRepr): + outcomes: tuple[_RecordOutcome, ...] + contract_version: str = _CONTRACT_VERSION + implementation_version: str = _IMPLEMENTATION_VERSION + + @property + def success_count(self) -> int: + return sum(isinstance(outcome, _ProtectionSucceeded) for outcome in self.outcomes) + + @property + def failure_count(self) -> int: + return len(self.outcomes) - self.success_count + + +@dataclass(frozen=True, slots=True, repr=False) +class _OperationPlan(_SafeRepr): + records: tuple[_ProtectionRecord, ...] + + +def _compile_protection_plan( + config: AnonymizerConfig, + selected_models: ModelSelection, + model_configs: list[ModelConfig], +) -> _CompileResult: + """Compile one explicitly release-qualified private protection profile.""" + if not isinstance(config, AnonymizerConfig): + return _PlanInvalid() + if isinstance(config.replace, Annotate): + return _PlanRejected() + if config.rewrite is not None: + return _PlanUnsupported() + if isinstance(config.replace, Redact): + if config.replace.format_template != "[REDACTED_{label}]" or not config.replace.normalize_label: + return _PlanRejected() + invocation = _CompiledInvocation.compile(config, selected_models, model_configs) + return _ProtectionPlan( + _REDACT_PROFILE, + _plan_fingerprint(invocation, profile=_REDACT_PROFILE), + invocation, + ) + if isinstance(config.replace, Substitute): + if config.replace.instructions is not None: + return _PlanRejected() + contract = _load_phase7_contract() + if not isinstance(contract, _Phase7StableSubstituteContract) or not _is_admitted_phase7_contract(contract): + return _PlanRejected() + invocation = _CompiledInvocation.compile(config, selected_models, model_configs) + max_records = min(_MAX_RECORDS, dict(contract.count_limits)["max_scopes_per_invocation"]) + return _ProtectionPlan( + _SUBSTITUTE_PROFILE, + _plan_fingerprint(invocation, profile=_SUBSTITUTE_PROFILE, contract_digest=contract.digest), + invocation, + contract, + max_records, + ) + return _PlanUnsupported() + + +def _plan_fingerprint( + invocation: _CompiledInvocation, + *, + profile: str, + contract_digest: str | None = None, +) -> str: + """Fingerprint the complete allowlisted private profile snapshot.""" + payload = { + "contract_version": _CONTRACT_VERSION, + "profile": profile, + "profile_contract_digest": contract_digest, + "implementation_version": _IMPLEMENTATION_VERSION, + "limits": { + "max_records": _MAX_RECORDS, + "max_record_bytes": _MAX_RECORD_BYTES, + "max_batch_bytes": _MAX_BATCH_BYTES, + }, + "invocation": { + "model_configs": [model.model_dump(mode="json") for model in invocation.model_configs], + "selected_models": invocation.selected_models.model_dump(mode="json"), + "gliner_detection_threshold": invocation.gliner_detection_threshold, + "validation_max_entities_per_call": invocation.validation_max_entities_per_call, + "validation_excerpt_window_chars": invocation.validation_excerpt_window_chars, + "entity_labels": invocation.entity_labels, + "replace_method": ( + invocation.replace_method.model_dump(mode="json") if invocation.replace_method is not None else None + ), + "rewrite": None, + }, + } + encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def _build_operation_plan(plan: _ProtectionPlan, records: object) -> _OperationPlan: + if not isinstance(records, tuple) or not records: + raise _ProtectionBatchError(_BatchFailureCode.MALFORMED_BATCH) + if len(records) > plan.max_records: + raise _ProtectionBatchError(_BatchFailureCode.TOO_MANY_RECORDS) + if not all(isinstance(record, _ProtectionRecord) for record in records): + raise _ProtectionBatchError(_BatchFailureCode.MALFORMED_BATCH) + typed_records = records + refs: list[str] = [] + for record in typed_records: + ref = getattr(record, "ref", None) + value = getattr(ref, "value", None) + if ( + not isinstance(ref, _RecordRef) + or not isinstance(value, str) + or not value + or len(value.encode("utf-8")) > _MAX_REF_BYTES + ): + raise _ProtectionBatchError(_BatchFailureCode.MALFORMED_BATCH) + refs.append(value) + if len(set(refs)) != len(refs): + raise _ProtectionBatchError(_BatchFailureCode.DUPLICATE_REF) + total = 0 + for record in typed_records: + segments = getattr(record, "segments", None) + if not isinstance(segments, tuple): + raise _ProtectionBatchError(_BatchFailureCode.MALFORMED_BATCH) + if len(segments) != 1: + raise _ProtectionBatchError(_BatchFailureCode.UNSUPPORTED_CARDINALITY) + segment = segments[0] + if not isinstance(segment, _TextSegment): + raise _ProtectionBatchError(_BatchFailureCode.MALFORMED_BATCH) + text = getattr(segment, "text", None) + if not isinstance(text, str): + raise _ProtectionBatchError(_BatchFailureCode.MALFORMED_BATCH) + size = len(text.encode("utf-8")) + if size > plan.max_record_bytes: + raise _ProtectionBatchError(_BatchFailureCode.RECORD_TOO_LARGE) + total += size + if total > plan.max_batch_bytes: + raise _ProtectionBatchError(_BatchFailureCode.BATCH_TOO_LARGE) + return _OperationPlan(typed_records) + + +class _ProtectionFlow(_SafeRepr): + """Reusable synchronous flow with non-waiting whole-invocation admission.""" + + def __init__( + self, + anonymizer: Anonymizer, + plan: _ProtectionPlan, + phase6_backend: _Phase6EffectBackend | None = None, + phase7_backend: _Phase7EffectBackend | None = None, + ) -> None: + self._plan = plan + backend = phase6_backend or _Phase6NddBackend(anonymizer._adapter, plan.invocation) + if plan.profile == _REDACT_PROFILE and plan.phase7_contract is None: + self._runtime = _Phase6RedactProtectionService( + backend, + mention_limits=_PHASE6_MENTION_LIMITS, + ) + elif ( + plan.profile == _SUBSTITUTE_PROFILE + and isinstance(plan.phase7_contract, _Phase7StableSubstituteContract) + and _is_admitted_phase7_contract(plan.phase7_contract) + ): + self._runtime = _Phase7SubstituteProtectionService( + backend, + ( + (lambda: phase7_backend) + if phase7_backend is not None + else (lambda: _Phase7NddBackend(anonymizer._adapter, plan.invocation)) + ), + mention_limits=_PHASE6_MENTION_LIMITS, + ) + else: + raise ValueError("private protection plan is not executable") + self._guard = threading.Lock() + self._state_lock = threading.Lock() + self._closed = False + self._adapter = anonymizer._adapter + + def protect(self, records: object, *, cancelled_before_admission: bool = False) -> _ProtectionRunRecord: + operation = _build_operation_plan(self._plan, records) + if cancelled_before_admission: + return self._reject_all(operation, _FailureCode.CANCELLED_BEFORE_ADMISSION) + with self._state_lock: + if self._closed: + return self._reject_all(operation, _FailureCode.CLOSED) + if not self._guard.acquire(blocking=False): + return self._reject_all(operation, _FailureCode.BUSY) + try: + with self._state_lock: + if self._closed: + return self._reject_all(operation, _FailureCode.CLOSED) + return self._execute(operation) + finally: + self._guard.release() + + def _execute(self, operation: _OperationPlan) -> _ProtectionRunRecord: + contract_digest = self._plan.phase7_contract.digest if self._plan.phase7_contract is not None else None + if ( + _plan_fingerprint( + self._plan.invocation, + profile=self._plan.profile, + contract_digest=contract_digest, + ) + != self._plan.digest + ): + return self._fail_all(operation) + graph = _trivial_graph( + tuple( + _TextDatum(_DatumId(f"datum-{index}"), record.segments[0].text) + for index, record in enumerate(operation.records) + ) + ) + try: + admitted = self._runtime.admit_context( + graph, + accounting_limits=_AccountingLimits( + max_datums=self._plan.max_records, + max_datum_bytes=self._plan.max_record_bytes, + max_graph_bytes=self._plan.max_batch_bytes, + max_stages=8, + ), + contract=_ContextExecutionContract( + profile=_ContextProfile.TARGET_CONTEXT_V1, + schema_version=_ContextSchemaVersion.V1, + limits=_ContextLimits( + max_context_members_per_target=0, + max_context_bytes_per_target=0, + max_total_context_references=0, + max_expanded_frame_bytes=_PHASE6_MAX_EXPANDED_FRAME_BYTES, + ), + allow_target_as_context=False, + ordering=_ContextOrdering.DECLARED, + required_artifacts=(_BackendArtifactClass.CONTEXT_REQUEST,), + ), + ) + if isinstance(admitted, (_AccountingRejected, _ContextRejected, _Phase6Rejected)): + return self._fail_all(operation) + with self._adapter.private_execution(): + if isinstance(self._runtime, _Phase7SubstituteProtectionService): + contract = self._plan.phase7_contract + if not isinstance(contract, _Phase7StableSubstituteContract): + return self._fail_all(operation) + execution = self._runtime.protect(admitted, contract=contract) + else: + execution = self._runtime.protect( + admitted, + invocation=self._plan.invocation, + ) + except Exception: + return self._fail_all(operation) + + try: + return self._materialize_outcomes(operation, execution) + except Exception as cause: + del cause + return self._fail_all(operation) + + def _materialize_outcomes( + self, + operation: _OperationPlan, + execution: _GraphProtectionResult, + ) -> _ProtectionRunRecord: + expected_ids = tuple(_DatumId(f"datum-{index}") for index in range(len(operation.records))) + outcome_by_id = {} + for graph_outcome in execution.outcomes: + datum_id = getattr(graph_outcome, "datum_id", None) + if not isinstance(datum_id, _DatumId) or datum_id in outcome_by_id: + return self._fail_all(operation) + outcome_by_id[datum_id] = graph_outcome + if set(outcome_by_id) != set(expected_ids): + return self._fail_all(operation) + if self._plan.profile == _SUBSTITUTE_PROFILE: + receipt: _ProtectionReceipt | _Phase7ProtectionReceipt = _Phase7ProtectionReceipt( + _CONTRACT_VERSION, + self._plan.profile, + _IMPLEMENTATION_VERSION, + True, + True, + ) + else: + receipt = _ProtectionReceipt( + _CONTRACT_VERSION, + self._plan.profile, + _IMPLEMENTATION_VERSION, + True, + True, + self._plan.digest, + secrets.token_hex(16), + ) + outcomes: list[_RecordOutcome] = [] + for record, datum_id in zip(operation.records, expected_ids, strict=True): + graph_outcome = outcome_by_id[datum_id] + if isinstance(graph_outcome, _GraphProtectionFailed): + code = ( + _FailureCode.INVOCATION_FAILED if graph_outcome.scope == "invocation" else _FailureCode.ROW_FAILED + ) + scope = "record" if graph_outcome.scope == "datum" else graph_outcome.scope + outcomes.append(_Failed(record.ref, _failure(code, graph_outcome.stage, scope))) + elif isinstance(graph_outcome, _GraphProtectionSucceeded): + disposition: _SuccessDisposition + disposition = _ProtectionApplied() if graph_outcome.applied else _NoAcceptedDetections() + outcomes.append(_ProtectionSucceeded(record.ref, graph_outcome.output, disposition, receipt)) + else: + return self._fail_all(operation) + return _ProtectionRunRecord(tuple(outcomes)) + + def _fail_all(self, operation: _OperationPlan) -> _ProtectionRunRecord: + failure = _failure(_FailureCode.INVOCATION_FAILED, "pipeline", "invocation") + return _ProtectionRunRecord(tuple(_Failed(record.ref, failure) for record in operation.records)) + + def _reject_all(self, operation: _OperationPlan, code: _FailureCode) -> _ProtectionRunRecord: + failure = _failure(code, "admission", "invocation") + return _ProtectionRunRecord(tuple(_Rejected(record.ref, failure) for record in operation.records)) + + def close(self) -> None: + """Reject new admission; borrowed Anonymizer resources are untouched.""" + with self._state_lock: + self._closed = True + + def __enter__(self) -> _ProtectionFlow: + return self + + def __exit__(self, *_args: object) -> None: + self.close() + + +def _failure(code: _FailureCode, stage: str, scope: str) -> _SafeFailure: + return _SafeFailure(code, stage, scope) diff --git a/src/anonymizer/interface/anonymizer.py b/src/anonymizer/interface/anonymizer.py index 1df2351f..381d99e5 100644 --- a/src/anonymizer/interface/anonymizer.py +++ b/src/anonymizer/interface/anonymizer.py @@ -49,7 +49,6 @@ COL_TYPE_FIDELITY_VALID, COL_UTILITY_SCORE, COL_WEIGHTED_LEAKAGE_RATE, - DEFAULT_ENTITY_LABELS, ) from anonymizer.engine.detection.detection_workflow import EntityDetectionWorkflow from anonymizer.engine.evaluation.detection_judge import DetectionJudgeWorkflow @@ -57,6 +56,8 @@ from anonymizer.engine.evaluation.replace.attribute_fidelity_judge import AttributeFidelityJudgeWorkflow from anonymizer.engine.evaluation.replace.relational_consistency_judge import RelationalConsistencyJudgeWorkflow from anonymizer.engine.evaluation.replace.type_fidelity_judge import TypeFidelityJudgeWorkflow +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.execution.pandas_runtime import _PandasRuntime from anonymizer.engine.io.reader import read_input from anonymizer.engine.ndd.adapter import FailedRecord, NddAdapter from anonymizer.engine.ndd.model_loader import ( @@ -65,13 +66,14 @@ validate_model_alias_references, validate_model_configs_reference_providers, ) +from anonymizer.engine.private_row_verification import PrivateRowVerificationError, _InvocationRowVerifier from anonymizer.engine.replace.llm_replace_workflow import LlmReplaceWorkflow from anonymizer.engine.replace.replace_runner import ReplacementWorkflow from anonymizer.engine.resolved_input import ResolvedInput from anonymizer.engine.rewrite.combined_rewrite_workflow import CombinedRewriteWorkflow from anonymizer.engine.rewrite.rewrite_workflow import RewriteWorkflow from anonymizer.engine.schemas import EntitiesByValueSchema -from anonymizer.interface.errors import InvalidConfigError +from anonymizer.interface.errors import AnonymizerWorkflowError, InvalidConfigError from anonymizer.interface.results import AnonymizerResult, PreviewResult from anonymizer.logging import LOG_INDENT, configure_logging, reapply_log_levels from anonymizer.measurement import ( @@ -98,6 +100,8 @@ logger = logging.getLogger("anonymizer") +_PUBLIC_PIPELINE_FAILURE_MESSAGE = "Anonymization pipeline failed." + def _has_entities_for_evaluation(raw: object) -> bool: """Return whether a row should receive entity-dependent evaluation scores.""" @@ -243,6 +247,13 @@ def run( No LLM evaluation judges run here — call :meth:`evaluate` on the result's ``trace_dataframe`` when you want the Judge Agreement scores. + The call publishes one complete :class:`AnonymizerResult` or raises; + exceptions never carry a partial result. Records dropped by an + underlying workflow are instead reported explicitly in a successful + result's ``failed_records``. This result-publication contract does not + make external providers, telemetry, source ingestion, or persistence + transactional. + Args: config: Workflow behavior — replace strategy, entity labels, thresholds. data: Input source with file path, text column, and optional data summary. @@ -253,12 +264,15 @@ def run( t_start = time.perf_counter() status = TaskStatusEnum.COMPLETED result: AnonymizerResult | None = None + public_error: AnonymizerWorkflowError | None = None try: result = self._run_internal(config=config, data=data, context=context, preview_num_records=None) - return result except KeyboardInterrupt: status = TaskStatusEnum.CANCELED raise + except PrivateRowVerificationError: + status = TaskStatusEnum.ERROR + public_error = AnonymizerWorkflowError(_PUBLIC_PIPELINE_FAILURE_MESSAGE) except Exception: status = TaskStatusEnum.ERROR raise @@ -272,6 +286,11 @@ def run( result=result, duration_sec=time.perf_counter() - t_start, ) + if public_error is not None: + raise public_error from None + if result is None: # pragma: no cover - defensive typing guard + raise RuntimeError("Anonymizer pipeline returned no result") + return result def export_detection_config( self, @@ -355,6 +374,13 @@ def preview( No LLM evaluation judges run here — call :meth:`evaluate` on the result's ``trace_dataframe`` when you want the Judge Agreement scores. + The call publishes one complete :class:`PreviewResult` for the selected + subset or raises; exceptions never carry a partial result. Records + dropped by an underlying workflow are instead reported explicitly in a + successful result's ``failed_records``. This result-publication contract + does not make external providers, telemetry, source ingestion, or + persistence transactional. + Args: config: Workflow behavior — replace strategy, entity labels, thresholds. data: Input source with file path, text column, and optional data summary. @@ -366,22 +392,15 @@ def preview( t_start = time.perf_counter() status = TaskStatusEnum.COMPLETED result: AnonymizerResult | None = None + public_error: AnonymizerWorkflowError | None = None try: result = self._run_internal(config=config, data=data, context=context, preview_num_records=num_records) - return PreviewResult( - dataframe=result.dataframe, - trace_dataframe=result.trace_dataframe, - resolved_text_column=result.resolved_text_column, - failed_records=result.failed_records, - preview_num_records=num_records, - replace_method=config.replace, - rewrite_config=config.rewrite.privacy_goal if config.rewrite is not None else None, - entity_labels=config.detect.entity_labels, - data_summary=result.data_summary, - ) except KeyboardInterrupt: status = TaskStatusEnum.CANCELED raise + except PrivateRowVerificationError: + status = TaskStatusEnum.ERROR + public_error = AnonymizerWorkflowError(_PUBLIC_PIPELINE_FAILURE_MESSAGE) except Exception: status = TaskStatusEnum.ERROR raise @@ -395,6 +414,21 @@ def preview( result=result, duration_sec=time.perf_counter() - t_start, ) + if public_error is not None: + raise public_error from None + if result is None: # pragma: no cover - defensive typing guard + raise RuntimeError("Anonymizer preview pipeline returned no result") + return PreviewResult( + dataframe=result.dataframe, + trace_dataframe=result.trace_dataframe, + resolved_text_column=result.resolved_text_column, + failed_records=result.failed_records, + preview_num_records=num_records, + replace_method=config.replace, + rewrite_config=config.rewrite.privacy_goal if config.rewrite is not None else None, + entity_labels=config.detect.entity_labels, + data_summary=result.data_summary, + ) def evaluate( self, @@ -606,8 +640,6 @@ def evaluate( if result.failed_records: logger.debug("%d evaluation failed record(s).", len(result.failed_records)) - for failure in result.failed_records: - logger.debug(" %s (%s: %s)", failure.record_id, failure.step, failure.reason) logger.info( "🎉 Evaluation complete — %d records processed [%.1fs]", num_records, @@ -619,6 +651,20 @@ def validate_config(self, config: AnonymizerConfig) -> None: """Validate that the active workflow config is compatible with model selections.""" self._validate_preflight_config(config) + def _compile_protection_plan(self, config: AnonymizerConfig): + """Compile the private Plan A profile without performing effects.""" + from anonymizer.interface._protection import _compile_protection_plan + + return _compile_protection_plan(config, self._selected_models, self._model_configs) + + def _open_protection_flow(self, plan): + """Open a private flow borrowing this facade's runtime resources.""" + from anonymizer.interface._protection import _ProtectionFlow, _ProtectionPlan + + if not isinstance(plan, _ProtectionPlan): + raise ValueError("private protection plan is not executable") + return _ProtectionFlow(self, plan) + def _run_internal( self, *, @@ -627,36 +673,51 @@ def _run_internal( context: ResolvedInput, preview_num_records: int | None, ) -> AnonymizerResult: + verifier = _InvocationRowVerifier(context.dataframe) + context = context.with_dataframe(verifier.bind(context.dataframe)) input_df = context.dataframe mode = "replace" if config.replace is not None else "rewrite" strategy = type(config.replace).__name__ if config.replace is not None else "Rewrite" - with stage_timer( - "Anonymizer._run_internal", - mode=mode, - strategy=strategy, - input_row_count=len(input_df), - preview_num_records=preview_num_records, - ) as measurement: - record_run_metadata( - config=config, - data=data, + result: AnonymizerResult | None = None + pipeline_error: PrivateRowVerificationError | None = None + try: + with stage_timer( + "Anonymizer._run_internal", mode=mode, strategy=strategy, input_row_count=len(input_df), preview_num_records=preview_num_records, - model_configs=self._model_configs, - ) - result = self._run_internal_impl( - config=config, - data=data, - context=context, - preview_num_records=preview_num_records, - ) - measurement.update( - output_row_count=len(result.trace_dataframe), - failed_record_count=len(result.failed_records), - ) - return result + ) as measurement: + record_run_metadata( + config=config, + data=data, + mode=mode, + strategy=strategy, + input_row_count=len(input_df), + preview_num_records=preview_num_records, + model_configs=self._model_configs, + ) + result = self._run_internal_impl( + config=config, + data=data, + context=context, + preview_num_records=preview_num_records, + verifier=verifier, + ) + measurement.update( + output_row_count=len(result.trace_dataframe), + failed_record_count=len(result.failed_records), + ) + except KeyboardInterrupt: + verifier.abort(cancelled=True) + raise + except Exception as exc: + pipeline_error = verifier.abort_with_failure(stage="pipeline", cause=exc) + if pipeline_error is not None: + raise pipeline_error from None + if result is None: # pragma: no cover - defensive typing guard + raise RuntimeError("Private anonymizer pipeline returned no result") + return result def _run_internal_impl( self, @@ -665,136 +726,23 @@ def _run_internal_impl( data: AnonymizerInput, context: ResolvedInput, preview_num_records: int | None, + verifier: _InvocationRowVerifier, ) -> AnonymizerResult: - input_df = context.dataframe - num_records = len(input_df) - if preview_num_records is not None and preview_num_records != num_records: - effective_records = min(preview_num_records, num_records) - if effective_records < preview_num_records: - logger.info( - LOG_INDENT + "🔍 Running entity detection on capped %d records (requested %d, available %d)", - effective_records, - preview_num_records, - num_records, - ) - else: - logger.info( - LOG_INDENT + "🔍 Running entity detection on %d of %d records", effective_records, num_records - ) - preview_num_records = effective_records - else: - logger.info("🔍 Running entity detection on %d records", num_records) - if logger.isEnabledFor(logging.DEBUG): - text_lengths = input_df[COL_TEXT].astype(str).str.len() - logger.debug( - "input text lengths: min=%d, max=%d, mean=%.0f chars (%d records)", - text_lengths.min(), - text_lengths.max(), - text_lengths.mean(), - num_records, - ) - logger.debug( - "detection config: threshold=%.2f, labels=%s", - config.detect.gliner_threshold, - config.detect.entity_labels - or f"(default: {len(DEFAULT_ENTITY_LABELS)} labels; see anonymizer.DEFAULT_ENTITY_LABELS for list)", - ) - else: - logger.info( - "detection labels in scope: %s", - config.detect.entity_labels - or f"(default: {len(DEFAULT_ENTITY_LABELS)} labels; see anonymizer.DEFAULT_ENTITY_LABELS for list)", - ) - - t0 = time.perf_counter() - detection_result = self._detection_workflow.run( - input_df, - model_configs=self._model_configs, - selected_models=self._selected_models.detection, - gliner_detection_threshold=config.detect.gliner_threshold, - validation_max_entities_per_call=config.detect.validation_max_entities_per_call, - validation_excerpt_window_chars=config.detect.validation_excerpt_window_chars, - entity_labels=config.detect.entity_labels, - privacy_goal=config.rewrite.privacy_goal if config.rewrite else None, + invocation = _CompiledInvocation.compile(config, self._selected_models, self._model_configs) + execution = _PandasRuntime( + detection_workflow=self._detection_workflow, + replace_runner=self._replace_runner, + rewrite_runner=self._rewrite_runner, + combined_rewrite_runner=self._combined_rewrite_runner, + ).run( + context.dataframe, + invocation=invocation, data_summary=data.data_summary, - tag_latent_entities=config.rewrite is not None, - compute_grouped_entities=config.replace is not None or config.rewrite is not None, preview_num_records=preview_num_records, + verifier=verifier, ) - detection_elapsed = time.perf_counter() - t0 - entity_count = _count_entities(detection_result.dataframe) - detection_failed = len(detection_result.failed_records) - logger.info( - LOG_INDENT + "📋 Detection complete — %d entities found across %d records (%d failed) [%.1fs]", - entity_count, - len(detection_result.dataframe), - detection_failed, - detection_elapsed, - ) - if COL_DETECTED_ENTITIES in detection_result.dataframe.columns: - label_counts = _count_labels(detection_result.dataframe[COL_DETECTED_ENTITIES]) - if label_counts: - summary = ", ".join(f"{label}={count}" for label, count in label_counts.most_common()) - logger.info(LOG_INDENT + "labels: %s", summary) - if config.replace is not None: - strategy_name = type(config.replace).__name__ - logger.info("🔄 Running %s replacement", strategy_name) - t0 = time.perf_counter() - replace_result = self._replace_runner.run( - detection_result.dataframe, - replace_method=config.replace, - model_configs=self._model_configs, - selected_models=self._selected_models.replace, - preview_num_records=preview_num_records, - ) - replace_elapsed = time.perf_counter() - t0 - final_df = replace_result.dataframe - post_detection_failures = replace_result.failed_records - logger.info( - LOG_INDENT + "📋 Replacement complete (%d failed) [%.1fs]", - len(post_detection_failures), - replace_elapsed, - ) - elif config.rewrite is not None: - logger.info("✏️ Running rewrite pipeline") - t0 = time.perf_counter() - privacy_goal = config.rewrite.privacy_goal - if privacy_goal is None: - raise InvalidConfigError("rewrite.privacy_goal must not be None") - rewrite_runner = ( - self._combined_rewrite_runner if config.rewrite.use_combined_graph else self._rewrite_runner - ) - rewrite_result = rewrite_runner.run( - detection_result.dataframe, - model_configs=self._model_configs, - selected_models=self._selected_models.rewrite, - replace_model_selection=self._selected_models.replace, - privacy_goal=privacy_goal, - evaluation=config.rewrite.evaluation, - data_summary=data.data_summary, - preview_num_records=preview_num_records, - strict_entity_protection=config.rewrite.strict_entity_protection, - ) - rewrite_elapsed = time.perf_counter() - t0 - final_df = rewrite_result.dataframe - post_detection_failures = rewrite_result.failed_records - logger.info( - LOG_INDENT + "📋 Rewrite complete (%d failed) [%.1fs]", - len(post_detection_failures), - rewrite_elapsed, - ) - else: - final_df = detection_result.dataframe - post_detection_failures = [] - - all_failures = [*detection_result.failed_records, *post_detection_failures] - if all_failures: - logger.warning("%d record(s) failed during pipeline processing.", len(all_failures)) - for f in all_failures: - logger.debug(" %s (%s: %s)", f.record_id, f.step, f.reason) - text_col = context.resolved_text_column - renamed_trace = _rename_output_columns(final_df, resolved_text_column=text_col) - logger.info("🎉 Pipeline complete — %d records processed, %d total failures", num_records, len(all_failures)) + final_df = execution.dataframe + renamed_trace = _rename_output_columns(final_df, resolved_text_column=context.resolved_text_column) record_record_metrics( final_df, mode="replace" if config.replace is not None else "rewrite", @@ -803,10 +751,10 @@ def _run_internal_impl( validation_max_entities_per_call=config.detect.validation_max_entities_per_call, ) return AnonymizerResult( - dataframe=_build_user_dataframe(renamed_trace, resolved_text_column=text_col), + dataframe=_build_user_dataframe(renamed_trace, resolved_text_column=context.resolved_text_column), trace_dataframe=renamed_trace, - resolved_text_column=text_col, - failed_records=all_failures, + resolved_text_column=context.resolved_text_column, + failed_records=execution.failed_records, replace_method=config.replace, rewrite_config=config.rewrite.privacy_goal if config.rewrite is not None else None, entity_labels=config.detect.entity_labels, diff --git a/src/anonymizer/interface/cli/main.py b/src/anonymizer/interface/cli/main.py index 54c1b79a..ad886e8c 100644 --- a/src/anonymizer/interface/cli/main.py +++ b/src/anonymizer/interface/cli/main.py @@ -32,7 +32,7 @@ from anonymizer.engine.io.constants import SUPPORTED_IO_FORMATS from anonymizer.interface.anonymizer import Anonymizer from anonymizer.interface.cli._output import write_result -from anonymizer.interface.errors import AnonymizerIOError, InvalidConfigError +from anonymizer.interface.errors import AnonymizerError, InvalidConfigError from anonymizer.logging import LoggingConfig, configure_logging app = cyclopts.App(help="NeMo Anonymizer CLI") @@ -158,7 +158,7 @@ def _cli_error_handler(fn): def wrapper(*args, **kwargs): try: return fn(*args, **kwargs) - except (ValidationError, ValueError, InvalidConfigError, AnonymizerIOError, OSError) as exc: + except (ValidationError, ValueError, AnonymizerError, OSError) as exc: print(f"Error: {exc}", file=sys.stderr) raise SystemExit(1) diff --git a/src/anonymizer/interface/errors.py b/src/anonymizer/interface/errors.py index 29a39672..1c2bc569 100644 --- a/src/anonymizer/interface/errors.py +++ b/src/anonymizer/interface/errors.py @@ -23,7 +23,11 @@ class AnonymizerIOError(AnonymizerError): class AnonymizerWorkflowError(AnonymizerError): """Raised when an underlying workflow step (preview, execution, or dataset load) fails. - The original backend exception is preserved as ``__cause__`` via ``raise ... from exc`` - so callers can inspect the exact failure without the anonymizer layer leaking backend - exception types into its own public error hierarchy. + Internal boundaries may preserve a backend exception as ``__cause__`` for + diagnostics. Public privacy-sensitive operations such as ``run()`` and + ``preview()`` deliberately suppress causes and use a generic message so + backend details, row correlations, and input values cannot escape. Those + operations do not attach a partial result to this exception. Explicit + per-record drops remain available only through ``failed_records`` on a + successfully returned result. """ diff --git a/src/anonymizer/interface/results.py b/src/anonymizer/interface/results.py index 7bfe13f0..972726f4 100644 --- a/src/anonymizer/interface/results.py +++ b/src/anonymizer/interface/results.py @@ -55,7 +55,10 @@ class AnonymizerResult(_DisplayMixin): user's requested ``text_column`` unless the reader had to rename it to avoid colliding with an Anonymizer output column, in which case it is the post-rename identifier (e.g. ``"final_entities__input"``). - failed_records: Records that failed during pipeline processing. + failed_records: Records dropped by an underlying workflow during an + otherwise successful call. Their presence is an explicit degraded + result, not transactional publication of a partially built result + after an exception. replace_method: The replace strategy that produced this result. Set by ``run()`` / ``preview()``; consumed by ``evaluate()`` to dispatch the right judges. ``None`` on results that were constructed by hand or @@ -101,7 +104,10 @@ class PreviewResult(_DisplayMixin): user's requested ``text_column`` unless the reader had to rename it to avoid colliding with an Anonymizer output column, in which case it is the post-rename identifier (e.g. ``"final_entities__input"``). - failed_records: Records that failed during pipeline processing. + failed_records: Records dropped by an underlying workflow during an + otherwise successful call. Their presence is an explicit degraded + result, not transactional publication of a partially built result + after an exception. preview_num_records: Number of records requested for the preview. replace_method: The replace strategy that produced this preview. Set by ``preview()``; consumed by ``evaluate()`` to dispatch the right diff --git a/src/anonymizer/measurement/session.py b/src/anonymizer/measurement/session.py index 1d985d4d..7b8c8b6c 100644 --- a/src/anonymizer/measurement/session.py +++ b/src/anonymizer/measurement/session.py @@ -87,6 +87,16 @@ def current_collector() -> MeasurementCollector | None: return _ACTIVE_COLLECTOR.get() +@contextmanager +def suppress_measurement() -> Iterator[None]: + """Temporarily detach any ambient collector in the current context.""" + token = _ACTIVE_COLLECTOR.set(None) + try: + yield + finally: + _ACTIVE_COLLECTOR.reset(token) + + def _write_collector_safely( *, config: MeasurementConfig, diff --git a/tests/engine/execution/phase4_conformance_manifest.json b/tests/engine/execution/phase4_conformance_manifest.json new file mode 100644 index 00000000..b5d3ff47 --- /dev/null +++ b/tests/engine/execution/phase4_conformance_manifest.json @@ -0,0 +1,6 @@ +{ + "canonical_trace_count": 397542, + "generator_version": "phase4-stream-v4", + "graph_count": 8278, + "sha256": "e778147bf77909ddb94117fe7e6c230de57e46a722fad49c563b36f0b5660efa" +} diff --git a/tests/engine/execution/phase4_reference_model.py b/tests/engine/execution/phase4_reference_model.py new file mode 100644 index 00000000..0c3177cc --- /dev/null +++ b/tests/engine/execution/phase4_reference_model.py @@ -0,0 +1,715 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Independent pure oracle for phase-4 dependency and atomic release semantics.""" + +from __future__ import annotations + +from collections.abc import Iterator +from dataclasses import dataclass +from enum import Enum +from itertools import product +from typing import Literal, TypeAlias, assert_never, final + + +class ReferenceTaskOutcome(str, Enum): + SUCCEEDED = "succeeded" + FAILED = "failed" + CANCELLED = "cancelled" + LOST = "lost" + BLOCKED = "blocked" + INCONSISTENT = "inconsistent" + + +class ReferenceInvocationOutcome(str, Enum): + COMPLETED = "completed" + FAILED = "failed" + CANCELLED = "cancelled" + LOST = "lost" + INCONSISTENT = "inconsistent" + + +class ReferenceCorruption(str, Enum): + """Closed reconciliation evidence algebra, kept independent of ledger types.""" + + MISSING = "missing" + DUPLICATE = "duplicate" + UNKNOWN = "unknown" + FOREIGN = "foreign" + STALE = "stale" + SWAPPED = "swapped" + PLAN_MISMATCH = "plan_mismatch" + CONTRADICTORY = "contradictory" + + +@dataclass(frozen=True) +class ReferenceDeclaration: + datum_ids: tuple[str, ...] + dependencies: tuple[tuple[str, str], ...] + atomic_groups: tuple[tuple[str, ...], ...] + stages: tuple[str, ...] = ("protect",) + rejected_datums: frozenset[str] = frozenset() + rejected_groups: frozenset[frozenset[str]] = frozenset() + + +@dataclass(frozen=True) +class ReferenceResult: + release_eligible: frozenset[str] + released_groups: frozenset[frozenset[str]] + + +ReferenceTaskKey: TypeAlias = tuple[str, str] + + +@final +@dataclass(frozen=True) +class ReferenceDispatch: + task: ReferenceTaskKey + + +@final +@dataclass(frozen=True) +class ReferenceSuccess: + task: ReferenceTaskKey + + +@final +@dataclass(frozen=True) +class ReferenceFailure: + task: ReferenceTaskKey + + +@final +@dataclass(frozen=True) +class ReferenceCancellationRequest: + pass + + +@final +@dataclass(frozen=True) +class ReferenceStopAcknowledgement: + task: ReferenceTaskKey + + +@final +@dataclass(frozen=True) +class ReferenceTransportLoss: + task: ReferenceTaskKey + + +@final +@dataclass(frozen=True) +class ReferenceContradiction: + pass + + +@final +@dataclass(frozen=True) +class ReferenceResultConstructionFailure: + pass + + +@final +@dataclass(frozen=True) +class ReferenceCorruptEvidence: + kind: ReferenceCorruption + + +ReferenceObservation: TypeAlias = ( + ReferenceDispatch + | ReferenceSuccess + | ReferenceFailure + | ReferenceCancellationRequest + | ReferenceStopAcknowledgement + | ReferenceTransportLoss + | ReferenceContradiction + | ReferenceResultConstructionFailure + | ReferenceCorruptEvidence +) + + +@dataclass(frozen=True) +class ReferenceHierarchyResult: + tasks: tuple[tuple[ReferenceTaskKey, ReferenceTaskOutcome], ...] + task_causes: tuple[tuple[ReferenceTaskKey, tuple[str, ...]], ...] + datums: tuple[tuple[str, ReferenceTaskOutcome], ...] + dependencies: tuple[tuple[tuple[str, str], bool], ...] + stages: tuple[tuple[str, ReferenceTaskOutcome], ...] + released_groups: frozenset[frozenset[str]] + released_group_order: tuple[tuple[str, ...], ...] + invocation: ReferenceInvocationOutcome + + +@dataclass(frozen=True) +class ReferenceCorpusCase: + """One canonical schedule from the bounded phase-4 conformance envelope.""" + + declaration: ReferenceDeclaration + observations: tuple[ReferenceObservation, ...] + graph_witness: bool = True + + +def flat_partitions(values: tuple[str, ...]) -> tuple[tuple[tuple[str, ...], ...], ...]: + """Enumerate set partitions without using production group code.""" + if not values: + return ((),) + head, *tail = values + tail_partitions = flat_partitions(tuple(tail)) + expanded = [] + for partition in tail_partitions: + expanded.append(((head,), *partition)) + expanded.extend( + tuple( + (*partition[:index], (head, *group), *partition[index + 1 :]) for index, group in enumerate(partition) + ) + ) + return tuple(expanded) + + +def acyclic_dependencies(values: tuple[str, ...]) -> tuple[tuple[tuple[str, str], ...], ...]: + """Enumerate labeled DAG edge sets independently of admission.""" + candidates = tuple((left, right) for left in values for right in values if left != right) + return tuple( + edges + for selected in product((False, True), repeat=len(candidates)) + if _is_acyclic(values, edges := tuple(edge for edge, included in zip(candidates, selected) if included)) + ) + + +def streaming_conformance_cases() -> Iterator[ReferenceCorpusCase]: + """Yield, without materializing, canonical schedules for the finite design envelope. + + Independent ready tasks use a stable, independently derived topological order. + Every admitted graph and stage topology is crossed with each primary terminal + and reconciliation-fault class. A separate witness axis places those events at + every logical task position across one through three stages. + """ + yield ReferenceCorpusCase(ReferenceDeclaration((), (), ()), ()) + for datum_count in range(1, 5): + datum_ids = tuple(chr(ord("a") + index) for index in range(datum_count)) + for dependencies in acyclic_dependencies(datum_ids): + for groups in flat_partitions(datum_ids): + for stage_count in range(1, 4): + declaration = ReferenceDeclaration( + datum_ids, + dependencies, + groups, + tuple(f"stage-{index}" for index in range(stage_count)), + ) + successful = _successful_observations(declaration) + yield ReferenceCorpusCase(declaration, successful) + yield from _topology_event_witnesses(declaration, successful) + yield from _semantic_transition_witnesses() + + +def _semantic_transition_witnesses() -> Iterator[ReferenceCorpusCase]: + """Cover non-commuting terminal and reconciliation transitions separately. + + The graph axis above exhausts all 0--4 datum DAG/partition fixed points and + crosses each topology with every event class at a canonical dispatch. This + axis uses a non-singleton dependency plus atomic peer declaration to place the + same transitions at every logical task position and stage cardinality. + """ + for stage_count in range(1, 4): + declaration = ReferenceDeclaration( + ("a", "b", "c"), + (("a", "b"),), + (("a", "c"), ("b",)), + tuple(f"stage-{index}" for index in range(stage_count)), + ) + success = _successful_observations(declaration) + for observations in ( + success, + (*success, ReferenceCancellationRequest()), + (*success, ReferenceContradiction()), + (*success, ReferenceResultConstructionFailure()), + ): + yield ReferenceCorpusCase(declaration, observations, graph_witness=False) + tasks = tuple((stage, datum_id) for stage in declaration.stages for datum_id in declaration.datum_ids) + for task in tasks: + dispatch_index = success.index(ReferenceDispatch(task)) + prefix = success[: dispatch_index + 1] + schedules: tuple[tuple[ReferenceObservation, ...], ...] = ( + (*prefix, ReferenceCancellationRequest(), ReferenceStopAcknowledgement(task)), + (*prefix, ReferenceTransportLoss(task)), + (*prefix, ReferenceFailure(task)), + (*prefix, ReferenceCancellationRequest(), ReferenceSuccess(task)), + (*prefix, ReferenceSuccess(task), ReferenceStopAcknowledgement(task)), + *(tuple((*prefix, ReferenceCorruptEvidence(kind))) for kind in ReferenceCorruption), + ) + for observations in schedules: + yield ReferenceCorpusCase(declaration, observations, graph_witness=False) + + +def _topology_event_witnesses( + declaration: ReferenceDeclaration, + success: tuple[ReferenceObservation, ...], +) -> Iterator[ReferenceCorpusCase]: + """Cross every admitted topology with every primary terminal/fault class.""" + first_dispatch = next(event for event in success if isinstance(event, ReferenceDispatch)) + dispatch_index = success.index(first_dispatch) + prefix = success[: dispatch_index + 1] + task = first_dispatch.task + schedules: list[tuple[ReferenceObservation, ...]] = [ + (*prefix, ReferenceFailure(task)), + (*prefix, ReferenceTransportLoss(task)), + (*prefix, ReferenceCancellationRequest()), + (*prefix, ReferenceCancellationRequest(), ReferenceStopAcknowledgement(task)), + (*prefix, ReferenceCancellationRequest(), ReferenceSuccess(task)), + (*prefix, ReferenceSuccess(task), ReferenceStopAcknowledgement(task)), + (*success, ReferenceResultConstructionFailure()), + ] + schedules.extend( + (*prefix, ReferenceCorruptEvidence(kind)) + for kind in ReferenceCorruption + if kind is not ReferenceCorruption.SWAPPED or len(declaration.datum_ids) * len(declaration.stages) > 1 + ) + yield from (ReferenceCorpusCase(declaration, observations, graph_witness=False) for observations in schedules) + + +def _successful_observations(declaration: ReferenceDeclaration) -> tuple[ReferenceObservation, ...]: + """Use the unique declaration-order representative of each commute class.""" + ordered_datums = _topological_datums(declaration) + return tuple( + event + for datum_id in ordered_datums + for stage in declaration.stages + for event in (ReferenceDispatch((stage, datum_id)), ReferenceSuccess((stage, datum_id))) + ) + + +def _topological_datums(declaration: ReferenceDeclaration) -> tuple[str, ...]: + """Derive a stable topological order independently from production admission.""" + predecessors = {dependent: set() for dependent in declaration.datum_ids} + for prerequisite, dependent in declaration.dependencies: + predecessors[dependent].add(prerequisite) + ordered_datums: list[str] = [] + remaining = set(declaration.datum_ids) + while remaining: + ready = next( + datum_id for datum_id in declaration.datum_ids if datum_id in remaining and not predecessors[datum_id] + ) + remaining.remove(ready) + ordered_datums.append(ready) + for blocked_by in predecessors.values(): + blocked_by.discard(ready) + return tuple(ordered_datums) + + +def _is_acyclic(values: tuple[str, ...], edges: tuple[tuple[str, str], ...]) -> bool: + remaining = set(values) + while remaining: + roots = {value for value in remaining if not any(right == value and left in remaining for left, right in edges)} + if not roots: + return False + remaining -= roots + return True + + +def reduce_reference( + declaration: ReferenceDeclaration, + task_outcomes: dict[str, ReferenceTaskOutcome], +) -> ReferenceResult: + """Derive the least fixed point without using production reducers.""" + eligible = { + datum_id for datum_id in declaration.datum_ids if task_outcomes[datum_id] is ReferenceTaskOutcome.SUCCEEDED + } + changed = True + while changed: + changed = False + for group in declaration.atomic_groups: + if not set(group).issubset(eligible): + previous = len(eligible) + eligible.difference_update(group) + changed = changed or len(eligible) != previous + for prerequisite, dependent in declaration.dependencies: + if prerequisite not in eligible and dependent in eligible: + eligible.remove(dependent) + changed = True + released = frozenset(frozenset(group) for group in declaration.atomic_groups if set(group).issubset(eligible)) + return ReferenceResult(frozenset(eligible), released) + + +def reduce_observations( + declaration: ReferenceDeclaration, + observations: tuple[ReferenceObservation, ...], +) -> ReferenceHierarchyResult: + """Derive hierarchy outcomes from exogenous observations only.""" + task_order = tuple( + (stage, datum_id) for stage in declaration.stages for datum_id in _topological_datums(declaration) + ) + states: dict[ReferenceTaskKey, str | ReferenceTaskOutcome] = dict.fromkeys(task_order, "planned") + causes: dict[ReferenceTaskKey, tuple[str, ...]] = dict.fromkeys(task_order, ()) + cancellation_requested = False + invocation_lost = False + global_inconsistent = False + result_construction_failed = False + + def mark_global_inconsistent(cause: str = "contradictory") -> None: + nonlocal global_inconsistent, states + global_inconsistent = True + updated: dict[ReferenceTaskKey, str | ReferenceTaskOutcome] = { + task: state if isinstance(state, ReferenceTaskOutcome) else ReferenceTaskOutcome.INCONSISTENT + for task, state in states.items() + } + causes.update({task: (cause,) for task, state in updated.items() if state is ReferenceTaskOutcome.INCONSISTENT}) + states = updated + + def datum_state(datum_id: str) -> str: + children = tuple(states[(stage, datum_id)] for stage in declaration.stages) + if all(child is ReferenceTaskOutcome.SUCCEEDED for child in children): + return "unsatisfied" if datum_id in declaration.rejected_datums else "satisfied" + if any( + isinstance(child, ReferenceTaskOutcome) and child is not ReferenceTaskOutcome.SUCCEEDED + for child in children + ): + return "unsatisfied" + return "waiting" + + def readiness(task: ReferenceTaskKey) -> str: + stage, datum_id = task + stage_index = declaration.stages.index(stage) + if stage_index: + previous = states[(declaration.stages[stage_index - 1], datum_id)] + if previous is not ReferenceTaskOutcome.SUCCEEDED: + return "blocked" if isinstance(previous, ReferenceTaskOutcome) else "waiting" + prerequisites = tuple( + prerequisite for prerequisite, dependent in declaration.dependencies if dependent == datum_id + ) + prerequisite_states = tuple(map(datum_state, prerequisites)) + if "unsatisfied" in prerequisite_states: + return "blocked" + return "ready" if all(state == "satisfied" for state in prerequisite_states) else "waiting" + + def advance() -> None: + changed = True + while changed: + changed = False + for task in task_order: + if states[task] != "planned": + continue + guard = readiness(task) + if guard == "ready": + states[task] = "ready" + changed = True + elif guard == "blocked": + states[task] = ReferenceTaskOutcome.BLOCKED + changed = True + + for observation in observations: + advance() + match observation: + case ReferenceDispatch(task=task): + if states.get(task) == "ready": + states[task] = "dispatched" + else: + mark_global_inconsistent() + case ReferenceSuccess(task=task): + if states.get(task) == "dispatched": + states[task] = ReferenceTaskOutcome.SUCCEEDED + elif not isinstance(states.get(task), ReferenceTaskOutcome): + mark_global_inconsistent() + case ReferenceFailure(task=task): + if states.get(task) == "dispatched": + states[task] = ReferenceTaskOutcome.FAILED + causes[task] = ("known_failure",) + elif not isinstance(states.get(task), ReferenceTaskOutcome): + mark_global_inconsistent() + case ReferenceCancellationRequest(): + cancellation_requested = True + states = { + task: ReferenceTaskOutcome.CANCELLED if state in {"planned", "ready"} else state + for task, state in states.items() + } + causes.update( + { + task: ("cancellation",) + for task, state in states.items() + if state is ReferenceTaskOutcome.CANCELLED + } + ) + case ReferenceStopAcknowledgement(task=task): + if states.get(task) == "dispatched": + states[task] = ReferenceTaskOutcome.CANCELLED + causes[task] = ("cancellation", "stop_acknowledged") + case ReferenceTransportLoss(task=task): + if states.get(task) == "dispatched": + states[task] = ReferenceTaskOutcome.LOST + causes[task] = ("transport_lost",) + invocation_lost = True + case ReferenceContradiction(): + mark_global_inconsistent() + case ReferenceResultConstructionFailure(): + result_construction_failed = True + case ReferenceCorruptEvidence(kind=ReferenceCorruption.MISSING): + for task, state in states.items(): + if state == "dispatched": + states[task] = ReferenceTaskOutcome.INCONSISTENT + causes[task] = ("missing",) + case ReferenceCorruptEvidence(kind=kind): + mark_global_inconsistent(kind.value) + case unreachable: + assert_never(unreachable) + + advance() + invocation_lost = invocation_lost or any(state == "dispatched" for state in states.values()) + states = { + task: ( + ReferenceTaskOutcome.LOST + if state == "dispatched" + else ReferenceTaskOutcome.BLOCKED + if state in {"planned", "ready"} + else state + ) + for task, state in states.items() + } + causes.update( + { + task: ("cancellation", "transport_lost") if cancellation_requested else ("transport_lost",) + for task, state in states.items() + if state is ReferenceTaskOutcome.LOST and not causes[task] + } + ) + causes.update( + { + task: ("prerequisite",) + for task, state in states.items() + if state is ReferenceTaskOutcome.BLOCKED and not causes[task] + } + ) + terminal_tasks = tuple((task, _terminal(state)) for task, state in states.items()) + terminal_causes = tuple((task, causes[task]) for task, _state in terminal_tasks) + task_by_key = dict(terminal_tasks) + datums = tuple( + ( + datum_id, + _reduce_children( + tuple(task_by_key[(stage, datum_id)] for stage in declaration.stages), + rejected=datum_id in declaration.rejected_datums, + ), + ) + for datum_id in declaration.datum_ids + ) + datum_by_id = dict(datums) + dependencies = tuple( + ((prerequisite, dependent), datum_by_id[prerequisite] is ReferenceTaskOutcome.SUCCEEDED) + for prerequisite, dependent in declaration.dependencies + ) + stages = tuple( + ( + stage, + _reduce_children(tuple(task_by_key[(stage, datum_id)] for datum_id in declaration.datum_ids)), + ) + for stage in declaration.stages + ) + release = reduce_reference( + declaration, + {datum_id: outcome for datum_id, outcome in datums}, + ) + released = release.released_groups - declaration.rejected_groups + if global_inconsistent or invocation_lost or cancellation_requested or result_construction_failed: + released = frozenset() + invocation = ( + ReferenceInvocationOutcome.FAILED + if result_construction_failed + else ReferenceInvocationOutcome.INCONSISTENT + if global_inconsistent + else ReferenceInvocationOutcome.LOST + if invocation_lost + else ReferenceInvocationOutcome.CANCELLED + if cancellation_requested + else ReferenceInvocationOutcome.COMPLETED + ) + released_order = tuple( + sorted( + (group for group in declaration.atomic_groups if frozenset(group) in released), + key=lambda group: tuple(sorted(group)), + ) + ) + return ReferenceHierarchyResult( + terminal_tasks, + terminal_causes, + datums, + dependencies, + stages, + released, + released_order, + invocation, + ) + + +def _terminal(state: str | ReferenceTaskOutcome) -> ReferenceTaskOutcome: + if isinstance(state, ReferenceTaskOutcome): + return state + raise AssertionError("reference model left a nonterminal task") + + +def _reduce_children( + children: tuple[ReferenceTaskOutcome, ...], + *, + rejected: bool = False, +) -> ReferenceTaskOutcome: + if all(child is ReferenceTaskOutcome.SUCCEEDED for child in children): + return ReferenceTaskOutcome.FAILED if rejected else ReferenceTaskOutcome.SUCCEEDED + return next( + outcome + for outcome in ( + ReferenceTaskOutcome.INCONSISTENT, + ReferenceTaskOutcome.LOST, + ReferenceTaskOutcome.CANCELLED, + ReferenceTaskOutcome.FAILED, + ReferenceTaskOutcome.BLOCKED, + ) + if outcome in children + ) + + +ReferenceMixedSubject: TypeAlias = tuple[Literal["datum", "scope"], str] +ReferenceMixedTaskKey: TypeAlias = tuple[str, ReferenceMixedSubject] + + +@dataclass(frozen=True) +class ReferenceMixedDeclaration: + """Independent declaration for non-rectangular datum/scope task plans.""" + + datum_ids: tuple[str, ...] + datum_stages: tuple[str, ...] + scope_tasks: tuple[tuple[str, str], ...] + datum_dependencies: tuple[tuple[str, str], ...] = () + task_predecessors: tuple[tuple[ReferenceMixedTaskKey, ReferenceMixedTaskKey], ...] = () + atomic_groups: tuple[tuple[str, ...], ...] = () + + +@dataclass(frozen=True) +class ReferenceMixedResult: + """Observable scheduling and reduction result for a mixed task plan.""" + + ready_frontiers: tuple[tuple[ReferenceMixedTaskKey, ...], ...] + tasks: tuple[tuple[ReferenceMixedTaskKey, ReferenceTaskOutcome], ...] + datums: tuple[tuple[str, ReferenceTaskOutcome], ...] + stages: tuple[tuple[str, ReferenceTaskOutcome], ...] + released_groups: frozenset[frozenset[str]] + + +def reduce_mixed_schedule( + declaration: ReferenceMixedDeclaration, + schedule: tuple[tuple[ReferenceMixedTaskKey, ReferenceTaskOutcome], ...], +) -> ReferenceMixedResult: + """Evaluate a mixed schedule without importing production planning or ledger code.""" + ordered_datums = _topological_mixed_datums(declaration) + task_order = ( + *((stage, ("datum", datum_id)) for stage in declaration.datum_stages for datum_id in ordered_datums), + *((stage, ("scope", scope_id)) for stage, scope_id in declaration.scope_tasks), + ) + states: dict[ReferenceMixedTaskKey, str | ReferenceTaskOutcome] = dict.fromkeys(task_order, "planned") + + def datum_state(datum_id: str) -> str: + children = tuple(states[(stage, ("datum", datum_id))] for stage in declaration.datum_stages) + if all(child is ReferenceTaskOutcome.SUCCEEDED for child in children): + return "satisfied" + if any(isinstance(child, ReferenceTaskOutcome) for child in children): + return "unsatisfied" + return "waiting" + + def readiness(task: ReferenceMixedTaskKey) -> str: + stage, (subject_kind, subject_id) = task + if subject_kind == "datum": + stage_index = declaration.datum_stages.index(stage) + if stage_index: + previous = states[(declaration.datum_stages[stage_index - 1], (subject_kind, subject_id))] + if previous is not ReferenceTaskOutcome.SUCCEEDED: + return "blocked" if isinstance(previous, ReferenceTaskOutcome) else "waiting" + explicit_states = tuple( + states[prerequisite] for prerequisite, dependent in declaration.task_predecessors if dependent == task + ) + if any( + isinstance(state, ReferenceTaskOutcome) and state is not ReferenceTaskOutcome.SUCCEEDED + for state in explicit_states + ): + return "blocked" + if any(state is not ReferenceTaskOutcome.SUCCEEDED for state in explicit_states): + return "waiting" + if subject_kind == "scope": + return "ready" + prerequisites = tuple( + prerequisite for prerequisite, dependent in declaration.datum_dependencies if dependent == subject_id + ) + prerequisite_states = tuple(datum_state(datum_id) for datum_id in prerequisites) + if "unsatisfied" in prerequisite_states: + return "blocked" + return "ready" if all(state == "satisfied" for state in prerequisite_states) else "waiting" + + def advance() -> None: + changed = True + while changed: + changed = False + for task in task_order: + if states[task] != "planned": + continue + guard = readiness(task) + if guard == "ready": + states[task] = "ready" + changed = True + elif guard == "blocked": + states[task] = ReferenceTaskOutcome.BLOCKED + changed = True + + ready_frontiers: list[tuple[ReferenceMixedTaskKey, ...]] = [] + for task, outcome in schedule: + advance() + ready = tuple(candidate for candidate in task_order if states[candidate] == "ready") + ready_frontiers.append(ready) + if task not in ready or outcome not in { + ReferenceTaskOutcome.SUCCEEDED, + ReferenceTaskOutcome.FAILED, + ReferenceTaskOutcome.CANCELLED, + ReferenceTaskOutcome.LOST, + ReferenceTaskOutcome.INCONSISTENT, + }: + raise AssertionError("reference mixed schedule is not executable") + states[task] = outcome + + advance() + states = { + task: ReferenceTaskOutcome.BLOCKED if state in {"planned", "ready"} else state for task, state in states.items() + } + terminal_tasks = tuple((task, _terminal(state)) for task, state in states.items()) + outcome_by_task = dict(terminal_tasks) + datums = tuple( + ( + datum_id, + _reduce_children( + tuple(outcome_by_task[(stage, ("datum", datum_id))] for stage in declaration.datum_stages) + ), + ) + for datum_id in declaration.datum_ids + ) + stages = tuple( + ( + stage, + _reduce_children( + tuple(outcome_by_task[(stage, ("datum", datum_id))] for datum_id in declaration.datum_ids) + ), + ) + for stage in declaration.datum_stages + ) + released = reduce_reference( + ReferenceDeclaration( + declaration.datum_ids, + declaration.datum_dependencies, + declaration.atomic_groups, + declaration.datum_stages, + ), + dict(datums), + ).released_groups + return ReferenceMixedResult(tuple(ready_frontiers), terminal_tasks, datums, stages, released) + + +def _topological_mixed_datums(declaration: ReferenceMixedDeclaration) -> tuple[str, ...]: + legacy_shape = ReferenceDeclaration( + declaration.datum_ids, + declaration.datum_dependencies, + declaration.atomic_groups, + declaration.datum_stages, + ) + return _topological_datums(legacy_shape) diff --git a/tests/engine/execution/phase5_reference_manifest.json b/tests/engine/execution/phase5_reference_manifest.json new file mode 100644 index 00000000..39d41d8f --- /dev/null +++ b/tests/engine/execution/phase5_reference_manifest.json @@ -0,0 +1,106 @@ +{ + "actual_event_count": 35138, + "canonical_trace_count": 2198, + "ceiling_domain": ["zero", "exact", "exact_plus_one"], + "ceiling_fields": ["datum_bytes", "id_bytes", "members", "context_bytes", "references", "expanded_bytes"], + "counts": { + "cardinality": { + "1t-0c": 51, + "1t-1c": 61, + "1t-2c": 122, + "1t-3c": 122, + "2t-0c": 111, + "2t-1c": 137, + "2t-2c": 187, + "2t-3c": 187, + "3t-0c": 169, + "3t-1c": 187, + "3t-2c": 249, + "3t-3c": 249, + "4t-0c": 53, + "4t-1c": 63, + "4t-2c": 125, + "4t-3c": 125 + }, + "context_cycle": {"2-target-0": 244, "3-target-0": 244, "3-target-1": 244, "none": 1466}, + "context_order": {"declared": 1692, "reversed": 494, "scope_declaration_reversed": 12}, + "limit_class": { + "all:exact": 1550, + "context_bytes:exact": 36, + "context_bytes:exact_plus_one": 36, + "context_bytes:zero": 36, + "datum_bytes:exact": 36, + "datum_bytes:exact_plus_one": 36, + "datum_bytes:zero": 36, + "expanded_bytes:exact": 36, + "expanded_bytes:exact_plus_one": 36, + "expanded_bytes:zero": 36, + "id_bytes:exact": 36, + "id_bytes:exact_plus_one": 36, + "id_bytes:zero": 36, + "members:exact": 36, + "members:exact_plus_one": 36, + "members:zero": 36, + "references:exact": 36, + "references:exact_plus_one": 36, + "references:zero": 36 + }, + "payload_class": {"empty": 29, "exact_limit": 29, "multibyte": 2082, "one_byte": 29, "one_over_limit": 29}, + "schedule_class": { + "contradictory:verified:success:accepted": 32, + "cross_target:verified:success:accepted": 32, + "duplicate:verified:success:accepted": 32, + "exact:contradictory:success:accepted": 36, + "exact:duplicate:success:accepted": 36, + "exact:failed:success:accepted": 36, + "exact:foreign:success:accepted": 36, + "exact:incompatible:success:accepted": 36, + "exact:missing:success:accepted": 36, + "exact:verified:cancel_pre_dispatch:accepted": 36, + "exact:verified:success:accepted": 1358, + "exact:verified:success:failed": 36, + "exact:verified:terminal_contradictory:accepted": 36, + "exact:verified:terminal_cross_target:accepted": 36, + "exact:verified:terminal_duplicate:accepted": 36, + "exact:verified:terminal_foreign:accepted": 36, + "exact:verified:terminal_missing:accepted": 36, + "exact:verified:terminal_plan_mismatch:accepted": 36, + "exact:verified:terminal_stale:accepted": 36, + "exact:verified:terminal_then_cancel:accepted": 36, + "exact:verified:transport_loss:accepted": 36, + "exact:verified:trusted_stop:accepted": 36, + "foreign:verified:success:accepted": 32, + "missing:verified:success:accepted": 32, + "wrong_ordinal:verified:success:accepted": 32 + }, + "t_b_emax": { + "T1-B0-E10": 51, + "T1-B1-E14": 61, + "T1-B2-E18": 122, + "T1-B3-E22": 122, + "T2-B0-E13": 54, + "T2-B1-E17": 70, + "T2-B2-E21": 187, + "T2-B3-E25": 187, + "T2-B4-E29": 62, + "T2-B5-E33": 62, + "T3-B0-E16": 53, + "T3-B1-E20": 63, + "T3-B2-E24": 125, + "T3-B3-E28": 241, + "T3-B4-E32": 124, + "T3-B5-E36": 124, + "T3-B6-E40": 124, + "T4-B0-E19": 53, + "T4-B1-E23": 63, + "T4-B2-E27": 125, + "T4-B3-E31": 125 + } + }, + "event_bound": "4B+3T+7", + "generator_version": "phase5-reference-v2", + "graph_count": 173, + "model_version": "target-context-event-workframe-v2", + "payload_domain": ["empty", "one_byte", "multibyte", "exact_limit", "one_over_limit"], + "sha256": "943405c8cdee3e714e8838a9263b9965de7b3574d85bc37172f85760b7128296" +} diff --git a/tests/engine/execution/phase5_reference_model.py b/tests/engine/execution/phase5_reference_model.py new file mode 100644 index 00000000..3d7ca8cc --- /dev/null +++ b/tests/engine/execution/phase5_reference_model.py @@ -0,0 +1,827 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Independent bounded oracle for Phase 5 context framing and release. + +This module deliberately imports no Anonymizer production code, pandas, +DataDesigner, or Phase 4 ledger implementation. +""" + +from __future__ import annotations + +import json +from collections.abc import Iterator +from dataclasses import asdict, dataclass, replace +from enum import Enum +from hashlib import sha256 + + +class ReferenceAdmission(str, Enum): + ADMITTED = "admitted" + STRUCTURAL = "structural" + LIMIT = "limit" + CONTRACT = "contract" + PREFLIGHT_CAPABILITY = "preflight_capability" + + +class ReferenceInvocation(str, Enum): + NOT_OPENED = "not_opened" + COMPLETED = "completed" + FAILED = "failed" + CANCELLED = "cancelled" + LOST = "lost" + INCONSISTENT = "inconsistent" + + +class ReferenceEventKind(str, Enum): + BINDING_CONSTRUCTION = "binding_construction" + BINDING_COMMITMENT = "binding_commitment" + BINDING_CONSUMPTION = "binding_consumption" + BINDING_CORRUPTION = "binding_corruption" + TASK_DISPATCH = "task_dispatch" + TASK_TERMINAL = "task_terminal" + TASK_CORRUPTION = "task_corruption" + CANCELLATION = "cancellation" + TRUSTED_STOP = "trusted_stop" + TRANSPORT_LOSS = "transport_loss" + CLEANUP_PRIMARY = "cleanup_primary" + CLEANUP_COMPETING = "cleanup_competing" + PUBLICATION = "publication" + TEARDOWN = "teardown" + + +@dataclass(frozen=True) +class ReferenceEvent: + kind: ReferenceEventKind + subject: str = "invocation" + outcome: str = "accepted" + + +@dataclass(frozen=True) +class ReferenceLimits: + datum_bytes: int + id_bytes: int + members: int + context_bytes: int + references: int + expanded_bytes: int + + +@dataclass(frozen=True) +class ReferenceScope: + target: str + context: tuple[str, ...] + + +@dataclass(frozen=True) +class ReferenceCase: + case_id: str + targets: tuple[tuple[str, str], ...] + context_only: tuple[tuple[str, str], ...] + scopes: tuple[ReferenceScope, ...] + limits: ReferenceLimits + events: tuple[ReferenceEvent, ...] + preflight_capability: str = "compatible" + runtime_capability: str = "compatible" + relation: str = "bounded_context" + profile: str = "target-context-v1" + schema: str = "context-workframe-v1" + ordering: str = "declared" + allow_target_as_context: bool = True + dependencies: tuple[tuple[str, str], ...] = () + groups: tuple[tuple[str, ...], ...] = () + order_class: str = "declared" + cycle_class: str = "none" + payload_class: str = "multibyte" + limit_class: str = "all:exact" + schedule_class: str = "success" + + +@dataclass(frozen=True) +class ReferenceResult: + admission: ReferenceAdmission + invocation: ReferenceInvocation + reason: str + binding_count: int + task_outcomes: tuple[tuple[str, str, str], ...] + private_succeeded: tuple[str, ...] + private_inconsistent: tuple[str, ...] + released: tuple[str, ...] + cleanup: str + published: bool + teardown: str + event_count: int + event_max: int + + +def evaluate(case: ReferenceCase) -> ReferenceResult: + """Reduce an explicit event schedule without production inputs.""" + targets = tuple(identifier for identifier, _text in case.targets) + bindings = _binding_keys(case.scopes) + event_max = 4 * len(bindings) + 3 * len(targets) + 7 + _validate_schedule(case.events, bindings, targets, event_max) + admission = _admit(case) + if admission is not ReferenceAdmission.ADMITTED: + return _not_opened(admission, len(bindings), len(case.events), event_max, "admission_rejected") + if case.runtime_capability != "compatible": + return _not_opened(admission, len(bindings), len(case.events), event_max, "runtime_capability") + if _is_pre_dispatch_cancellation(case.events): + task_outcomes = tuple((target, "cancelled", "cancellation") for target in targets) + return ReferenceResult( + admission, + ReferenceInvocation.CANCELLED, + "cancellation", + len(bindings), + task_outcomes, + (), + (), + (), + "not_entered", + False, + "not_entered", + len(case.events), + event_max, + ) + binding_states, global_binding_fault = _binding_outcomes(case.events, bindings) + task_outcomes = _task_outcomes(case.events, targets, binding_states) + if global_binding_fault: + task_outcomes = tuple((target, "inconsistent", "contradictory") for target in targets) + task_corruption = next( + (event.outcome for event in case.events if event.kind is ReferenceEventKind.TASK_CORRUPTION), + None, + ) + if task_corruption is not None: + task_outcomes = tuple((target, "inconsistent", task_corruption) for target in targets) + cleanup = _cleanup_outcome(case.events) + invocation, reason = _invocation_outcome(case.events, task_outcomes, global_binding_fault, cleanup) + published = _event_outcome(case.events, ReferenceEventKind.PUBLICATION) == "accepted" + released = _released(case, task_outcomes) if invocation is ReferenceInvocation.COMPLETED and published else () + private_succeeded = tuple(target for target, state, _reason in task_outcomes if state == "succeeded") + private_inconsistent = tuple(target for target, state, _reason in task_outcomes if state == "inconsistent") + return ReferenceResult( + admission, + invocation, + reason, + len(bindings), + task_outcomes, + private_succeeded, + private_inconsistent, + released, + cleanup, + published, + _event_outcome(case.events, ReferenceEventKind.TEARDOWN), + len(case.events), + event_max, + ) + + +def reference_cases() -> Iterator[ReferenceCase]: + """Yield the frozen finite Phase 5 semantic-class envelope.""" + for target_count in range(1, 5): + for context_count in range(4): + targets = tuple((f"t{index}", f"target-{index}") for index in range(target_count)) + contexts = tuple((f"c{index}", "é" if index == 0 else f"context-{index}") for index in range(context_count)) + base_scopes = _base_scopes(targets, contexts) + yield from _case_family(targets, contexts, base_scopes, "declared", "none") + if target_count > 1: + yield _make_case( + f"{target_count}t-{context_count}c-scope-declaration-reversed", + targets, + contexts, + tuple(reversed(base_scopes)), + "scope_declaration_reversed", + "none", + ) + if context_count > 1: + reversed_scopes = ( + replace(base_scopes[0], context=tuple(reversed(base_scopes[0].context))), + *base_scopes[1:], + ) + yield from _case_family(targets, contexts, reversed_scopes, "reversed", "none") + if 2 <= target_count <= 3: + for cycle_index, cyclic in enumerate(_cycle_scopes(targets, contexts)): + cycle_class = f"{target_count}-target-{cycle_index}" + yield from _case_family(targets, contexts, cyclic, "declared", cycle_class) + yield from _invalid_semantic_cases() + + +def corpus_manifest() -> dict[str, object]: + digest = sha256() + counts: dict[str, dict[str, int]] = { + "cardinality": {}, + "context_order": {}, + "context_cycle": {}, + "payload_class": {}, + "limit_class": {}, + "schedule_class": {}, + "t_b_emax": {}, + } + graph_keys: set[tuple[object, ...]] = set() + trace_count = 0 + actual_event_count = 0 + for case in reference_cases(): + result = evaluate(case) + digest.update(canonical_case(case, result)) + trace_count += 1 + actual_event_count += result.event_count + _increment(counts["cardinality"], f"{len(case.targets)}t-{len(case.context_only)}c") + _increment(counts["context_order"], case.order_class) + _increment(counts["context_cycle"], case.cycle_class) + _increment(counts["payload_class"], case.payload_class) + _increment(counts["limit_class"], case.limit_class) + _increment(counts["schedule_class"], case.schedule_class) + _increment(counts["t_b_emax"], f"T{len(case.targets)}-B{result.binding_count}-E{result.event_max}") + graph_keys.add((case.targets, case.context_only, case.scopes, case.order_class, case.cycle_class)) + return { + "generator_version": "phase5-reference-v2", + "model_version": "target-context-event-workframe-v2", + "graph_count": len(graph_keys), + "canonical_trace_count": trace_count, + "actual_event_count": actual_event_count, + "counts": counts, + "ceiling_fields": ["datum_bytes", "id_bytes", "members", "context_bytes", "references", "expanded_bytes"], + "ceiling_domain": ["zero", "exact", "exact_plus_one"], + "payload_domain": ["empty", "one_byte", "multibyte", "exact_limit", "one_over_limit"], + "event_bound": "4B+3T+7", + "sha256": digest.hexdigest(), + } + + +def canonical_case(case: ReferenceCase, result: ReferenceResult) -> bytes: + payload = {"case": asdict(case), "result": asdict(result)} + return (json.dumps(payload, sort_keys=True, separators=(",", ":")) + "\n").encode() + + +def schedule_for( + case: ReferenceCase, + *, + binding_evidence: str = "exact", + cleanup: str = "verified", + task_schedule: str = "success", + teardown: str = "accepted", +) -> tuple[ReferenceEvent, ...]: + """Build one bounded canonical representative from independent event classes.""" + bindings = _binding_keys(case.scopes) + if task_schedule == "cancel_pre_dispatch": + return (ReferenceEvent(ReferenceEventKind.CANCELLATION),) + events = _binding_events(bindings, binding_evidence) + events += _task_events(tuple(identifier for identifier, _text in case.targets), task_schedule) + events += _cleanup_events(cleanup) + events += ( + ReferenceEvent(ReferenceEventKind.PUBLICATION), + ReferenceEvent(ReferenceEventKind.TEARDOWN, outcome=teardown), + ) + return events + + +def _case_family( + targets: tuple[tuple[str, str], ...], + contexts: tuple[tuple[str, str], ...], + scopes: tuple[ReferenceScope, ...], + order_class: str, + cycle_class: str, +) -> Iterator[ReferenceCase]: + stem = f"{len(targets)}t-{len(contexts)}c-{order_class}-{cycle_class}" + base = _make_case(stem, targets, contexts, scopes, order_class, cycle_class) + yield base + yield from _payload_cases(base) + yield from _ceiling_cases(base) + for state in ("missing", "incompatible", "weakened", "retention_enabled", "profile", "schema", "ordering"): + yield _reschedule(replace(base, case_id=f"{stem}-preflight-{state}", preflight_capability=state)) + yield _reschedule(replace(base, case_id=f"{stem}-runtime-{state}", runtime_capability=state)) + if _binding_keys(scopes): + for evidence in ("missing", "duplicate", "wrong_ordinal", "foreign", "cross_target", "contradictory"): + yield _reschedule(replace(base, case_id=f"{stem}-binding-{evidence}"), binding_evidence=evidence) + for cleanup in ("failed", "missing", "duplicate", "foreign", "incompatible", "contradictory"): + yield _reschedule(replace(base, case_id=f"{stem}-cleanup-{cleanup}"), cleanup=cleanup) + for task_schedule in ("cancel_pre_dispatch", "trusted_stop", "transport_loss", "terminal_then_cancel"): + yield _reschedule(replace(base, case_id=f"{stem}-{task_schedule}"), task_schedule=task_schedule) + yield _reschedule(replace(base, case_id=f"{stem}-teardown-failed"), teardown="failed") + for terminal_fault in ( + "missing", + "duplicate", + "foreign", + "stale", + "cross_target", + "plan_mismatch", + "contradictory", + ): + yield _reschedule( + replace(base, case_id=f"{stem}-terminal-{terminal_fault}"), + task_schedule=f"terminal_{terminal_fault}", + ) + if len(targets) > 1: + dependency = ((targets[0][0], targets[1][0]),) + overlap = "overlap" if cycle_class != "none" else "disjoint" + yield _reschedule( + replace( + base, + case_id=f"{stem}-dependency-{overlap}", + dependencies=dependency, + schedule_class=f"dependency_{overlap}", + ) + ) + + +def _make_case( + case_id: str, + targets: tuple[tuple[str, str], ...], + contexts: tuple[tuple[str, str], ...], + scopes: tuple[ReferenceScope, ...], + order_class: str, + cycle_class: str, +) -> ReferenceCase: + groups = tuple((identifier,) for identifier, _text in targets) + seed = ReferenceCase( + case_id, + targets, + contexts, + scopes, + _exact_limits(targets, contexts, scopes), + (), + groups=groups, + order_class=order_class, + cycle_class=cycle_class, + ) + return _reschedule(seed) + + +def _reschedule( + case: ReferenceCase, + *, + binding_evidence: str = "exact", + cleanup: str = "verified", + task_schedule: str = "success", + teardown: str = "accepted", +) -> ReferenceCase: + schedule_class = ":".join((binding_evidence, cleanup, task_schedule, teardown)) + return replace( + case, + events=schedule_for( + case, binding_evidence=binding_evidence, cleanup=cleanup, task_schedule=task_schedule, teardown=teardown + ), + schedule_class=schedule_class, + ) + + +def _payload_cases(base: ReferenceCase) -> Iterator[ReferenceCase]: + if not base.context_only: + return + for payload_class, payload in (("empty", ""), ("one_byte", "x"), ("exact_limit", "x" * 8)): + contexts = tuple((identifier, payload) for identifier, _text in base.context_only) + scopes = base.scopes + candidate = replace( + base, + case_id=f"{base.case_id}-payload-{payload_class}", + context_only=contexts, + limits=_exact_limits(base.targets, contexts, scopes), + payload_class=payload_class, + ) + yield _reschedule(candidate) + contexts = tuple((identifier, "x" * 9) for identifier, _text in base.context_only) + exact = _exact_limits(base.targets, contexts, base.scopes) + one_over = replace(exact, context_bytes=max(0, exact.context_bytes - 1)) + candidate = replace( + base, + case_id=f"{base.case_id}-payload-one_over_limit", + context_only=contexts, + limits=one_over, + payload_class="one_over_limit", + ) + yield _reschedule(candidate) + + +def _ceiling_cases(base: ReferenceCase) -> Iterator[ReferenceCase]: + exact = _exact_limits(base.targets, base.context_only, base.scopes) + for field in asdict(exact): + for ceiling, value in ( + ("zero", 0), + ("exact", getattr(exact, field)), + ("exact_plus_one", getattr(exact, field) + 1), + ): + limits = replace(exact, **{field: value}) + candidate = replace( + base, case_id=f"{base.case_id}-{field}-{ceiling}", limits=limits, limit_class=f"{field}:{ceiling}" + ) + yield _reschedule(candidate) + + +def _invalid_semantic_cases() -> Iterator[ReferenceCase]: + targets = (("t0", "target"), ("t1", "peer")) + contexts = (("c0", "context"),) + valid = _make_case( + "invalid-base", targets, contexts, (ReferenceScope("t0", ("c0",)), ReferenceScope("t1", ())), "declared", "none" + ) + mutations = ( + ("missing_scope", {"scopes": valid.scopes[:1]}), + ("duplicate_scope", {"scopes": (*valid.scopes, valid.scopes[0])}), + ("unknown_target", {"scopes": (ReferenceScope("unknown", ("c0",)), valid.scopes[1])}), + ("unknown_context", {"scopes": (ReferenceScope("t0", ("unknown",)), valid.scopes[1])}), + ("self_context", {"scopes": (ReferenceScope("t0", ("t0", "c0")), valid.scopes[1])}), + ("duplicate_member", {"scopes": (ReferenceScope("t0", ("c0", "c0")), valid.scopes[1])}), + ("orphan", {"scopes": (ReferenceScope("t0", ()), valid.scopes[1])}), + ( + "target_disabled", + {"scopes": (ReferenceScope("t0", ("c0", "t1")), valid.scopes[1]), "allow_target_as_context": False}, + ), + ("relation", {"relation": "wildcard"}), + ("profile", {"profile": "future"}), + ("schema", {"schema": "future"}), + ("ordering", {"ordering": "implicit"}), + ) + for name, changes in mutations: + candidate = replace(valid, case_id=f"invalid-{name}", **changes) + yield _reschedule(candidate) + + +def _base_scopes( + targets: tuple[tuple[str, str], ...], + contexts: tuple[tuple[str, str], ...], +) -> tuple[ReferenceScope, ...]: + context_ids = tuple(identifier for identifier, _text in contexts) + return tuple( + ReferenceScope(target, context_ids if index == 0 else ()) for index, (target, _text) in enumerate(targets) + ) + + +def _cycle_scopes( + targets: tuple[tuple[str, str], ...], + contexts: tuple[tuple[str, str], ...], +) -> Iterator[tuple[ReferenceScope, ...]]: + target_ids = tuple(identifier for identifier, _text in targets) + context_ids = tuple(identifier for identifier, _text in contexts) + directions = (1,) if len(targets) == 2 else (1, -1) + for direction in directions: + scopes: list[ReferenceScope] = [] + remaining = list(context_ids) + for index, target in enumerate(target_ids): + capacity = 2 + owned = tuple(remaining[:capacity]) + del remaining[:capacity] + scopes.append(ReferenceScope(target, (*owned, target_ids[(index + direction) % len(target_ids)]))) + if remaining: + scopes[0] = replace(scopes[0], context=(*scopes[0].context, *remaining)) + yield tuple(scopes) + + +def _binding_keys(scopes: tuple[ReferenceScope, ...]) -> tuple[tuple[str, str, int], ...]: + return tuple((scope.target, member, ordinal) for scope in scopes for ordinal, member in enumerate(scope.context)) + + +def _binding_events( + bindings: tuple[tuple[str, str, int], ...], + evidence: str, +) -> tuple[ReferenceEvent, ...]: + events: list[ReferenceEvent] = [] + for index, (owner, _member, ordinal) in enumerate(bindings): + subject = f"{owner}:{ordinal}" + events.append(ReferenceEvent(ReferenceEventKind.BINDING_CONSTRUCTION, subject)) + events.append(ReferenceEvent(ReferenceEventKind.BINDING_COMMITMENT, subject)) + if not (index == 0 and evidence == "missing"): + events.append(ReferenceEvent(ReferenceEventKind.BINDING_CONSUMPTION, subject)) + if bindings and evidence not in {"exact", "missing"}: + owner, _member, ordinal = bindings[0] + events.append(ReferenceEvent(ReferenceEventKind.BINDING_CORRUPTION, f"{owner}:{ordinal}", evidence)) + return tuple(events) + + +def _task_events(targets: tuple[str, ...], schedule: str) -> tuple[ReferenceEvent, ...]: + events = [ReferenceEvent(ReferenceEventKind.TASK_DISPATCH, target) for target in targets] + if schedule == "trusted_stop": + events.extend( + (ReferenceEvent(ReferenceEventKind.CANCELLATION), ReferenceEvent(ReferenceEventKind.TRUSTED_STOP)) + ) + elif schedule == "transport_loss": + events.extend( + (ReferenceEvent(ReferenceEventKind.CANCELLATION), ReferenceEvent(ReferenceEventKind.TRANSPORT_LOSS)) + ) + elif schedule == "terminal_missing": + events.extend(ReferenceEvent(ReferenceEventKind.TASK_TERMINAL, target, "success") for target in targets[1:]) + else: + events.extend(ReferenceEvent(ReferenceEventKind.TASK_TERMINAL, target, "success") for target in targets) + if schedule == "terminal_then_cancel": + events.append(ReferenceEvent(ReferenceEventKind.CANCELLATION)) + elif schedule.startswith("terminal_"): + events.append( + ReferenceEvent( + ReferenceEventKind.TASK_CORRUPTION, + targets[0], + schedule.removeprefix("terminal_"), + ) + ) + return tuple(events) + + +def _cleanup_events(cleanup: str) -> tuple[ReferenceEvent, ...]: + primary = ReferenceEvent(ReferenceEventKind.CLEANUP_PRIMARY, outcome=cleanup) + if cleanup in {"duplicate", "contradictory"}: + return (primary, ReferenceEvent(ReferenceEventKind.CLEANUP_COMPETING, outcome=cleanup)) + return (primary,) + + +def _validate_schedule( + events: tuple[ReferenceEvent, ...], + bindings: tuple[tuple[str, str, int], ...], + targets: tuple[str, ...], + event_max: int, +) -> None: + if len(events) > event_max: + raise AssertionError("reference trace exceeded the deterministic Phase 5 bound") + binding_slots = {kind for kind in ReferenceEventKind if kind.value.startswith("binding_")} + task_slots = { + ReferenceEventKind.TASK_DISPATCH, + ReferenceEventKind.TASK_TERMINAL, + ReferenceEventKind.TASK_CORRUPTION, + } + invocation_slots = set(ReferenceEventKind) - binding_slots - task_slots + if sum(event.kind in binding_slots for event in events) > 4 * len(bindings): + raise AssertionError("reference binding event cardinality exceeded") + if sum(event.kind in task_slots for event in events) > 3 * len(targets): + raise AssertionError("reference task event cardinality exceeded") + if sum(event.kind in invocation_slots for event in events) > 7: + raise AssertionError("reference invocation event cardinality exceeded") + _validate_lifecycle_order(events) + + +def _validate_lifecycle_order(events: tuple[ReferenceEvent, ...]) -> None: + if _is_pre_dispatch_cancellation(events): + return + positions = {id(event): index for index, event in enumerate(events)} + for subject in {event.subject for event in events if event.kind.value.startswith("binding_")}: + construction = _first_position(events, ReferenceEventKind.BINDING_CONSTRUCTION, subject) + commitment = _first_position(events, ReferenceEventKind.BINDING_COMMITMENT, subject) + consumption = _first_position(events, ReferenceEventKind.BINDING_CONSUMPTION, subject) + if construction is None or commitment is None or construction >= commitment: + raise AssertionError("binding construction must precede commitment") + if consumption is not None and commitment >= consumption: + raise AssertionError("binding commitment must precede consumption") + binding_end = max( + (positions[id(event)] for event in events if event.kind.value.startswith("binding_")), + default=-1, + ) + for event in events: + if event.kind is ReferenceEventKind.TASK_DISPATCH and positions[id(event)] <= binding_end: + raise AssertionError("dispatch must follow binding construction") + if event.kind is ReferenceEventKind.TASK_TERMINAL: + dispatch = _first_position(events, ReferenceEventKind.TASK_DISPATCH, event.subject) + if dispatch is None or dispatch >= positions[id(event)]: + raise AssertionError("task terminal must follow dispatch") + cleanup_positions = [ + positions[id(event)] + for event in events + if event.kind in {ReferenceEventKind.CLEANUP_PRIMARY, ReferenceEventKind.CLEANUP_COMPETING} + ] + pre_cleanup = [ + positions[id(event)] + for event in events + if event.kind + not in { + ReferenceEventKind.CLEANUP_PRIMARY, + ReferenceEventKind.CLEANUP_COMPETING, + ReferenceEventKind.PUBLICATION, + ReferenceEventKind.TEARDOWN, + } + ] + publication = _first_position(events, ReferenceEventKind.PUBLICATION) + teardown = _first_position(events, ReferenceEventKind.TEARDOWN) + if not cleanup_positions or (pre_cleanup and min(cleanup_positions) <= max(pre_cleanup)): + raise AssertionError("cleanup must follow terminal evidence") + if publication is None or publication <= max(cleanup_positions): + raise AssertionError("publication must follow cleanup") + if teardown is None or teardown <= publication: + raise AssertionError("teardown must follow publication") + + +def _first_position( + events: tuple[ReferenceEvent, ...], + kind: ReferenceEventKind, + subject: str | None = None, +) -> int | None: + return next( + ( + index + for index, event in enumerate(events) + if event.kind is kind and (subject is None or event.subject == subject) + ), + None, + ) + + +def _is_pre_dispatch_cancellation(events: tuple[ReferenceEvent, ...]) -> bool: + return any(event.kind is ReferenceEventKind.CANCELLATION for event in events) and not any( + event.kind is ReferenceEventKind.TASK_DISPATCH for event in events + ) + + +def _admit(case: ReferenceCase) -> ReferenceAdmission: + targets = tuple(identifier for identifier, _text in case.targets) + contexts = tuple(identifier for identifier, _text in case.context_only) + text = dict((*case.targets, *case.context_only)) + scope_targets = tuple(scope.target for scope in case.scopes) + if set(scope_targets) != set(targets) or len(scope_targets) != len(set(scope_targets)): + return ReferenceAdmission.STRUCTURAL + referenced_context: set[str] = set() + for scope in case.scopes: + if ( + scope.target not in targets + or scope.target in scope.context + or len(scope.context) != len(set(scope.context)) + ): + return ReferenceAdmission.STRUCTURAL + if any(member not in text for member in scope.context): + return ReferenceAdmission.STRUCTURAL + if not case.allow_target_as_context and any(member in targets for member in scope.context): + return ReferenceAdmission.STRUCTURAL + referenced_context.update(member for member in scope.context if member in contexts) + if referenced_context != set(contexts): + return ReferenceAdmission.STRUCTURAL + if (case.relation, case.profile, case.schema, case.ordering) != ( + "bounded_context", + "target-context-v1", + "context-workframe-v1", + "declared", + ): + return ReferenceAdmission.CONTRACT + if _exceeds(_exact_limits(case.targets, case.context_only, case.scopes), case.limits): + return ReferenceAdmission.LIMIT + return ( + ReferenceAdmission.ADMITTED + if case.preflight_capability == "compatible" + else ReferenceAdmission.PREFLIGHT_CAPABILITY + ) + + +def _exact_limits( + targets: tuple[tuple[str, str], ...], + contexts: tuple[tuple[str, str], ...], + scopes: tuple[ReferenceScope, ...], +) -> ReferenceLimits: + values = (*targets, *contexts) + text = dict(values) + scope_bytes = tuple( + sum(len(text[member].encode()) for member in scope.context if member in text) for scope in scopes + ) + return ReferenceLimits( + max((len(value.encode()) for _identifier, value in values), default=0), + max((len(identifier.encode()) for identifier, _value in values), default=0), + max((len(scope.context) for scope in scopes), default=0), + max(scope_bytes, default=0), + sum(len(scope.context) for scope in scopes), + sum(len(value.encode()) for _identifier, value in targets) + sum(scope_bytes), + ) + + +def _exceeds(actual: ReferenceLimits, ceiling: ReferenceLimits) -> bool: + return any( + actual_value > ceiling_value + for actual_value, ceiling_value in zip(asdict(actual).values(), asdict(ceiling).values(), strict=True) + ) + + +def _binding_outcomes( + events: tuple[ReferenceEvent, ...], + bindings: tuple[tuple[str, str, int], ...], +) -> tuple[dict[str, tuple[str, str]], bool]: + states: dict[str, tuple[str, str]] = {} + global_fault = False + for owner, _member, ordinal in bindings: + subject = f"{owner}:{ordinal}" + kinds = {event.kind for event in events if event.subject == subject} + if ReferenceEventKind.BINDING_CONSTRUCTION not in kinds or ReferenceEventKind.BINDING_COMMITMENT not in kinds: + states[subject] = ("failed", "construction_missing") + elif ReferenceEventKind.BINDING_CONSUMPTION not in kinds: + states[subject] = ("inconsistent", "missing") + else: + states[subject] = ("available", "none") + for event in events: + if event.kind is not ReferenceEventKind.BINDING_CORRUPTION: + continue + if event.outcome in {"foreign", "cross_target", "contradictory"}: + global_fault = True + elif event.subject in states: + reason = "contradictory" if event.outcome == "wrong_ordinal" else event.outcome + states[event.subject] = ("inconsistent", reason) + return states, global_fault + + +def _task_outcomes( + events: tuple[ReferenceEvent, ...], + targets: tuple[str, ...], + binding_states: dict[str, tuple[str, str]], +) -> tuple[tuple[str, str, str], ...]: + outcomes: list[tuple[str, str, str]] = [] + cancellation = any(event.kind is ReferenceEventKind.CANCELLATION for event in events) + trusted_stop = any(event.kind is ReferenceEventKind.TRUSTED_STOP for event in events) + transport_loss = any(event.kind is ReferenceEventKind.TRANSPORT_LOSS for event in events) + for target in targets: + owned_faults = tuple( + value + for subject, value in binding_states.items() + if subject.startswith(f"{target}:") and value[0] != "available" + ) + if owned_faults: + state, reason = owned_faults[0] + elif not any(event.kind is ReferenceEventKind.TASK_DISPATCH and event.subject == target for event in events): + state, reason = ("cancelled", "cancellation") if cancellation else ("failed", "not_dispatched") + elif any(event.kind is ReferenceEventKind.TASK_TERMINAL and event.subject == target for event in events): + terminal = next( + event for event in events if event.kind is ReferenceEventKind.TASK_TERMINAL and event.subject == target + ) + state, reason = ("succeeded", "none") if terminal.outcome == "success" else ("failed", terminal.outcome) + elif trusted_stop: + state, reason = "cancelled", "stop_acknowledged" + elif transport_loss: + state, reason = "lost", "transport_lost" + else: + state, reason = "inconsistent", "terminal_missing" + outcomes.append((target, state, reason)) + return tuple(outcomes) + + +def _cleanup_outcome(events: tuple[ReferenceEvent, ...]) -> str: + cleanup = tuple( + event + for event in events + if event.kind in {ReferenceEventKind.CLEANUP_PRIMARY, ReferenceEventKind.CLEANUP_COMPETING} + ) + if len(cleanup) != 1: + return "unconfirmed" + outcome = cleanup[0].outcome + if outcome == "verified": + return "verified" + if outcome == "failed": + return "failed" + return "unconfirmed" + + +def _invocation_outcome( + events: tuple[ReferenceEvent, ...], + tasks: tuple[tuple[str, str, str], ...], + global_binding_fault: bool, + cleanup: str, +) -> tuple[ReferenceInvocation, str]: + if global_binding_fault: + return ReferenceInvocation.INCONSISTENT, "contradictory" + if cleanup == "failed": + return ReferenceInvocation.FAILED, "cleanup_failed" + if cleanup != "verified": + return ReferenceInvocation.INCONSISTENT, "cleanup_unconfirmed" + states = {state for _target, state, _reason in tasks} + if "lost" in states: + return ReferenceInvocation.LOST, "transport_lost" + if states == {"cancelled"}: + return ReferenceInvocation.CANCELLED, "cancellation" + if any(event.kind is ReferenceEventKind.CANCELLATION for event in events): + return ReferenceInvocation.CANCELLED, "cancellation" + if any(event.kind is ReferenceEventKind.TASK_CORRUPTION for event in events): + return ReferenceInvocation.INCONSISTENT, "terminal_attribution_invalid" + return ReferenceInvocation.COMPLETED, "none" + + +def _released(case: ReferenceCase, tasks: tuple[tuple[str, str, str], ...]) -> tuple[str, ...]: + eligible = {target for target, state, _reason in tasks if state == "succeeded"} + changed = True + while changed: + changed = False + for prerequisite, dependent in case.dependencies: + if prerequisite not in eligible and dependent in eligible: + eligible.remove(dependent) + changed = True + for group in case.groups: + if not set(group).issubset(eligible) and eligible.intersection(group): + eligible.difference_update(group) + changed = True + return tuple(target for target, _text in case.targets if target in eligible) + + +def _event_outcome(events: tuple[ReferenceEvent, ...], kind: ReferenceEventKind) -> str: + matches = tuple(event.outcome for event in events if event.kind is kind) + return matches[0] if len(matches) == 1 else "missing" if not matches else "contradictory" + + +def _not_opened( + admission: ReferenceAdmission, + binding_count: int, + event_count: int, + event_max: int, + reason: str, +) -> ReferenceResult: + return ReferenceResult( + admission, + ReferenceInvocation.NOT_OPENED, + reason, + binding_count, + (), + (), + (), + (), + "not_entered", + False, + "not_entered", + event_count, + event_max, + ) + + +def _increment(counts: dict[str, int], key: str) -> None: + counts[key] = counts.get(key, 0) + 1 diff --git a/tests/engine/execution/phase6_reference_manifest.json b/tests/engine/execution/phase6_reference_manifest.json new file mode 100644 index 00000000..fd10de10 --- /dev/null +++ b/tests/engine/execution/phase6_reference_manifest.json @@ -0,0 +1,91 @@ +{ + "alphabet": [ + "A", + "é", + "😀", + "é", + " " + ], + "actual_event_count": 446, + "canonical_trace_count": 44, + "case_count": 21, + "digest": "37d92a09fda08d77d6f7f69447543d5eff37c0796ddf4a0098cb736b2c45902f", + "event_alphabet": [ + "dispatch", + "terminal", + "candidate_decision", + "evidence", + "finalize", + "patch", + "transform", + "verify", + "group_verify", + "release", + "cancel", + "trusted_stop", + "loss", + "cleanup", + "immutable_accept", + "teardown" + ], + "generator_version": "phase6-finite-envelope/v3", + "independence_relation": [ + [ + "candidate_decision", + "evidence" + ], + [ + "patch", + "group_verify" + ], + [ + "terminal", + "terminal" + ] + ], + "lifecycle_trace_count": 13, + "max_event_count": 45, + "ordered_races": [ + [ + "dispatch", + "terminal" + ], + [ + "cancel", + "terminal" + ], + [ + "verify", + "release" + ], + [ + "finalize", + "release" + ], + [ + "teardown", + "immutable_accept" + ] + ], + "race_trace_count": 23, + "reference_model_version": "phase6-reference/v3", + "schedule_class_counts": { + "cancel-after-verification": 1, + "cancel-dispatch": 2, + "cancel-terminal": 2, + "cleanup-failure": 1, + "contradictory-record-patch": 2, + "dispatch-terminal": 2, + "duplicate-resolver-completion": 1, + "finalize-release": 2, + "late-candidate-after-cancel": 1, + "late-candidate-after-loss": 1, + "late-evidence-after-cancel": 1, + "late-evidence-after-loss": 1, + "local-failure-independent-success": 1, + "success": 21, + "teardown-failure-after-acceptance": 1, + "teardown-acceptance": 2, + "verify-release": 2 + } +} diff --git a/tests/engine/execution/phase6_reference_model.py b/tests/engine/execution/phase6_reference_model.py new file mode 100644 index 00000000..2be9e139 --- /dev/null +++ b/tests/engine/execution/phase6_reference_model.py @@ -0,0 +1,997 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pure Phase 6 oracle with no production, pandas, or DataDesigner imports.""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import asdict, dataclass, field, replace +from enum import Enum +from typing import TypeAlias + +REFERENCE_MODEL_VERSION = "phase6-reference/v3" +GENERATOR_VERSION = "phase6-finite-envelope/v3" +SYMBOLIC_ALPHABET = ("A", "é", "😀", "e\u0301", " ") +EVENT_ALPHABET = ( + "dispatch", + "terminal", + "candidate_decision", + "evidence", + "finalize", + "patch", + "transform", + "verify", + "group_verify", + "release", + "cancel", + "trusted_stop", + "loss", + "cleanup", + "immutable_accept", + "teardown", +) +INDEPENDENCE_RELATION = ( + ("candidate_decision", "evidence"), + ("patch", "group_verify"), + ("terminal", "terminal"), +) +ORDERED_RACES = ( + ("dispatch", "terminal"), + ("cancel", "terminal"), + ("verify", "release"), + ("finalize", "release"), + ("teardown", "immutable_accept"), +) +_REPLACEMENT = "[REDACTED]" +_CandidateKey: TypeAlias = tuple[int, int, int, str, str, str, str | None] + + +class ReferenceEventKind(str, Enum): + DISPATCH = "dispatch" + TERMINAL = "terminal" + CANDIDATE_DECISION = "candidate_decision" + EVIDENCE = "evidence" + FINALIZE = "finalize" + PATCH = "patch" + TRANSFORM = "transform" + VERIFY = "verify" + GROUP_VERIFY = "group_verify" + RELEASE = "release" + CANCEL = "cancel" + TRUSTED_STOP = "trusted_stop" + LOSS = "loss" + CLEANUP = "cleanup" + IMMUTABLE_ACCEPT = "immutable_accept" + TEARDOWN = "teardown" + + +@dataclass(frozen=True, slots=True) +class ReferenceEvent: + kind: ReferenceEventKind + subject: str = "invocation" + outcome: str = "accepted" + + +@dataclass(frozen=True, slots=True) +class ReferenceScheduleResult: + invocation: str + task_terminal: str + task_outcomes: tuple[tuple[str, str], ...] + cancellation: str + finalized: bool + verified: bool + cleanup: str + teardown: str + immutable_result: bool + release: str + released_subjects: tuple[str, ...] + event_count: int + + +def default_schedule() -> tuple[ReferenceEvent, ...]: + """Return the canonical bounded successful Phase 6 schedule.""" + return tuple( + ReferenceEvent(kind) + for kind in ( + ReferenceEventKind.DISPATCH, + ReferenceEventKind.CANDIDATE_DECISION, + ReferenceEventKind.EVIDENCE, + ReferenceEventKind.FINALIZE, + ReferenceEventKind.PATCH, + ReferenceEventKind.TRANSFORM, + ReferenceEventKind.VERIFY, + ReferenceEventKind.GROUP_VERIFY, + ReferenceEventKind.TERMINAL, + ReferenceEventKind.CLEANUP, + ReferenceEventKind.IMMUTABLE_ACCEPT, + ReferenceEventKind.TEARDOWN, + ReferenceEventKind.RELEASE, + ) + ) + + +@dataclass(frozen=True, slots=True) +class ReferenceCandidate: + target: int + start: int + end: int + source_slice: str + label: str + decision: str = "keep" + reclassified_label: str | None = None + + +@dataclass(frozen=True, slots=True) +class ReferenceEvidence: + kind: str + left_candidate: int + right_candidate: int + + +@dataclass(frozen=True, slots=True) +class ReferenceCase: + name: str + texts: tuple[str, ...] + candidates: tuple[ReferenceCandidate, ...] + evidence: tuple[ReferenceEvidence, ...] = () + dependencies: tuple[tuple[int, int], ...] = () + groups: tuple[tuple[int, ...], ...] = () + group_passes: tuple[bool, ...] = () + returned: tuple[str, ...] | None = None + events: tuple[ReferenceEvent, ...] = field(default_factory=default_schedule) + schedule_class: str = "success" + + +@dataclass(frozen=True, slots=True) +class ReferenceMention: + candidate_indexes: tuple[int, ...] + target: int + start: int + end: int + source_slice: str + label: str + + +@dataclass(frozen=True, slots=True) +class ReferenceResult: + rejection: str | None + mentions: tuple[ReferenceMention, ...] + clusters: tuple[tuple[int, ...], ...] + outputs: tuple[str, ...] + released_groups: tuple[int, ...] + schedule: ReferenceScheduleResult + event_count: int + max_event_count: int + + +def reduce_reference(case: ReferenceCase) -> ReferenceResult: + """Derive the complete bounded outcome without production decisions as inputs.""" + groups = case.groups or tuple((index,) for index in range(len(case.texts))) + group_passes = case.group_passes or tuple(True for _group in groups) + base_bound = _event_bound(case, groups) + schedule = _reduce_schedule(case.events, base_bound) + rejected = _validate_declaration(case, groups, group_passes) + if rejected is not None: + return _rejected_result(rejected, schedule, base_bound) + finalized = _finalize(case) + if isinstance(finalized, str): + return _rejected_result(finalized, schedule, base_bound) + mentions, mention_by_candidate = finalized + clustered = _cluster(mentions, mention_by_candidate, case.evidence) + if isinstance(clustered, str): + return _rejected_result(clustered, schedule, base_bound, mentions=mentions) + outputs = _redact(case.texts, mentions) + if isinstance(outputs, str): + return _rejected_result(outputs, schedule, base_bound, mentions=mentions, clusters=clustered) + if case.returned is not None and case.returned != outputs: + return _rejected_result( + "release_predicate_failed", + schedule, + base_bound, + mentions=mentions, + clusters=clustered, + outputs=outputs, + ) + eligible = set(range(len(case.texts))) + for group, passed in zip(groups, group_passes, strict=True): + if not passed: + eligible.difference_update(group) + scheduled_subjects = { + int(subject.removeprefix("target-")) + for subject, outcome in schedule.task_outcomes + if subject.startswith("target-") and outcome != "succeeded" + } + eligible.difference_update(scheduled_subjects) + while True: + before = set(eligible) + for group in groups: + if not set(group).issubset(eligible): + eligible.difference_update(group) + for prerequisite, dependent in case.dependencies: + if prerequisite not in eligible: + eligible.discard(dependent) + if eligible == before: + break + released = ( + tuple(index for index, group in enumerate(groups) if set(group).issubset(eligible)) + if schedule.release == "accepted" + else () + ) + return ReferenceResult( + None, + mentions, + clustered, + outputs, + released, + schedule, + schedule.event_count, + base_bound, + ) + + +def finite_reference_cases() -> tuple[ReferenceCase, ...]: + return (*_alphabet_cases(), *_anchor_cases(), *_evidence_cases(), *_fault_cases()) + + +def _alphabet_cases() -> tuple[ReferenceCase, ...]: + cases: list[ReferenceCase] = [] + for index, text in enumerate(SYMBOLIC_ALPHABET): + cases.append(ReferenceCase(f"alphabet-{index}-empty", (text,), ())) + cases.append( + ReferenceCase( + f"alphabet-{index}-whole", + (text,), + (ReferenceCandidate(0, 0, len(text), text, "label"),), + ) + ) + return tuple(cases) + + +def _anchor_cases() -> tuple[ReferenceCase, ...]: + return ( + ReferenceCase("repeated-first", ("A A",), (ReferenceCandidate(0, 0, 1, "A", "name"),)), + ReferenceCase( + "repeated-both", + ("A A",), + ( + ReferenceCandidate(0, 0, 1, "A", "name"), + ReferenceCandidate(0, 2, 3, "A", "name"), + ), + ), + ReferenceCase( + "adjacent-unicode", + ("A😀",), + ( + ReferenceCandidate(0, 0, 1, "A", "name"), + ReferenceCandidate(0, 1, 2, "😀", "symbol", "reclass", "emoji"), + ), + ), + ) + + +def _fault_cases() -> tuple[ReferenceCase, ...]: + candidates = ( + ReferenceCandidate(0, 0, 1, "A", "name"), + ReferenceCandidate(0, 2, 3, "B", "name"), + ReferenceCandidate(0, 4, 5, "C", "name"), + ) + return ( + ReferenceCase( + "transitive-contradiction", + ("A B C",), + candidates, + ( + ReferenceEvidence("same_subject", 0, 1), + ReferenceEvidence("same_subject", 1, 2), + ReferenceEvidence("distinct_subject", 0, 2), + ), + ), + ReferenceCase("invalid-offset", ("A",), (ReferenceCandidate(0, -1, 1, "A", "name"),)), + ReferenceCase("slice-mismatch", ("Alice",), (ReferenceCandidate(0, 0, 5, "Mallory", "name"),)), + ReferenceCase( + "missing-decision", + ("Alice",), + (ReferenceCandidate(0, 0, 5, "Alice", "name", "missing"),), + ), + ReferenceCase( + "group-propagation", + ("A", "B"), + (), + dependencies=((0, 1),), + groups=((0,), (1,)), + group_passes=(False, True), + ), + ) + + +def reference_manifest() -> dict[str, object]: + cases = finite_reference_cases() + races = ordered_race_schedules() + lifecycle = lifecycle_reference_cases() + payload: list[dict[str, object]] = [ + {"case": _case_payload(case), "result": asdict(reduce_reference(case))} for case in cases + ] + payload.extend( + { + "race": name, + "orientation": orientation, + "events": [asdict(event) for event in events], + "result": asdict(reduce_reference(ReferenceCase(f"{name}-{orientation}", ("A",), (), events=events))), + } + for name, first, second in races + for orientation, events in (("first", first), ("second", second)) + ) + payload.extend({"lifecycle": _case_payload(case), "result": asdict(reduce_reference(case))} for case in lifecycle) + digest = hashlib.sha256(json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()).hexdigest() + race_event_count = sum(len(events) for _name, first, second in races for events in (first, second)) + lifecycle_event_count = sum(len(case.events) for case in lifecycle) + schedule_class_counts: dict[str, int] = {"success": len(cases)} + for name, _first, _second in races: + schedule_class_counts[name] = 2 + for case in lifecycle: + schedule_class_counts[case.schedule_class] = schedule_class_counts.get(case.schedule_class, 0) + 1 + return { + "alphabet": list(SYMBOLIC_ALPHABET), + "actual_event_count": sum(len(case.events) for case in cases) + race_event_count + lifecycle_event_count, + "canonical_trace_count": len(cases) + 2 * len(races) + len(lifecycle), + "case_count": len(cases), + "digest": digest, + "event_alphabet": list(EVENT_ALPHABET), + "generator_version": GENERATOR_VERSION, + "independence_relation": [list(pair) for pair in INDEPENDENCE_RELATION], + "lifecycle_trace_count": len(lifecycle), + "max_event_count": max(reduce_reference(case).max_event_count for case in (*cases, *lifecycle)), + "ordered_races": [list(pair) for pair in ORDERED_RACES], + "race_trace_count": 2 * len(races) + len(lifecycle), + "reference_model_version": REFERENCE_MODEL_VERSION, + "schedule_class_counts": schedule_class_counts, + } + + +def canonical_schedule(events: tuple[ReferenceEvent, ...]) -> tuple[ReferenceEvent, ...]: + """Canonicalize only adjacent events declared to commute.""" + relation = {frozenset(pair) for pair in INDEPENDENCE_RELATION} + rank = {value: index for index, value in enumerate(EVENT_ALPHABET)} + canonical = list(events) + changed = True + while changed: + changed = False + for index in range(len(canonical) - 1): + left = canonical[index] + right = canonical[index + 1] + if frozenset((left.kind.value, right.kind.value)) in relation and (rank[left.kind.value], left.subject) > ( + rank[right.kind.value], + right.subject, + ): + canonical[index], canonical[index + 1] = right, left + changed = True + return tuple(canonical) + + +def ordered_race_schedules() -> tuple[tuple[str, tuple[ReferenceEvent, ...], tuple[ReferenceEvent, ...]], ...]: + """Return both orientations of each release-critical ordered race.""" + event = ReferenceEvent + kind = ReferenceEventKind + return ( + ( + "dispatch-terminal", + (event(kind.DISPATCH), event(kind.TERMINAL)), + (event(kind.TERMINAL), event(kind.DISPATCH)), + ), + ( + "cancel-terminal", + (event(kind.DISPATCH), event(kind.CANCEL), event(kind.TERMINAL)), + (event(kind.DISPATCH), event(kind.TERMINAL), event(kind.CANCEL)), + ), + ( + "verify-release", + ( + event(kind.DISPATCH), + event(kind.TERMINAL), + event(kind.FINALIZE), + event(kind.CLEANUP), + event(kind.TEARDOWN), + event(kind.VERIFY), + event(kind.IMMUTABLE_ACCEPT), + event(kind.RELEASE), + ), + ( + event(kind.DISPATCH), + event(kind.TERMINAL), + event(kind.FINALIZE), + event(kind.CLEANUP), + event(kind.TEARDOWN), + event(kind.RELEASE), + event(kind.VERIFY), + event(kind.IMMUTABLE_ACCEPT), + ), + ), + ( + "finalize-release", + ( + event(kind.DISPATCH), + event(kind.TERMINAL), + event(kind.VERIFY), + event(kind.CLEANUP), + event(kind.TEARDOWN), + event(kind.FINALIZE), + event(kind.IMMUTABLE_ACCEPT), + event(kind.RELEASE), + ), + ( + event(kind.DISPATCH), + event(kind.TERMINAL), + event(kind.VERIFY), + event(kind.CLEANUP), + event(kind.TEARDOWN), + event(kind.RELEASE), + event(kind.FINALIZE), + event(kind.IMMUTABLE_ACCEPT), + ), + ), + ( + "teardown-acceptance", + ( + event(kind.DISPATCH), + event(kind.TERMINAL), + event(kind.FINALIZE), + event(kind.VERIFY), + event(kind.CLEANUP), + event(kind.TEARDOWN, outcome="failed"), + event(kind.IMMUTABLE_ACCEPT), + event(kind.RELEASE), + ), + ( + event(kind.DISPATCH), + event(kind.TERMINAL), + event(kind.FINALIZE), + event(kind.VERIFY), + event(kind.CLEANUP), + event(kind.IMMUTABLE_ACCEPT), + event(kind.TEARDOWN, outcome="failed"), + event(kind.RELEASE), + ), + ), + ) + + +def lifecycle_reference_cases() -> tuple[ReferenceCase, ...]: + """Return the frozen lifecycle schedules required by the Phase 6 design.""" + event = ReferenceEvent + kind = ReferenceEventKind + + def tail(*, teardown: str = "accepted", cleanup: str = "accepted") -> tuple[ReferenceEvent, ...]: + return ( + event(kind.FINALIZE), + event(kind.VERIFY), + event(kind.CLEANUP, outcome=cleanup), + event(kind.IMMUTABLE_ACCEPT), + event(kind.TEARDOWN, outcome=teardown), + event(kind.RELEASE), + ) + + target = "target-0" + return ( + ReferenceCase( + "cancel-before-dispatch", + ("A",), + (), + events=(event(kind.CANCEL), event(kind.DISPATCH, target), event(kind.TERMINAL, target), *tail()), + schedule_class="cancel-dispatch", + ), + ReferenceCase( + "cancel-after-dispatch", + ("A",), + (), + events=(event(kind.DISPATCH, target), event(kind.CANCEL), event(kind.TERMINAL, target), *tail()), + schedule_class="cancel-dispatch", + ), + ReferenceCase( + "late-candidate-after-cancel", + ("A",), + (), + events=(event(kind.DISPATCH, target), event(kind.CANCEL), event(kind.CANDIDATE_DECISION, target), *tail()), + schedule_class="late-candidate-after-cancel", + ), + ReferenceCase( + "late-evidence-after-cancel", + ("A",), + (), + events=(event(kind.DISPATCH, target), event(kind.CANCEL), event(kind.EVIDENCE, target), *tail()), + schedule_class="late-evidence-after-cancel", + ), + ReferenceCase( + "late-candidate-after-loss", + ("A",), + (), + events=(event(kind.DISPATCH, target), event(kind.LOSS), event(kind.CANDIDATE_DECISION, target), *tail()), + schedule_class="late-candidate-after-loss", + ), + ReferenceCase( + "late-evidence-after-loss", + ("A",), + (), + events=(event(kind.DISPATCH, target), event(kind.LOSS), event(kind.EVIDENCE, target), *tail()), + schedule_class="late-evidence-after-loss", + ), + ReferenceCase( + "duplicate-resolver-completion", + ("A",), + (), + events=(event(kind.DISPATCH, target), event(kind.EVIDENCE, target, "duplicate"), *tail()), + schedule_class="duplicate-resolver-completion", + ), + ReferenceCase( + "patch-before-contradictory-record", + ("A",), + (), + events=( + event(kind.DISPATCH, target), + event(kind.PATCH, target), + event(kind.TERMINAL, target, "contradictory"), + *tail(), + ), + schedule_class="contradictory-record-patch", + ), + ReferenceCase( + "contradictory-record-before-patch", + ("A",), + (), + events=( + event(kind.DISPATCH, target), + event(kind.TERMINAL, target, "contradictory"), + event(kind.PATCH, target), + *tail(), + ), + schedule_class="contradictory-record-patch", + ), + ReferenceCase( + "local-failure-independent-success", + ("A", "B"), + (), + groups=((0,), (1,)), + events=( + event(kind.DISPATCH, "target-0"), + event(kind.TERMINAL, "target-0", "failed"), + event(kind.DISPATCH, "target-1"), + event(kind.TERMINAL, "target-1"), + *tail(), + ), + schedule_class="local-failure-independent-success", + ), + ReferenceCase( + "cancel-after-verification", + ("A",), + (), + events=( + event(kind.DISPATCH, target), + event(kind.TERMINAL, target), + event(kind.FINALIZE), + event(kind.VERIFY), + event(kind.CANCEL), + event(kind.CLEANUP), + event(kind.IMMUTABLE_ACCEPT), + event(kind.TEARDOWN), + event(kind.RELEASE), + ), + schedule_class="cancel-after-verification", + ), + ReferenceCase( + "cleanup-failure", + ("A",), + (), + events=(event(kind.DISPATCH, target), event(kind.TERMINAL, target), *tail(cleanup="unconfirmed")), + schedule_class="cleanup-failure", + ), + ReferenceCase( + "teardown-failure-after-acceptance", + ("A",), + (), + events=(event(kind.DISPATCH, target), event(kind.TERMINAL, target), *tail(teardown="failed")), + schedule_class="teardown-failure-after-acceptance", + ), + ) + + +def schedule_reference_cases() -> tuple[ReferenceCase, ...]: + """Return every frozen schedule that must be compared with production accounting.""" + ordered = tuple( + ReferenceCase( + f"{name}-{orientation}", + ("A",), + (), + events=tuple( + replace(event, subject="target-0") if event.subject == "invocation" else event for event in events + ), + schedule_class=name, + ) + for name, first, second in ordered_race_schedules() + for orientation, events in (("first", first), ("second", second)) + ) + return (*ordered, *lifecycle_reference_cases()) + + +def _evidence_cases() -> tuple[ReferenceCase, ...]: + candidates = ( + ReferenceCandidate(0, 0, 5, "Alice", "name"), + ReferenceCandidate(1, 0, 2, "Al", "alias"), + ) + return tuple( + ReferenceCase(f"evidence-{kind}", ("Alice", "Al"), candidates, evidence) + for kind, evidence in ( + ("none", ()), + ("same", (ReferenceEvidence("same_subject", 0, 1),)), + ("distinct", (ReferenceEvidence("distinct_subject", 0, 1),)), + ) + ) + + +def _validate_declaration( + case: ReferenceCase, + groups: tuple[tuple[int, ...], ...], + group_passes: tuple[bool, ...], +) -> str | None: + target_indexes = set(range(len(case.texts))) + members = tuple(member for group in groups for member in group) + if ( + not case.texts + or set(members) != target_indexes + or len(members) != len(set(members)) + or len(groups) != len(group_passes) + or any(type(passed) is not bool for passed in group_passes) + ): + return "malformed_graph" + if any( + prerequisite not in target_indexes or dependent not in target_indexes or prerequisite == dependent + for prerequisite, dependent in case.dependencies + ): + return "malformed_dependency" + return None + + +def _finalize( + case: ReferenceCase, +) -> tuple[tuple[ReferenceMention, ...], dict[int, int]] | str: + indexed = _index_candidates(case) + if isinstance(indexed, str): + return indexed + accepted = _materialize_mentions(indexed) + rejection = _validate_mention_spans(accepted) + if rejection is not None: + return rejection + mention_by_candidate = { + candidate_index: mention_index + for mention_index, mention in enumerate(accepted) + for candidate_index in mention.candidate_indexes + } + return accepted, mention_by_candidate + + +def _index_candidates(case: ReferenceCase) -> dict[_CandidateKey, list[int]] | str: + exact: dict[_CandidateKey, list[int]] = {} + for index, candidate in enumerate(case.candidates): + if ( + candidate.target not in range(len(case.texts)) + or type(candidate.start) is not int + or type(candidate.end) is not int + or candidate.start < 0 + or candidate.end <= candidate.start + or candidate.end > len(case.texts[candidate.target]) + ): + return "invalid_offset" + if case.texts[candidate.target][candidate.start : candidate.end] != candidate.source_slice: + return "source_slice_mismatch" + if not candidate.label: + return "invalid_label" + if candidate.decision not in {"keep", "reclass", "drop"}: + return "missing_decision" + if candidate.decision == "reclass" and not candidate.reclassified_label: + return "invalid_label" + key = ( + candidate.target, + candidate.start, + candidate.end, + candidate.source_slice, + candidate.label, + candidate.decision, + candidate.reclassified_label, + ) + exact.setdefault(key, []).append(index) + return exact + + +def _materialize_mentions(exact: dict[_CandidateKey, list[int]]) -> tuple[ReferenceMention, ...]: + accepted: list[ReferenceMention] = [] + for key, indexes in exact.items(): + target, start, end, source_slice, label, decision, reclassified = key + if decision == "drop": + continue + accepted.append( + ReferenceMention( + tuple(indexes), + target, + start, + end, + source_slice, + reclassified if decision == "reclass" and reclassified is not None else label, + ) + ) + return tuple(sorted(accepted, key=lambda mention: (mention.target, mention.start, mention.end))) + + +def _validate_mention_spans(accepted: tuple[ReferenceMention, ...]) -> str | None: + previous_end: dict[int, int] = {} + by_span: set[tuple[int, int, int]] = set() + for mention in accepted: + span = (mention.target, mention.start, mention.end) + if span in by_span: + return "contradictory_candidate" + if mention.start < previous_end.get(mention.target, 0): + return "overlap" + by_span.add(span) + previous_end[mention.target] = mention.end + return None + + +def _cluster( + mentions: tuple[ReferenceMention, ...], + mention_by_candidate: dict[int, int], + evidence: tuple[ReferenceEvidence, ...], +) -> tuple[tuple[int, ...], ...] | str: + parents = list(range(len(mentions))) + + def find(index: int) -> int: + while parents[index] != index: + parents[index] = parents[parents[index]] + index = parents[index] + return index + + normalized = _normalize_evidence(mention_by_candidate, evidence) + if isinstance(normalized, str): + return normalized + for kind, left, right in sorted(normalized): + if kind != "same_subject": + continue + left_root = find(left) + right_root = find(right) + if left_root != right_root: + parents[max(left_root, right_root)] = min(left_root, right_root) + if any(find(left) == find(right) for kind, left, right in normalized if kind == "distinct_subject"): + return "evidence_contradiction" + components: dict[int, list[int]] = {} + for index in range(len(mentions)): + components.setdefault(find(index), []).append(index) + return tuple(tuple(component) for _root, component in sorted(components.items())) + + +def _normalize_evidence( + mention_by_candidate: dict[int, int], + evidence: tuple[ReferenceEvidence, ...], +) -> set[tuple[str, int, int]] | str: + normalized: set[tuple[str, int, int]] = set() + for item in evidence: + if ( + item.kind not in {"same_subject", "distinct_subject"} + or item.left_candidate not in mention_by_candidate + or item.right_candidate not in mention_by_candidate + ): + return "invalid_evidence" + left = mention_by_candidate[item.left_candidate] + right = mention_by_candidate[item.right_candidate] + if left == right: + return "invalid_evidence" + edge = (item.kind, min(left, right), max(left, right)) + if edge in normalized: + return "invalid_evidence" + if ("same_subject" if item.kind == "distinct_subject" else "distinct_subject", edge[1], edge[2]) in normalized: + return "evidence_contradiction" + normalized.add(edge) + return normalized + + +def _redact(texts: tuple[str, ...], mentions: tuple[ReferenceMention, ...]) -> tuple[str, ...] | str: + by_target: list[list[ReferenceMention]] = [[] for _text in texts] + for mention in mentions: + if mention.source_slice in _REPLACEMENT: + return "invalid_patch" + by_target[mention.target].append(mention) + outputs: list[str] = [] + for text, target_mentions in zip(texts, by_target, strict=True): + cursor = 0 + parts: list[str] = [] + for mention in target_mentions: + parts.extend((text[cursor : mention.start], _REPLACEMENT)) + cursor = mention.end + parts.append(text[cursor:]) + outputs.append("".join(parts)) + return tuple(outputs) + + +def _reduce_schedule(events: tuple[ReferenceEvent, ...], event_bound: int) -> ReferenceScheduleResult: + if not events: + raise AssertionError("schedule must contain at least one event") + if len(events) > event_bound: + raise AssertionError("schedule exceeds computed event bound") + if any(not isinstance(event, ReferenceEvent) for event in events): + raise AssertionError("schedule contains an invalid event") + + subjects = tuple( + dict.fromkeys( + event.subject + for event in events + if event.subject != "invocation" + and event.kind + in { + ReferenceEventKind.DISPATCH, + ReferenceEventKind.TERMINAL, + ReferenceEventKind.CANDIDATE_DECISION, + ReferenceEventKind.EVIDENCE, + } + ) + ) or ("invocation",) + dispatched: set[str] = set() + task_outcomes = {subject: "missing" for subject in subjects} + finalized = False + verified = False + cancelled = False + cancellation = "none" + cleanup = "unconfirmed" + teardown = "unconfirmed" + immutable_result = False + release = "not_requested" + release_observed = False + released_subjects: tuple[str, ...] = () + lost = False + stopped = False + inconsistent = False + for event in events: + subject = event.subject if event.subject != "invocation" else subjects[0] + match event.kind: + case ReferenceEventKind.DISPATCH: + if not cancelled and not lost and task_outcomes.get(subject) == "missing": + dispatched.add(subject) + case ReferenceEventKind.TERMINAL: + if event.outcome == "contradictory": + inconsistent = True + task_outcomes[subject] = "inconsistent" + elif subject in dispatched and task_outcomes.get(subject) == "missing" and not lost and not stopped: + task_outcomes[subject] = "succeeded" if event.outcome == "accepted" else "failed" + else: + task_outcomes.setdefault(subject, "rejected") + case ReferenceEventKind.CANCEL: + cancellation = ( + "before_dispatch" + if not dispatched + else "after_terminal" + if any(outcome not in {"missing", "cancelled"} for outcome in task_outcomes.values()) + else "after_dispatch" + ) + cancelled = True + for planned in subjects: + if task_outcomes[planned] == "missing": + task_outcomes[planned] = "cancelled" + case ReferenceEventKind.TRUSTED_STOP: + stopped = True + case ReferenceEventKind.LOSS: + lost = True + for active in dispatched: + if task_outcomes[active] == "missing": + task_outcomes[active] = "lost" + case ReferenceEventKind.CANDIDATE_DECISION | ReferenceEventKind.EVIDENCE: + if event.outcome == "duplicate": + inconsistent = True + task_outcomes[subject] = "inconsistent" + case ReferenceEventKind.FINALIZE: + finalized = True + case ReferenceEventKind.VERIFY: + verified = True + case ReferenceEventKind.CLEANUP: + cleanup = event.outcome + case ReferenceEventKind.IMMUTABLE_ACCEPT: + immutable_result = ( + event.outcome == "accepted" + and any(outcome == "succeeded" for outcome in task_outcomes.values()) + and finalized + and verified + and cleanup == "accepted" + and teardown != "failed" + and not cancelled + and not lost + and not stopped + and not inconsistent + and not release_observed + ) + if immutable_result: + released_subjects = tuple( + subject for subject, outcome in task_outcomes.items() if outcome == "succeeded" + ) + case ReferenceEventKind.TEARDOWN: + teardown = event.outcome + case ReferenceEventKind.RELEASE: + release_observed = True + release = "accepted" if immutable_result else "withheld" + case _: + pass + + for subject in subjects: + if task_outcomes[subject] == "missing": + if subject in dispatched: + task_outcomes[subject] = "lost" + lost = True + else: + task_outcomes[subject] = "blocked" + terminal_states = tuple(task_outcomes.values()) + task_terminal = terminal_states[0] if len(set(terminal_states)) == 1 else "mixed" + if immutable_result: + invocation = "completed" + elif cleanup == "unconfirmed": + invocation = "inconsistent" + elif teardown == "failed": + invocation = "failed" + elif cleanup == "failed": + invocation = "failed" + elif inconsistent: + invocation = "inconsistent" + elif lost: + invocation = "lost" + elif cancelled: + invocation = "cancelled" + elif stopped: + invocation = "stopped" + else: + invocation = "completed" + return ReferenceScheduleResult( + invocation, + task_terminal, + tuple(task_outcomes.items()), + cancellation, + finalized, + verified, + cleanup, + teardown, + immutable_result, + release, + released_subjects, + len(events), + ) + + +def _rejected_result( + rejection: str, + schedule: ReferenceScheduleResult, + event_bound: int, + *, + mentions: tuple[ReferenceMention, ...] = (), + clusters: tuple[tuple[int, ...], ...] = (), + outputs: tuple[str, ...] = (), +) -> ReferenceResult: + failed_schedule = replace(schedule, release="withheld") + return ReferenceResult( + rejection, + mentions, + clusters, + outputs, + (), + failed_schedule, + failed_schedule.event_count, + event_bound, + ) + + +def _event_bound(case: ReferenceCase, groups: tuple[tuple[int, ...], ...]) -> int: + task_events = len(case.texts) * 8 * 2 + return ( + task_events + + len(case.candidates) + + len(case.evidence) + + len(case.candidates) + + len(case.texts) + + len(groups) + + 4 + ) + + +def _case_payload(case: ReferenceCase) -> dict[str, object]: + return asdict(case) diff --git a/tests/engine/execution/phase7_reference_manifest.json b/tests/engine/execution/phase7_reference_manifest.json new file mode 100644 index 00000000..56c46e7b --- /dev/null +++ b/tests/engine/execution/phase7_reference_manifest.json @@ -0,0 +1,45 @@ +{ + "actual_event_count": 549, + "canonical_serialization": "UTF8_compact_sorted_key_JSON_complete_corpus_no_trailing_newline", + "canonical_trace_count": 78, + "case_count": 83, + "digest": "ad8b1fb63414571c4b5f50ea3fdf94c8f8c8849d15ea529987d15e656b3149d6", + "event_alphabet": [ + "dispatch_accepted", + "dispatch_rejected", + "candidate_rows", + "failed_record", + "backend_exception", + "cancellation", + "trusted_stop", + "loss", + "transform", + "verify", + "finalize", + "cleanup", + "immutable_accept", + "teardown", + "release" + ], + "generator_version": "phase7-finite-envelope/v1", + "graph_count": 54, + "independence_rules": [ + "different_scopes", + "different_datums", + "different_groups", + "disjoint_scope_and_datum", + "disjoint_scope_and_group", + "disjoint_datum_and_group" + ], + "max_exogenous_observations": 16, + "owner_case_count": 30, + "owner_corpus_version": "anonymizer-phase7-owner-contract-corpus/v1", + "reference_model_version": "phase7-reference-model/v1", + "slot_count_case_counts": { + "0": 3, + "1": 27, + "2": 7, + "3": 4, + "4": 3 + } +} diff --git a/tests/engine/execution/phase7_reference_model.py b/tests/engine/execution/phase7_reference_model.py new file mode 100644 index 00000000..cc3222da --- /dev/null +++ b/tests/engine/execution/phase7_reference_model.py @@ -0,0 +1,1533 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pure Phase 7 oracle with no production, pandas, or DataDesigner imports.""" + +from __future__ import annotations + +import hashlib +import json +import re +import unicodedata +from dataclasses import asdict, dataclass, field, replace +from enum import Enum +from itertools import combinations +from typing import cast + +REFERENCE_MODEL_VERSION = "phase7-reference-model/v1" +GENERATOR_VERSION = "phase7-finite-envelope/v1" +MAX_EXOGENOUS_OBSERVATIONS = 16 +EVENT_ALPHABET = ( + "dispatch_accepted", + "dispatch_rejected", + "candidate_rows", + "failed_record", + "backend_exception", + "cancellation", + "trusted_stop", + "loss", + "transform", + "verify", + "finalize", + "cleanup", + "immutable_accept", + "teardown", + "release", +) +INDEPENDENCE_RULES = ( + "different_scopes", + "different_datums", + "different_groups", + "disjoint_scope_and_datum", + "disjoint_scope_and_group", + "disjoint_datum_and_group", +) +OWNER_CORPUS_VERSION = "anonymizer-phase7-owner-contract-corpus/v1" +OWNER_CASE_IDS = ( + "valid_empty_scope_zero_dispatch", + "valid_single_given_name", + "valid_given_family_email_relation", + "valid_phone_source_mask", + "unknown_contract_version", + "contract_digest_mismatch", + "missing_detector_disposition", + "unknown_role", + "unknown_relation", + "unknown_mask", + "unsupported_detector_label", + "selector_resolves_zero_slots", + "selector_resolves_multiple_slots", + "relation_crosses_scopes", + "email_relation_wrong_roles", + "distinct_slots_same_canonical_value", + "candidate_matches_own_original", + "candidate_matches_other_slot_original", + "email_local_part_omits_name", + "count_limits_exact", + "count_limits_one_over", + "byte_limits_exact", + "byte_limits_one_over", + "runtime_capability_missing", + "trusted_task_failure", + "unattributable_failure", + "cleanup_attestation_verified", + "cleanup_attestation_missing", + "cleanup_attestation_contradictory", + "redact_policy_role_bearing_scope", +) +ROLE_BY_LABEL = { + "email": "email_address", + "fax_number": "fax_number", + "first_name": "person_given_name", + "last_name": "person_family_name", + "phone_number": "voice_phone_number", + "user_name": "user_name", +} +ROLE_CONTRACT = { + "email_address": ("email_addr_spec_ascii/v1", "none/v1"), + "fax_number": ("telephone_ascii/v1", "digit_literal/v1"), + "person_family_name": ("unicode_person_name/v1", "none/v1"), + "person_given_name": ("unicode_person_name/v1", "none/v1"), + "user_name": ("username_ascii/v1", "none/v1"), + "voice_phone_number": ("telephone_ascii/v1", "digit_literal/v1"), +} +SLOT_LABELS = ("first_name", "last_name", "email", "phone_number") +SLOT_SOURCES = ("Alice", "Adams", "alice@example.com", "555-0100") +SLOT_CANDIDATES = ("Nova", "Vale", "nova.vale@example.test", "555-0199") + + +class ReferencePolicy(str, Enum): + CURRENT_EMPTY = "phase6-redact-empty/v1" + FUTURE_V1 = "phase6-substitute-role-policy/v1" + + +class ReferenceEventKind(str, Enum): + DISPATCH_ACCEPTED = "dispatch_accepted" + DISPATCH_REJECTED = "dispatch_rejected" + CANDIDATE_ROWS = "candidate_rows" + FAILED_RECORD = "failed_record" + BACKEND_EXCEPTION = "backend_exception" + CANCELLATION = "cancellation" + TRUSTED_STOP = "trusted_stop" + LOSS = "loss" + TRANSFORM = "transform" + VERIFY = "verify" + FINALIZE = "finalize" + CLEANUP = "cleanup" + IMMUTABLE_ACCEPT = "immutable_accept" + TEARDOWN = "teardown" + RELEASE = "release" + + +@dataclass(frozen=True, slots=True) +class ReferenceDatum: + id: str + text: str + + +@dataclass(frozen=True, slots=True) +class ReferenceScope: + members: tuple[str, ...] + + +@dataclass(frozen=True, slots=True) +class ReferenceMention: + datum: str + start: int + end: int + source: str + label: str + cluster: str + + +@dataclass(frozen=True, slots=True) +class ReferenceSelector: + cluster: str + role: str + + +@dataclass(frozen=True, slots=True) +class ReferenceRelation: + version: str + upstream: tuple[ReferenceSelector, ...] + downstream: ReferenceSelector + + +@dataclass(frozen=True, slots=True) +class ReferenceDeclaration: + datums: tuple[ReferenceDatum, ...] + scopes: tuple[ReferenceScope, ...] + mentions: tuple[ReferenceMention, ...] = () + relations: tuple[ReferenceRelation, ...] = () + dependencies: tuple[tuple[str, str], ...] = () + groups: tuple[tuple[str, ...], ...] = () + policy: ReferencePolicy = ReferencePolicy.FUTURE_V1 + capability: bool = True + contract_version: str = "anonymizer-phase7-stable-substitute/v1" + contract_digest_valid: bool = True + detector_universe_complete: bool = True + declared_role: str | None = None + declared_mask: str | None = None + + +@dataclass(frozen=True, slots=True) +class ReferenceEvent: + kind: ReferenceEventKind + subject_kind: str = "invocation" + subject: int = 0 + attempt: int = 0 + assignments: tuple[tuple[str, str], ...] = () + outcome: str = "accepted" + + +@dataclass(frozen=True, slots=True) +class ReferenceCase: + name: str + declaration: ReferenceDeclaration + events: tuple[ReferenceEvent, ...] = field(default_factory=tuple) + owner_case: str | None = None + + +@dataclass(frozen=True, slots=True) +class ReferenceSlot: + key: str + cluster: str + role: str + format: str + mask: str + mention_indexes: tuple[int, ...] + + +@dataclass(frozen=True, slots=True) +class ReferenceManifest: + members: tuple[str, ...] + slots: tuple[ReferenceSlot, ...] + required_pairs: tuple[tuple[str, str], ...] + relations: tuple[tuple[str, tuple[str, ...], str], ...] + + +@dataclass(frozen=True, slots=True) +class ReferenceResult: + admission: str + manifests: tuple[ReferenceManifest, ...] + scope_outcomes: tuple[str, ...] + reason_codes: tuple[str | None, ...] + bundles: tuple[tuple[tuple[str, str], ...], ...] + task_outcomes: tuple[tuple[str, str, str], ...] + outputs: tuple[tuple[str, str], ...] + released_groups: tuple[int, ...] + released_datums: tuple[str, ...] + invocation: str + cleanup: str + immutable_result: bool + dispatch_count: int + attempt_count: int + event_count: int + max_event_count: int + + +def events_commute( + left: ReferenceEvent, + right: ReferenceEvent, + declaration: ReferenceDeclaration | None = None, +) -> bool: + """Return whether two adjacent exogenous observations are independent.""" + if left.subject_kind == "invocation" or right.subject_kind == "invocation": + return False + if left.subject_kind == right.subject_kind: + return left.subject != right.subject + if declaration is None: + return True + scope_members = tuple(set(scope.members) for scope in declaration.scopes) + scoped_datums = {("scope", scope_index): members for scope_index, members in enumerate(scope_members)} + group_members = {("group", group_index): set(group) for group_index, group in enumerate(declaration.groups)} + + def affected(event: ReferenceEvent) -> set[str]: + if event.subject_kind == "scope": + return scoped_datums.get(("scope", event.subject), set()) + if event.subject_kind == "group": + return group_members.get(("group", event.subject), set()) + if event.subject_kind == "datum" and event.subject < len(declaration.datums): + return {declaration.datums[event.subject].id} + return set() + + return affected(left).isdisjoint(affected(right)) + + +def canonical_events( + events: tuple[ReferenceEvent, ...], + declaration: ReferenceDeclaration | None = None, +) -> tuple[ReferenceEvent, ...]: + """Order only adjacent commuting observations by opaque declaration position.""" + rank = {value: index for index, value in enumerate(EVENT_ALPHABET)} + canonical = list(events) + changed = True + while changed: + changed = False + for index in range(len(canonical) - 1): + left = canonical[index] + right = canonical[index + 1] + left_key = (left.subject_kind, left.subject, rank[left.kind.value], left.attempt, left.outcome) + right_key = (right.subject_kind, right.subject, rank[right.kind.value], right.attempt, right.outcome) + if events_commute(left, right, declaration) and left_key > right_key: + canonical[index], canonical[index + 1] = right, left + changed = True + return tuple(canonical) + + +def reduce_reference(case: ReferenceCase) -> ReferenceResult: + """Derive the bounded Phase 7 result without accepting a production verdict.""" + if len(case.events) > MAX_EXOGENOUS_OBSERVATIONS: + raise AssertionError("trace exceeds the frozen 16-observation bound") + if any(not isinstance(event, ReferenceEvent) for event in case.events): + raise AssertionError("trace contains an invalid observation") + compiled = _compile_reference(case.declaration) + if isinstance(compiled, str): + return ReferenceResult( + compiled, + (), + (), + (), + (), + (), + (), + (), + (), + "not_opened", + "not_entered", + False, + 0, + 0, + len(case.events), + MAX_EXOGENOUS_OBSERVATIONS, + ) + manifests, mention_slots = compiled + groups = case.declaration.groups or tuple((datum.id,) for datum in case.declaration.datums) + scope_outcomes = [ + "planned" + if not manifest.slots + else "blocked" + if case.declaration.policy is ReferencePolicy.CURRENT_EMPTY + else "reserved" + for manifest in manifests + ] + reasons: list[str | None] = ["prerequisite_blocked" if outcome == "blocked" else None for outcome in scope_outcomes] + bundles: list[tuple[tuple[str, str], ...]] = [() for _manifest in manifests] + dispatched = [False for _manifest in manifests] + attempts = [-1 for _manifest in manifests] + transformed: dict[str, str] = {} + transform_failed: set[str] = set() + verified_groups: set[int] = set() + dispatch_count = 0 + attempt_count = 0 + cancellation = False + cancellation_after_release = False + global_inconsistent = False + loss = False + finalized = False + cleanup = "unconfirmed" + immutable_result = False + release_observed = False + teardown = "unconfirmed" + + for event in case.events: + if event.kind is ReferenceEventKind.CANCELLATION: + if release_observed and immutable_result: + cancellation_after_release = True + continue + cancellation = True + for index, outcome in enumerate(scope_outcomes): + if outcome == "reserved": + scope_outcomes[index] = "cancelling" if dispatched[index] else "cancelled" + continue + if event.kind in { + ReferenceEventKind.DISPATCH_ACCEPTED, + ReferenceEventKind.DISPATCH_REJECTED, + ReferenceEventKind.CANDIDATE_ROWS, + ReferenceEventKind.FAILED_RECORD, + ReferenceEventKind.BACKEND_EXCEPTION, + ReferenceEventKind.TRUSTED_STOP, + ReferenceEventKind.LOSS, + }: + if event.subject_kind != "scope" or not 0 <= event.subject < len(manifests): + global_inconsistent = True + continue + scope_index = event.subject + state = scope_outcomes[scope_index] + if event.kind is ReferenceEventKind.DISPATCH_ACCEPTED: + if state == "reserved" and not dispatched[scope_index]: + dispatched[scope_index] = True + attempts[scope_index] = event.attempt + dispatch_count += 1 + attempt_count += 1 + else: + global_inconsistent = True + elif event.kind is ReferenceEventKind.DISPATCH_REJECTED: + if state == "reserved": + scope_outcomes[scope_index] = "failed" + reasons[scope_index] = "backend_failed" + elif state != "planned": + global_inconsistent = True + elif event.kind is ReferenceEventKind.CANDIDATE_ROWS: + if state == "planned": + continue + if state != "reserved" or not dispatched[scope_index] or event.attempt != attempts[scope_index]: + if state in {"cancelling", "cancelled", "lost", "failed", "inconsistent", "blocked"}: + continue + scope_outcomes[scope_index] = "inconsistent" + reasons[scope_index] = "evidence_unattributable" + global_inconsistent = True + continue + validated = _validate_candidate( + case.declaration, + manifests[scope_index], + event.assignments, + mention_slots, + ) + if isinstance(validated, str): + reasons[scope_index] = validated + if validated in {"partial_bundle", "duplicate_slot", "foreign_slot"}: + scope_outcomes[scope_index] = "inconsistent" + global_inconsistent = True + else: + scope_outcomes[scope_index] = "failed" + else: + scope_outcomes[scope_index] = "planned" + bundles[scope_index] = validated + elif event.kind is ReferenceEventKind.FAILED_RECORD: + if state == "planned": + continue + if ( + state == "reserved" + and dispatched[scope_index] + and event.attempt == attempts[scope_index] + and event.outcome == "attributed" + ): + scope_outcomes[scope_index] = "failed" + reasons[scope_index] = "backend_failed" + else: + scope_outcomes[scope_index] = "inconsistent" + reasons[scope_index] = "evidence_unattributable" + global_inconsistent = True + elif event.kind is ReferenceEventKind.BACKEND_EXCEPTION: + if state == "reserved" and dispatched[scope_index] and event.attempt == attempts[scope_index]: + scope_outcomes[scope_index] = "failed" + reasons[scope_index] = "backend_failed" + elif state != "planned": + global_inconsistent = True + elif event.kind is ReferenceEventKind.TRUSTED_STOP: + if state in {"reserved", "cancelling"} and dispatched[scope_index]: + scope_outcomes[scope_index] = "cancelled" + elif state != "planned": + global_inconsistent = True + elif event.kind is ReferenceEventKind.LOSS: + if state in {"reserved", "cancelling"} and dispatched[scope_index]: + scope_outcomes[scope_index] = "lost" + reasons[scope_index] = "transport_lost" + loss = True + elif state != "planned": + global_inconsistent = True + continue + if event.kind is ReferenceEventKind.TRANSFORM: + if event.subject_kind != "datum" or not 0 <= event.subject < len(case.declaration.datums): + global_inconsistent = True + continue + datum = case.declaration.datums[event.subject] + if event.outcome != "accepted": + transform_failed.add(datum.id) + continue + output = _apply_anchored(case.declaration, manifests, bundles, mention_slots, datum.id) + if output is None: + transform_failed.add(datum.id) + else: + transformed[datum.id] = output + elif event.kind is ReferenceEventKind.VERIFY: + if event.subject_kind != "group" or not 0 <= event.subject < len(groups): + global_inconsistent = True + elif event.outcome == "accepted": + verified_groups.add(event.subject) + elif event.kind is ReferenceEventKind.FINALIZE: + finalized = event.outcome == "accepted" + global_inconsistent = global_inconsistent or not finalized + elif event.kind is ReferenceEventKind.CLEANUP: + cleanup = event.outcome + global_inconsistent = global_inconsistent or cleanup != "verified" + elif event.kind is ReferenceEventKind.IMMUTABLE_ACCEPT: + immutable_result = ( + event.outcome == "accepted" + and all( + outcome in {"planned", "blocked", "failed", "cancelled", "lost", "inconsistent"} + for outcome in scope_outcomes + ) + and finalized + and cleanup == "verified" + and not global_inconsistent + and not cancellation + and not loss + ) + elif event.kind is ReferenceEventKind.TEARDOWN: + teardown = event.outcome + if teardown == "failed" and not immutable_result: + global_inconsistent = True + elif event.kind is ReferenceEventKind.RELEASE: + release_observed = True + + for index, outcome in enumerate(scope_outcomes): + if outcome in {"reserved", "cancelling"}: + scope_outcomes[index] = "lost" if dispatched[index] else "blocked" + reasons[index] = "transport_lost" if dispatched[index] else "prerequisite_blocked" + loss = loss or dispatched[index] + + eligible = _eligible_datums( + case.declaration, + manifests, + tuple(scope_outcomes), + transformed, + transform_failed, + ) + while True: + before = set(eligible) + for group in groups: + if not set(group).issubset(eligible): + eligible.difference_update(group) + for prerequisite, dependent in case.declaration.dependencies: + if prerequisite not in eligible: + eligible.discard(dependent) + if eligible == before: + break + legal_groups = tuple( + index for index, group in enumerate(groups) if set(group).issubset(eligible) and index in verified_groups + ) + released_groups = legal_groups if immutable_result and release_observed else () + released_datums = tuple(datum_id for group_index in released_groups for datum_id in groups[group_index]) + outputs = tuple( + (datum.id, transformed.get(datum.id, datum.text)) for datum in case.declaration.datums if datum.id in eligible + ) + task_outcomes = tuple( + ("scope", str(index), _phase4_scope_outcome(outcome)) for index, outcome in enumerate(scope_outcomes) + ) + tuple( + ( + "datum", + datum.id, + "succeeded" if datum.id in eligible else "failed" if datum.id in transform_failed else "blocked", + ) + for datum in case.declaration.datums + ) + if immutable_result: + invocation = "completed" + elif global_inconsistent or cleanup != "verified": + invocation = "inconsistent" + elif loss: + invocation = "lost" + elif cancellation: + invocation = "cancelled" + elif any(outcome == "failed" for outcome in scope_outcomes): + invocation = "failed" + else: + invocation = "completed" + if cancellation_after_release: + invocation = "completed" + return ReferenceResult( + "admitted", + manifests, + tuple(scope_outcomes), + tuple(reasons), + tuple(bundles), + task_outcomes, + outputs, + released_groups, + released_datums, + invocation, + cleanup, + immutable_result, + dispatch_count, + attempt_count, + len(case.events), + MAX_EXOGENOUS_OBSERVATIONS, + ) + + +def _compile_reference( + declaration: ReferenceDeclaration, +) -> tuple[tuple[ReferenceManifest, ...], tuple[str, ...]] | str: + if declaration.contract_version != "anonymizer-phase7-stable-substitute/v1": + return "contract_invalid" + if not declaration.contract_digest_valid: + return "digest_mismatch" + if not declaration.detector_universe_complete: + return "detector_universe_incomplete" + if declaration.declared_role is not None and declaration.declared_role not in ROLE_CONTRACT: + return "unsupported_role" + if declaration.declared_mask is not None and declaration.declared_mask not in {"none/v1", "digit_literal/v1"}: + return "unsupported_mask" + datum_ids = tuple(datum.id for datum in declaration.datums) + if ( + len(datum_ids) > 4 + or len(declaration.scopes) > 2 + or len(declaration.mentions) > 6 + or len({mention.cluster for mention in declaration.mentions}) > 3 + or any(len(datum.text.encode("utf-8")) > 1536 for datum in declaration.datums) + or any(len(mention.source.encode("utf-8")) > 256 for mention in declaration.mentions) + ): + return "limit_exceeded" + scopes = declaration.scopes + if any(not scope.members for scope in scopes): + return "empty_scope" + normalized_scopes = tuple(tuple(sorted(scope.members)) for scope in scopes) + if len(set(normalized_scopes)) != len(normalized_scopes): + return "duplicate_scope" + if any(len(set(scope.members)) != len(scope.members) for scope in scopes): + return "duplicate_scope_member" + members = tuple(member for scope in scopes for member in scope.members) + if any(member not in set(datum_ids) for member in members): + return "unknown_scope_datum" + if set(members) != set(datum_ids): + return "scope_coverage_gap" + if len(members) != len(set(members)): + scope_sets = tuple(set(scope.members) for scope in scopes) + if any(left < right or right < left for left, right in combinations(scope_sets, 2)): + return "unsupported_scope_nesting" + return "scope_overlap" + groups = declaration.groups or tuple((datum_id,) for datum_id in datum_ids) + group_members = tuple(member for group in groups for member in group) + if set(group_members) != set(datum_ids) or len(group_members) != len(set(group_members)): + return "malformed_graph" + if any( + prerequisite not in set(datum_ids) or dependent not in set(datum_ids) or prerequisite == dependent + for prerequisite, dependent in declaration.dependencies + ): + return "malformed_graph" + datum_by_id = {datum.id: datum for datum in declaration.datums} + scope_by_datum = {datum_id: scope_index for scope_index, scope in enumerate(scopes) for datum_id in scope.members} + structural_slots: dict[tuple[object, ...], list[int]] = {} + for mention_index, mention in enumerate(declaration.mentions): + datum = datum_by_id.get(mention.datum) + if datum is None: + return "phase6_handoff_mismatch" + if ( + mention.start < 0 + or mention.end <= mention.start + or mention.end > len(datum.text) + or datum.text[mention.start : mention.end] != mention.source + ): + return "phase6_handoff_mismatch" + role = ROLE_BY_LABEL.get(mention.label) + if role is None: + return "unsupported_label" + scope_index = scope_by_datum[mention.datum] + structural_key = (scope_index, mention.cluster, role) + structural_slots.setdefault(structural_key, []).append(mention_index) + if len(structural_slots) > 4: + return "limit_exceeded" + slots_by_scope: list[list[ReferenceSlot]] = [[] for _scope in scopes] + mention_slots = ["" for _mention in declaration.mentions] + for scope_index in range(len(scopes)): + keys = tuple(key for key in structural_slots if key[0] == scope_index) + for slot_index, key in enumerate(keys): + _scope, cluster, role = key + mention_indexes = tuple(structural_slots[key]) + slot_key = f"slot-{scope_index}-{slot_index}" + format_name, mask = ROLE_CONTRACT[cast(str, role)] + slot = ReferenceSlot( + slot_key, + cast(str, cluster), + cast(str, role), + format_name, + mask, + mention_indexes, + ) + slots_by_scope[scope_index].append(slot) + for mention_index in mention_indexes: + mention_slots[mention_index] = slot_key + manifests: list[ReferenceManifest] = [] + for scope_index, scope in enumerate(scopes): + slots = tuple(slots_by_scope[scope_index]) + required_pairs = tuple((left.key, right.key) for left, right in combinations(slots, 2)) + if len(required_pairs) > 6: + return "limit_exceeded" + manifests.append(ReferenceManifest(tuple(sorted(scope.members)), slots, required_pairs, ())) + compiled_relations: list[list[tuple[str, tuple[str, ...], str]]] = [[] for _scope in scopes] + all_slots = tuple(slot for slots in slots_by_scope for slot in slots) + for relation in declaration.relations: + if relation.version != "email_from_name/v1": + return "unsupported_constraint" + if len(set(relation.upstream)) != len(relation.upstream): + return "selector_ambiguous" + + def resolve(selector: ReferenceSelector) -> tuple[ReferenceSlot, ...]: + return tuple(slot for slot in all_slots if slot.cluster == selector.cluster and slot.role == selector.role) + + upstream_matches = tuple(resolve(selector) for selector in relation.upstream) + downstream_matches = resolve(relation.downstream) + if any(not matches for matches in upstream_matches) or not downstream_matches: + return "selector_missing" + if any(len(matches) != 1 for matches in upstream_matches) or len(downstream_matches) != 1: + return "selector_ambiguous" + upstream = tuple(matches[0] for matches in upstream_matches) + downstream = downstream_matches[0] + relation_scopes = {int(slot.key.split("-", maxsplit=2)[1]) for slot in (*upstream, downstream)} + if len(relation_scopes) != 1: + return "cross_scope_relation" + if ( + not 1 <= len(upstream) <= 2 + or any(slot.role not in {"person_given_name", "person_family_name"} for slot in upstream) + or downstream.role != "email_address" + ): + return "relation_role_mismatch" + scope_index = relation_scopes.pop() + compiled_relations[scope_index].append((relation.version, tuple(slot.key for slot in upstream), downstream.key)) + manifests = [ + replace(manifest, relations=tuple(compiled_relations[index])) for index, manifest in enumerate(manifests) + ] + if not declaration.capability: + return "missing_capability" + return tuple(manifests), tuple(mention_slots) + + +def _validate_candidate( + declaration: ReferenceDeclaration, + manifest: ReferenceManifest, + assignments: tuple[tuple[str, str], ...], + mention_slots: tuple[str, ...], +) -> tuple[tuple[str, str], ...] | str: + expected = {slot.key for slot in manifest.slots} + keys = tuple(key for key, _value in assignments) + if len(keys) != len(set(keys)): + return "duplicate_slot" + if set(keys) != expected: + return "foreign_slot" if set(keys) - expected else "partial_bundle" + assignment_by_slot = dict(assignments) + canonical: dict[str, str] = {} + originals = tuple( + mention.source + for mention_index, mention in enumerate(declaration.mentions) + if mention_slots[mention_index] in expected + ) + original_skeletons = {_canonical_value(original) for original in originals} + for slot in manifest.slots: + value = assignment_by_slot[slot.key] + skeleton = _canonical_value(value) + if not value or not skeleton or skeleton in original_skeletons: + return "candidate_matches_original" + if len(value.encode("utf-8")) > 256 or not _format_valid(slot.format, value): + return "unsupported_role" + if slot.mask == "digit_literal/v1" and any( + not _digit_mask_valid(declaration.mentions[index].source, value) for index in slot.mention_indexes + ): + return "relation_failed" + canonical[slot.key] = skeleton + for left, right in manifest.required_pairs: + if canonical[left] == canonical[right]: + return "canonical_collision" + for version, upstream, downstream in manifest.relations: + if version != "email_from_name/v1": + return "unsupported_constraint" + local = assignment_by_slot[downstream].split("@", maxsplit=1)[0] + local_skeleton = _canonical_value(local) + if not any(canonical[slot_key] in local_skeleton for slot_key in upstream): + return "relation_failed" + return tuple(sorted(assignments)) + + +def _canonical_value(value: str) -> str: + normalized = unicodedata.normalize("NFKC", value).strip().casefold() + return "".join(character for character in normalized if unicodedata.category(character)[0] in {"L", "N"}) + + +def _format_valid(format_name: str, value: str) -> bool: + if format_name == "unicode_person_name/v1": + normalized = unicodedata.normalize("NFKC", value) + return ( + 1 <= len(normalized.encode("utf-8")) <= 128 + and any(unicodedata.category(character).startswith("L") for character in normalized) + and all( + unicodedata.category(character)[:1] in {"L", "M"} + or unicodedata.category(character) == "Zs" + or character in "'.-" + for character in normalized + ) + ) + if format_name == "username_ascii/v1": + return bool(re.fullmatch(r"[A-Za-z0-9](?:[A-Za-z0-9._-]{0,62}[A-Za-z0-9])?", value)) + if format_name == "telephone_ascii/v1": + return ( + bool(re.fullmatch(r"[0-9 ()+.-]+", value)) + and 7 <= sum(character.isascii() and character.isdigit() for character in value) <= 15 + and value.count("+") <= 1 + and ("+" not in value or value.startswith("+")) + ) + if format_name == "email_addr_spec_ascii/v1": + if len(value.encode("utf-8")) > 254 or value.count("@") != 1 or not value.isascii(): + return False + local, domain = value.split("@") + labels = domain.split(".") + return ( + 1 <= len(local) <= 64 + and not local.startswith(".") + and not local.endswith(".") + and ".." not in local + and bool(re.fullmatch(r"[A-Za-z0-9.!#$%&'*+/=?^_`{|}~-]+", local)) + and len(labels) >= 2 + and all( + 1 <= len(label) <= 63 and bool(re.fullmatch(r"[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?", label)) + for label in labels + ) + and 2 <= len(labels[-1]) <= 63 + and labels[-1].isalpha() + ) + return False + + +def _digit_mask_valid(source: str, candidate: str) -> bool: + normalized_source = unicodedata.normalize("NFKC", source) + normalized_candidate = unicodedata.normalize("NFKC", candidate) + return len(normalized_source) == len(normalized_candidate) and all( + (candidate_character.isascii() and candidate_character.isdigit()) + if source_character.isascii() and source_character.isdigit() + else source_character == candidate_character + for source_character, candidate_character in zip(normalized_source, normalized_candidate, strict=True) + ) + + +def _apply_anchored( + declaration: ReferenceDeclaration, + manifests: tuple[ReferenceManifest, ...], + bundles: list[tuple[tuple[str, str], ...]], + mention_slots: tuple[str, ...], + datum_id: str, +) -> str | None: + datum = next(datum for datum in declaration.datums if datum.id == datum_id) + assignment_by_slot = {key: value for manifest_bundle in bundles for key, value in manifest_bundle} + mentions = tuple( + (mention, mention_slots[index]) + for index, mention in enumerate(declaration.mentions) + if mention.datum == datum_id + ) + if not mentions: + return datum.text + if any(slot_key not in assignment_by_slot for _mention, slot_key in mentions): + return None + ordered = sorted(mentions, key=lambda item: item[0].start) + if any(left[0].end > right[0].start for left, right in zip(ordered, ordered[1:], strict=False)): + return None + output = datum.text + for mention, slot_key in reversed(ordered): + if output[mention.start : mention.end] != mention.source: + return None + output = output[: mention.start] + assignment_by_slot[slot_key] + output[mention.end :] + return output + + +def _eligible_datums( + declaration: ReferenceDeclaration, + manifests: tuple[ReferenceManifest, ...], + scope_outcomes: tuple[str, ...], + transformed: dict[str, str], + transform_failed: set[str], +) -> set[str]: + scope_by_datum = { + datum_id: scope_index for scope_index, manifest in enumerate(manifests) for datum_id in manifest.members + } + mentioned = {mention.datum for mention in declaration.mentions} + return { + datum.id + for datum in declaration.datums + if scope_outcomes[scope_by_datum[datum.id]] == "planned" + and datum.id not in transform_failed + and (datum.id not in mentioned or datum.id in transformed) + } + + +def _phase4_scope_outcome(outcome: str) -> str: + return { + "planned": "succeeded", + "blocked": "blocked", + "failed": "failed", + "cancelled": "cancelled", + "lost": "lost", + "inconsistent": "inconsistent", + }[outcome] + + +def finite_reference_cases() -> tuple[ReferenceCase, ...]: + """Return the frozen governed corpus, not an unbounded Cartesian product.""" + return ( + *_owner_contract_cases(), + *_slot_envelope_cases(), + *_admission_cases(), + *_lifecycle_cases(), + ) + + +def case_by_name(name: str) -> ReferenceCase: + return next(case for case in finite_reference_cases() if case.name == name) + + +def corpus_document() -> dict[str, object]: + cases = tuple(sorted(finite_reference_cases(), key=lambda case: case.name)) + return { + "cases": [_canonical_case_record(case) for case in cases], + "generator_version": GENERATOR_VERSION, + "reference_model_version": REFERENCE_MODEL_VERSION, + "schema_version": "phase7-reference-corpus/v1", + } + + +def _canonical_case_record(case: ReferenceCase) -> dict[str, object]: + canonical = replace(case, events=canonical_events(case.events, case.declaration)) + return {"case": asdict(canonical), "result": asdict(reduce_reference(canonical))} + + +def canonical_corpus_bytes() -> bytes: + """Serialize the complete corpus as compact sorted-key UTF-8 JSON.""" + return json.dumps( + corpus_document(), + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + + +def reference_manifest() -> dict[str, object]: + cases = finite_reference_cases() + graph_payloads = { + json.dumps(asdict(case.declaration), ensure_ascii=False, sort_keys=True, separators=(",", ":")) + for case in cases + } + trace_payloads = { + json.dumps( + { + "declaration": asdict(case.declaration), + "events": [asdict(event) for event in canonical_events(case.events, case.declaration)], + }, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ) + for case in cases + } + slot_count_cases = { + str(slot_count): sum( + len(result.manifests) == 1 and sum(len(manifest.slots) for manifest in result.manifests) == slot_count + for result in (reduce_reference(case) for case in cases) + ) + for slot_count in range(5) + } + return { + "actual_event_count": sum(len(case.events) for case in cases), + "canonical_serialization": "UTF8_compact_sorted_key_JSON_complete_corpus_no_trailing_newline", + "canonical_trace_count": len(trace_payloads), + "case_count": len(cases), + "digest": hashlib.sha256(canonical_corpus_bytes()).hexdigest(), + "event_alphabet": list(EVENT_ALPHABET), + "generator_version": GENERATOR_VERSION, + "graph_count": len(graph_payloads), + "independence_rules": list(INDEPENDENCE_RULES), + "max_exogenous_observations": MAX_EXOGENOUS_OBSERVATIONS, + "owner_case_count": sum(case.owner_case is not None for case in cases), + "owner_corpus_version": OWNER_CORPUS_VERSION, + "reference_model_version": REFERENCE_MODEL_VERSION, + "slot_count_case_counts": slot_count_cases, + } + + +def owner_case_outcome(case: ReferenceCase) -> str: + """Project a derived result into the frozen owner's 30-case vocabulary.""" + result = reduce_reference(case) + if result.admission != "admitted": + return result.admission + if any( + reason in {"canonical_collision", "candidate_matches_original", "relation_failed"} + for reason in result.reason_codes + ): + return next(reason for reason in result.reason_codes if reason is not None) + if result.invocation == "inconsistent": + return "inconsistent_global_embargo" + if result.scope_outcomes and all(outcome == "blocked" for outcome in result.scope_outcomes): + return "blocked_zero_effects" + if any(outcome == "failed" for outcome in result.scope_outcomes): + return "failed" + if result.manifests and all(not manifest.slots for manifest in result.manifests): + return "planned_empty" + if result.released_groups: + return "release_eligible" if case.owner_case == "cleanup_attestation_verified" else "planned" + return "planned" + + +def _owner_contract_cases() -> tuple[ReferenceCase, ...]: + empty = _declaration_for_labels(((),)) + one = _declaration_for_labels((("first_name",),)) + two_names = _declaration_for_labels((("first_name", "last_name"),)) + relation = _relation_declaration() + phone = _declaration_for_labels((("phone_number",),)) + four = _declaration_for_labels((SLOT_LABELS,)) + five = _declaration_for_labels(((*SLOT_LABELS, "user_name"),)) + + def owner(case_id: str, declaration: ReferenceDeclaration, events: tuple[ReferenceEvent, ...]) -> ReferenceCase: + return ReferenceCase(f"owner-{case_id}", declaration, events, case_id) + + invalid_label = replace( + one, + mentions=(replace(one.mentions[0], label="account_number"),), + ) + missing_selector = replace( + relation, + relations=( + ReferenceRelation( + "email_from_name/v1", + (ReferenceSelector("c0", "user_name"),), + ReferenceSelector("c0", "email_address"), + ), + ), + ) + duplicate_selector = replace( + relation, + relations=( + ReferenceRelation( + "email_from_name/v1", + ( + ReferenceSelector("c0", "person_given_name"), + ReferenceSelector("c0", "person_given_name"), + ), + ReferenceSelector("c0", "email_address"), + ), + ), + ) + cross_scope = _declaration_for_labels((("first_name",), ("email",))) + cross_scope = replace( + cross_scope, + relations=( + ReferenceRelation( + "email_from_name/v1", + (ReferenceSelector("c0", "person_given_name"),), + ReferenceSelector("c1", "email_address"), + ), + ), + ) + wrong_roles = replace( + relation, + relations=( + ReferenceRelation( + "email_from_name/v1", + (ReferenceSelector("c0", "email_address"),), + ReferenceSelector("c0", "person_given_name"), + ), + ), + ) + exact_bytes = _single_source_declaration("A" * 256) + over_bytes = _single_source_declaration("A" * 257) + success_one = _success_events(one) + return ( + owner("valid_empty_scope_zero_dispatch", empty, _success_events(empty)), + owner("valid_single_given_name", one, success_one), + owner("valid_given_family_email_relation", relation, _success_events(relation)), + owner("valid_phone_source_mask", phone, _success_events(phone)), + owner("unknown_contract_version", replace(one, contract_version="phase7/v2"), ()), + owner("contract_digest_mismatch", replace(one, contract_digest_valid=False), ()), + owner("missing_detector_disposition", replace(one, detector_universe_complete=False), ()), + owner("unknown_role", replace(one, declared_role="unknown_role"), ()), + owner( + "unknown_relation", + replace( + relation, + relations=(replace(relation.relations[0], version="unknown_relation/v1"),), + ), + (), + ), + owner("unknown_mask", replace(one, declared_mask="unknown_mask/v1"), ()), + owner("unsupported_detector_label", invalid_label, ()), + owner("selector_resolves_zero_slots", missing_selector, ()), + owner("selector_resolves_multiple_slots", duplicate_selector, ()), + owner("relation_crosses_scopes", cross_scope, ()), + owner("email_relation_wrong_roles", wrong_roles, ()), + owner( + "distinct_slots_same_canonical_value", + two_names, + _success_events(two_names, {"slot-0-0": "Nova", "slot-0-1": "Ｎｏｖａ"}), + ), + owner( + "candidate_matches_own_original", + one, + _success_events(one, {"slot-0-0": " Alice "}), + ), + owner( + "candidate_matches_other_slot_original", + two_names, + _success_events(two_names, {"slot-0-0": "Adams", "slot-0-1": "Vale"}), + ), + owner( + "email_local_part_omits_name", + relation, + _success_events( + relation, + { + "slot-0-0": "Nova", + "slot-0-1": "Vale", + "slot-0-2": "other@example.test", + }, + ), + ), + owner("count_limits_exact", four, _success_events(four)), + owner("count_limits_one_over", five, ()), + owner("byte_limits_exact", exact_bytes, _success_events(exact_bytes)), + owner("byte_limits_one_over", over_bytes, ()), + owner("runtime_capability_missing", replace(one, capability=False), ()), + owner( + "trusted_task_failure", + one, + ( + ReferenceEvent(ReferenceEventKind.DISPATCH_ACCEPTED, "scope", 0), + ReferenceEvent(ReferenceEventKind.FAILED_RECORD, "scope", 0, outcome="attributed"), + *_failed_tail(), + ), + ), + owner( + "unattributable_failure", + one, + ( + ReferenceEvent(ReferenceEventKind.DISPATCH_ACCEPTED, "scope", 0), + ReferenceEvent(ReferenceEventKind.FAILED_RECORD, "scope", 0, attempt=1, outcome="foreign"), + *_failed_tail(), + ), + ), + owner("cleanup_attestation_verified", one, success_one), + owner( + "cleanup_attestation_missing", + one, + tuple(event for event in success_one if event.kind is not ReferenceEventKind.CLEANUP), + ), + owner( + "cleanup_attestation_contradictory", + one, + tuple( + replace(event, outcome="contradictory") if event.kind is ReferenceEventKind.CLEANUP else event + for event in success_one + ), + ), + owner( + "redact_policy_role_bearing_scope", + replace(one, policy=ReferencePolicy.CURRENT_EMPTY), + _failed_tail(), + ), + ) + + +def _slot_envelope_cases() -> tuple[ReferenceCase, ...]: + cases: list[ReferenceCase] = [] + for slot_count in range(5): + declaration = _declaration_for_labels((SLOT_LABELS[:slot_count],)) + cases.append( + ReferenceCase( + f"future-slots-{slot_count}", + declaration, + _success_events(declaration), + ) + ) + current = replace(declaration, policy=ReferencePolicy.CURRENT_EMPTY) + cases.append( + ReferenceCase( + f"current-empty-policy-slots-{slot_count}", + current, + _success_events(current) if slot_count == 0 else _failed_tail(), + ) + ) + for total_slots in range(5): + for left_slots in range(total_slots + 1): + right_slots = total_slots - left_slots + declaration = _declaration_for_labels((SLOT_LABELS[:left_slots], SLOT_LABELS[:right_slots])) + cases.append( + ReferenceCase( + f"independent-scopes-{left_slots}-{right_slots}", + declaration, + _success_events(declaration), + ) + ) + equal_text = _equal_text_distinct_clusters_declaration() + cases.append(ReferenceCase("equal-text-distinct-clusters", equal_text, _success_events(equal_text))) + shared = _shared_slot_declaration() + cases.append(ReferenceCase("shared-slot-reuse", shared, _success_events(shared))) + return tuple(cases) + + +def _admission_cases() -> tuple[ReferenceCase, ...]: + datums = tuple(ReferenceDatum(f"d{index}", f"plain-{index}") for index in range(4)) + base = ReferenceDeclaration( + datums, + (ReferenceScope(("d0", "d1")), ReferenceScope(("d2", "d3"))), + groups=tuple((datum.id,) for datum in datums), + ) + first, second, third, fourth = (datum.id for datum in base.datums) + cases = { + "empty-scope": replace( + base, + scopes=(ReferenceScope(()), ReferenceScope((first, second, third, fourth))), + ), + "duplicate-scope": replace( + base, + scopes=( + ReferenceScope((first, second, third, fourth)), + ReferenceScope((fourth, third, second, first)), + ), + ), + "duplicate-member": replace( + base, + scopes=(ReferenceScope((first, first, second)), ReferenceScope((third, fourth))), + ), + "unknown-datum": replace( + base, + scopes=( + ReferenceScope((first, second, third)), + ReferenceScope(("foreign",)), + ), + ), + "coverage-gap": replace(base, scopes=(ReferenceScope((first, second, third)),)), + "overlap": replace( + base, + scopes=( + ReferenceScope((first, second)), + ReferenceScope((second, third, fourth)), + ), + ), + "nesting": replace( + base, + scopes=( + ReferenceScope((first, second, third, fourth)), + ReferenceScope((first, second)), + ), + ), + } + return tuple(ReferenceCase(f"admission-{name}", declaration) for name, declaration in cases.items()) + + +def _lifecycle_cases() -> tuple[ReferenceCase, ...]: + one = _declaration_for_labels((("first_name",),)) + success = _success_events(one) + dispatch = ReferenceEvent(ReferenceEventKind.DISPATCH_ACCEPTED, "scope", 0) + candidate = next(event for event in success if event.kind is ReferenceEventKind.CANDIDATE_ROWS) + tail = tuple( + event + for event in success + if event.kind not in {ReferenceEventKind.DISPATCH_ACCEPTED, ReferenceEventKind.CANDIDATE_ROWS} + ) + two_datums = _declaration_for_labels((("first_name",), ("first_name",))) + grouped = replace(two_datums, groups=(("d0", "d1"),)) + dependent = replace(two_datums, dependencies=(("d0", "d1"),)) + cascading = _cascading_declaration() + cases = ( + ReferenceCase( + "dispatch-rejected", + one, + (ReferenceEvent(ReferenceEventKind.DISPATCH_REJECTED, "scope", 0), *_failed_tail()), + ), + ReferenceCase( + "backend-exception", + one, + ( + dispatch, + ReferenceEvent(ReferenceEventKind.BACKEND_EXCEPTION, "scope", 0), + *_failed_tail(), + ), + ), + ReferenceCase( + "contradictory-candidate-evidence", + one, + ( + dispatch, + replace(candidate, assignments=(("slot-0-0", "Nova"), ("slot-0-0", "Vale"))), + *_failed_tail(), + ), + ), + ReferenceCase( + "cancel-before-dispatch", one, (ReferenceEvent(ReferenceEventKind.CANCELLATION), *_failed_tail()) + ), + ReferenceCase( + "dispatch-cancel-without-stop", + one, + (dispatch, ReferenceEvent(ReferenceEventKind.CANCELLATION), *_failed_tail()), + ), + ReferenceCase( + "dispatch-cancel-trusted-stop", + one, + ( + dispatch, + ReferenceEvent(ReferenceEventKind.CANCELLATION), + ReferenceEvent(ReferenceEventKind.TRUSTED_STOP, "scope", 0), + *_failed_tail(), + ), + ), + ReferenceCase( + "late-candidate-after-stop", + one, + ( + dispatch, + ReferenceEvent(ReferenceEventKind.CANCELLATION), + ReferenceEvent(ReferenceEventKind.TRUSTED_STOP, "scope", 0), + candidate, + *_failed_tail(), + ), + ), + ReferenceCase( + "late-candidate-after-loss", + one, + (dispatch, ReferenceEvent(ReferenceEventKind.LOSS, "scope", 0), candidate, *_failed_tail()), + ), + ReferenceCase( + "foreign-candidate-before-acceptance", + one, + (dispatch, replace(candidate, attempt=1), *_failed_tail()), + ), + ReferenceCase( + "partial-candidate", + _declaration_for_labels((("first_name", "last_name"),)), + _partial_candidate_events(), + ), + ReferenceCase( + "planned-then-foreign-is-absorbing", + one, + (dispatch, candidate, replace(candidate, attempt=1), *tail), + ), + ReferenceCase( + "finalization-failure", + one, + tuple( + replace(event, outcome="failed") if event.kind is ReferenceEventKind.FINALIZE else event + for event in success + ), + ), + ReferenceCase( + "cleanup-failure", + one, + tuple( + replace(event, outcome="failed") if event.kind is ReferenceEventKind.CLEANUP else event + for event in success + ), + ), + ReferenceCase( + "teardown-failure-after-acceptance", + one, + tuple( + replace(event, outcome="failed") if event.kind is ReferenceEventKind.TEARDOWN else event + for event in success + ), + ), + ReferenceCase( + "atomic-group-member-failure", + grouped, + _success_events(grouped, transform_failures={"d1"}), + ), + ReferenceCase( + "dependent-datum-withheld", + dependent, + _success_events(dependent, transform_failures={"d0"}), + ), + ReferenceCase( + "independent-scope-local-failure", + two_datums, + _success_events(two_datums, transform_failures={"d0"}), + ), + ReferenceCase( + "anchored-non-cascading-application", + cascading, + _success_events(cascading, {"slot-0-0": "Nova Blake", "slot-0-1": "Vale"}), + ), + ReferenceCase( + "release-then-cancel-is-absorbing", + one, + (*success, ReferenceEvent(ReferenceEventKind.CANCELLATION)), + ), + ) + return cases + + +def _declaration_for_labels( + labels_by_scope: tuple[tuple[str, ...], ...], +) -> ReferenceDeclaration: + datums: list[ReferenceDatum] = [] + scopes: list[ReferenceScope] = [] + mentions: list[ReferenceMention] = [] + source_sets = ( + { + "first_name": "Alice", + "last_name": "Adams", + "email": "alice@example.com", + "phone_number": "555-0100", + "user_name": "alice_1", + }, + { + "first_name": "Bob", + "last_name": "Stone", + "email": "bob@example.com", + "phone_number": "555-0200", + "user_name": "bob_2", + }, + ) + for scope_index, labels in enumerate(labels_by_scope): + datum_id = f"d{scope_index}" + sources = tuple(source_sets[scope_index][label] for label in labels) + text = " ".join(sources) if sources else f"plain-{scope_index}" + datums.append(ReferenceDatum(datum_id, text)) + scopes.append(ReferenceScope((datum_id,))) + cursor = 0 + for label, source in zip(labels, sources, strict=True): + start = text.index(source, cursor) + mentions.append(ReferenceMention(datum_id, start, start + len(source), source, label, f"c{scope_index}")) + cursor = start + len(source) + return ReferenceDeclaration( + tuple(datums), + tuple(scopes), + tuple(mentions), + groups=tuple((datum.id,) for datum in datums), + ) + + +def _relation_declaration() -> ReferenceDeclaration: + declaration = _declaration_for_labels((("first_name", "last_name", "email"),)) + return replace( + declaration, + relations=( + ReferenceRelation( + "email_from_name/v1", + ( + ReferenceSelector("c0", "person_given_name"), + ReferenceSelector("c0", "person_family_name"), + ), + ReferenceSelector("c0", "email_address"), + ), + ), + ) + + +def _single_source_declaration(source: str) -> ReferenceDeclaration: + datum = ReferenceDatum("d0", source) + mention = ReferenceMention("d0", 0, len(source), source, "first_name", "c0") + return ReferenceDeclaration((datum,), (ReferenceScope(("d0",)),), (mention,), groups=(("d0",),)) + + +def _equal_text_distinct_clusters_declaration() -> ReferenceDeclaration: + datum = ReferenceDatum("d0", "Alice Alice") + return ReferenceDeclaration( + (datum,), + (ReferenceScope(("d0",)),), + ( + ReferenceMention("d0", 0, 5, "Alice", "first_name", "c0"), + ReferenceMention("d0", 6, 11, "Alice", "first_name", "c1"), + ), + groups=(("d0",),), + ) + + +def _shared_slot_declaration() -> ReferenceDeclaration: + datum = ReferenceDatum("d0", "Alice and Alicia") + return ReferenceDeclaration( + (datum,), + (ReferenceScope(("d0",)),), + ( + ReferenceMention("d0", 0, 5, "Alice", "first_name", "c0"), + ReferenceMention("d0", 10, 16, "Alicia", "first_name", "c0"), + ), + groups=(("d0",),), + ) + + +def _cascading_declaration() -> ReferenceDeclaration: + datum = ReferenceDatum("d0", "Alice met Nova") + return ReferenceDeclaration( + (datum,), + (ReferenceScope(("d0",)),), + ( + ReferenceMention("d0", 0, 5, "Alice", "first_name", "c0"), + ReferenceMention("d0", 10, 14, "Nova", "last_name", "c1"), + ), + groups=(("d0",),), + ) + + +def _success_events( + declaration: ReferenceDeclaration, + assignments: dict[str, str] | None = None, + *, + transform_failures: set[str] | None = None, +) -> tuple[ReferenceEvent, ...]: + compiled = _compile_reference(declaration) + if isinstance(compiled, str): + return () + manifests, _mention_slots = compiled + candidate_values = dict(assignments or {}) + events: list[ReferenceEvent] = [] + for scope_index, manifest in enumerate(manifests): + if not manifest.slots or declaration.policy is ReferencePolicy.CURRENT_EMPTY: + continue + for slot in manifest.slots: + candidate_values.setdefault(slot.key, _candidate_for_slot(slot, scope_index)) + events.append(ReferenceEvent(ReferenceEventKind.DISPATCH_ACCEPTED, "scope", scope_index)) + events.append( + ReferenceEvent( + ReferenceEventKind.CANDIDATE_ROWS, + "scope", + scope_index, + assignments=tuple((slot.key, candidate_values[slot.key]) for slot in manifest.slots), + ) + ) + failures = transform_failures or set() + mentioned = {mention.datum for mention in declaration.mentions} + for datum_index, datum in enumerate(declaration.datums): + if datum.id in mentioned: + events.append( + ReferenceEvent( + ReferenceEventKind.TRANSFORM, + "datum", + datum_index, + outcome="failed" if datum.id in failures else "accepted", + ) + ) + groups = declaration.groups or tuple((datum.id,) for datum in declaration.datums) + events.extend( + ReferenceEvent(ReferenceEventKind.VERIFY, "group", group_index) for group_index, _group in enumerate(groups) + ) + events.extend( + ( + ReferenceEvent(ReferenceEventKind.FINALIZE), + ReferenceEvent(ReferenceEventKind.CLEANUP, outcome="verified"), + ReferenceEvent(ReferenceEventKind.IMMUTABLE_ACCEPT), + ReferenceEvent(ReferenceEventKind.TEARDOWN), + ReferenceEvent(ReferenceEventKind.RELEASE), + ) + ) + if len(events) > MAX_EXOGENOUS_OBSERVATIONS: + raise AssertionError("generated trace exceeds frozen bound") + return tuple(events) + + +def _candidate_for_slot(slot: ReferenceSlot, scope_index: int) -> str: + candidates = { + "person_given_name": ("Nova", "Orion"), + "person_family_name": ("Vale", "Stonebridge"), + "email_address": ("nova.vale@example.test", "orion.stonebridge@example.test"), + "voice_phone_number": ("555-0199", "555-0299"), + "fax_number": ("555-0198", "555-0298"), + "user_name": ("nova_1", "orion_2"), + } + return candidates[slot.role][scope_index] + + +def _failed_tail() -> tuple[ReferenceEvent, ...]: + return ( + ReferenceEvent(ReferenceEventKind.FINALIZE), + ReferenceEvent(ReferenceEventKind.CLEANUP, outcome="verified"), + ReferenceEvent(ReferenceEventKind.IMMUTABLE_ACCEPT), + ReferenceEvent(ReferenceEventKind.TEARDOWN), + ReferenceEvent(ReferenceEventKind.RELEASE), + ) + + +def _partial_candidate_events() -> tuple[ReferenceEvent, ...]: + return ( + ReferenceEvent(ReferenceEventKind.DISPATCH_ACCEPTED, "scope", 0), + ReferenceEvent( + ReferenceEventKind.CANDIDATE_ROWS, + "scope", + 0, + assignments=(("slot-0-0", "Nova"),), + ), + *_failed_tail(), + ) diff --git a/tests/engine/execution/test_compiled_invocation.py b/tests/engine/execution/test_compiled_invocation.py new file mode 100644 index 00000000..b7e961d9 --- /dev/null +++ b/tests/engine/execution/test_compiled_invocation.py @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import pickle +from dataclasses import FrozenInstanceError + +import pytest +from data_designer.config.models import ModelConfig + +from anonymizer.config.anonymizer_config import AnonymizerConfig, Rewrite +from anonymizer.config.models import ModelSelection +from anonymizer.config.replace_strategies import Annotate, Hash, Redact, Substitute +from anonymizer.engine.execution.invocation import _CompiledInvocation + + +@pytest.mark.parametrize("replace", [Redact(), Annotate(), Hash(), Substitute(instructions="keep tone")]) +def test_compiled_replace_invocation_is_immutable_and_preserves_strategy( + replace: Redact | Annotate | Hash | Substitute, stub_slim_model_selection: ModelSelection +) -> None: + compiled = _CompiledInvocation.compile(AnonymizerConfig(replace=replace), stub_slim_model_selection) + + assert compiled.replace_method == replace + assert compiled.rewrite is None + with pytest.raises(FrozenInstanceError): + setattr(compiled, "replace_method", Redact()) + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(compiled) + + +def test_compiled_rewrite_uses_rewrite_evaluation_property( + stub_slim_model_selection: ModelSelection, monkeypatch: pytest.MonkeyPatch +) -> None: + rewrite = Rewrite(max_repair_iterations=2, use_combined_graph=True, strict_entity_protection=True) + expected = rewrite.evaluation + calls = 0 + original = Rewrite.evaluation.fget + assert original is not None + + def evaluation(self: Rewrite): + nonlocal calls + calls += 1 + return original(self) + + monkeypatch.setattr(Rewrite, "evaluation", property(evaluation)) + compiled = _CompiledInvocation.compile(AnonymizerConfig(rewrite=rewrite), stub_slim_model_selection) + + assert calls == 1 + assert compiled.rewrite is not None + assert compiled.rewrite.evaluation == expected + assert compiled.rewrite.use_combined_graph is True + assert compiled.rewrite.strict_entity_protection is True + + +def test_compiled_invocation_detaches_model_config_snapshot( + stub_slim_model_selection: ModelSelection, +) -> None: + source_configs = [ModelConfig(alias="original", model="provider/original", provider="stub")] + + compiled = _CompiledInvocation.compile( + AnonymizerConfig(replace=Redact()), + stub_slim_model_selection, + source_configs, + ) + source_configs[0].model = "provider/mutated" + + assert compiled.model_configs[0].model == "provider/original" diff --git a/tests/engine/execution/test_context_admission.py b/tests/engine/execution/test_context_admission.py new file mode 100644 index 00000000..3b1bb28c --- /dev/null +++ b/tests/engine/execution/test_context_admission.py @@ -0,0 +1,494 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib.util +from collections.abc import Callable +from dataclasses import replace +from typing import cast + +import pytest + +from anonymizer.engine.execution import context_admission +from anonymizer.engine.execution.accounting_admission import _AccountingAdmissionCode +from anonymizer.engine.execution.accounting_plan import _AccountingLimits +from anonymizer.engine.execution.context_admission import ( + _compile_context_plan, + _ContextAdmissionCode, + _ContextPlan, + _ContextRejected, + _is_admitted_context_plan, +) +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _capability_satisfies, + _ContextBackendCapability, + _ContextExecutionContract, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.graph import ( + _AtomicGroup, + _CoherenceScope, + _ContextScope, + _DatumId, + _DatumPurpose, + _ProtectionGraph, + _TextDatum, +) +from anonymizer.engine.execution.graph_runtime import _AccountingGraphRuntime, _FrameExecutionBackend +from anonymizer.engine.execution.protection_service import _GraphRuntimeBackend, _RedactProtectionService + + +def test_graph_declares_separate_target_and_context_only_purposes() -> None: + assert {purpose.value for purpose in _DatumPurpose} == {"target", "context_only"} + + +def test_context_admission_has_a_dedicated_private_module() -> None: + assert importlib.util.find_spec("anonymizer.engine.execution.context_admission") is not None + + +def test_context_contract_and_workframe_modules_are_separate() -> None: + assert importlib.util.find_spec("anonymizer.engine.execution.context_contract") is not None + assert importlib.util.find_spec("anonymizer.engine.execution.context_workframes") is not None + + +def test_context_capability_is_typed_bounded_and_retention_disabled() -> None: + contract, capability = _contract_and_capability() + + assert _capability_satisfies(capability, contract) + assert not _capability_satisfies( + replace(capability, retention=_RetentionPosture.ENABLED), + contract, + ) + assert not _capability_satisfies( + replace( + capability, + limits=replace(capability.limits, max_context_bytes_per_target=3), + ), + contract, + ) + with pytest.raises(TypeError): + contract.__reduce__() + + +def test_preflight_rejects_a_raising_or_malformed_capability_snapshot() -> None: + contract, capability = _contract_and_capability() + + class _Runtime: + def __init__(self, snapshot: object) -> None: + self.snapshot = snapshot + + def context_capability(self) -> object: + if isinstance(self.snapshot, BaseException): + raise self.snapshot + return self.snapshot + + for snapshot in (RuntimeError("backend unavailable"), object(), capability): + service = _RedactProtectionService( + cast( + _GraphRuntimeBackend, + _AccountingGraphRuntime(cast(_FrameExecutionBackend, _Runtime(snapshot))), + ) + ) + result = service.admit_context(_context_graph(), accounting_limits=_ACCOUNTING_LIMITS, contract=contract) + if snapshot is capability: + assert isinstance(result, _ContextPlan) + else: + assert result == _ContextRejected(_ContextAdmissionCode.BACKEND_INCOMPATIBLE) + + +@pytest.mark.parametrize( + "invalid_limits", + [ + _ContextLimits(True, 32, 4, 128), + _ContextLimits(2, -1, 4, 128), + _ContextLimits(2, 32, cast(int, 4.0), 128), + ], +) +def test_context_capability_rejects_non_integer_or_negative_limits( + invalid_limits: _ContextLimits, +) -> None: + contract, capability = _contract_and_capability() + + assert not _capability_satisfies(replace(capability, limits=invalid_limits), contract) + + +def test_context_compiler_contract_is_available_before_behavior_tests() -> None: + assert callable(getattr(context_admission, "_compile_context_plan", None)) + + +def test_context_compiler_detaches_targets_context_and_ordered_bindings() -> None: + graph = _context_graph() + contract, capability = _contract_and_capability() + + result = _compile_context_plan( + graph, + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=capability, + ) + + assert isinstance(result, _ContextPlan) + assert _is_admitted_context_plan(result) + assert tuple(datum.id.value for datum in result.accounting.datums) == ("target-a", "target-b") + assert tuple(datum.id.value for datum in result.context_only_datums) == ("context-c",) + assert tuple( + tuple(binding.datum_id.value for binding in projection.bindings) for projection in result.projections + ) == (("context-c", "target-b"), ("target-a",)) + assert tuple(tuple(binding.ordinal for binding in projection.bindings) for projection in result.projections) == ( + (0, 1), + (0,), + ) + assert result.projections[0].bindings[0].scope is result.projections[0].scope + assert result.projections[0].scope is not result.projections[1].scope + + object.__setattr__(graph.datums[2], "text", "mutated-context") + object.__setattr__(graph.datums[0], "text", "mutated-target") + assert result.context_only_datums[0].text == "gamma" + assert result.accounting.datums[0].text == "alpha" + assert _is_admitted_context_plan(result) + + +def test_scope_declaration_order_does_not_change_compiled_projection_order() -> None: + graph = _context_graph() + contract, capability = _contract_and_capability() + declared = _compile_context_plan( + graph, + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=capability, + ) + reversed_declarations = _compile_context_plan( + replace(graph, context_scopes=tuple(reversed(graph.context_scopes))), + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=capability, + ) + assert isinstance(declared, _ContextPlan) + assert isinstance(reversed_declarations, _ContextPlan) + + def manifest(plan: _ContextPlan) -> tuple[tuple[str, tuple[str, ...]], ...]: + return tuple( + ( + projection.target_datum_id.value, + tuple(binding.datum_id.value for binding in projection.bindings), + ) + for projection in plan.projections + ) + + assert manifest(declared) == manifest(reversed_declarations) + + +def test_nested_contract_tampering_invalidates_the_compiled_plan() -> None: + contract, capability = _contract_and_capability() + result = _compile_context_plan( + _context_graph(), + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=capability, + ) + assert isinstance(result, _ContextPlan) + + object.__setattr__(result.contract.limits, "max_context_bytes_per_target", 1_000_000) + + assert not _is_admitted_context_plan(result) + + +@pytest.mark.parametrize( + ("mutation", "code"), + [ + ( + lambda graph: replace(graph, context_scopes=graph.context_scopes[:1]), + _ContextAdmissionCode.MISSING_CONTEXT_SCOPE, + ), + ( + lambda graph: replace(graph, context_scopes=(*graph.context_scopes, graph.context_scopes[0])), + _ContextAdmissionCode.DUPLICATE_CONTEXT_SCOPE, + ), + ( + lambda graph: replace( + graph, + context_scopes=( + replace(graph.context_scopes[0], context=(graph.datums[0].id,)), + graph.context_scopes[1], + ), + ), + _ContextAdmissionCode.SELF_CONTEXT, + ), + ( + lambda graph: replace( + graph, + context_scopes=( + replace( + graph.context_scopes[0], + context=(graph.datums[2].id, graph.datums[2].id), + ), + graph.context_scopes[1], + ), + ), + _ContextAdmissionCode.DUPLICATE_CONTEXT_MEMBER, + ), + ( + lambda graph: replace( + graph, + context_scopes=( + replace(graph.context_scopes[0], context=(graph.datums[1].id,)), + replace(graph.context_scopes[1], context=(graph.datums[0].id,)), + ), + ), + _ContextAdmissionCode.ORPHAN_CONTEXT_DATUM, + ), + ], +) +def test_context_admission_rejects_incomplete_semantics_before_capability( + mutation: Callable[[_ProtectionGraph], _ProtectionGraph], + code: _ContextAdmissionCode, +) -> None: + graph = mutation(_context_graph()) + contract, capability = _contract_and_capability() + weakened = replace(capability, retention=_RetentionPosture.ENABLED) + + assert _compile_context_plan( + graph, + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=weakened, + ) == _ContextRejected(code) + + +def test_context_limits_use_utf8_bytes_and_reject_before_backend_compatibility() -> None: + graph = _context_graph() + graph = replace( + graph, + datums=(*graph.datums[:2], replace(graph.datums[2], text="éé")), + ) + contract, capability = _contract_and_capability() + exact_limits = replace(contract.limits, max_context_bytes_per_target=8) + exact_contract = replace(contract, limits=exact_limits) + exact_capability = replace(capability, limits=exact_limits) + + assert isinstance( + _compile_context_plan( + graph, + accounting_limits=_ACCOUNTING_LIMITS, + contract=exact_contract, + capability=exact_capability, + ), + _ContextPlan, + ) + too_small = replace(exact_contract, limits=replace(exact_limits, max_context_bytes_per_target=7)) + assert _compile_context_plan( + graph, + accounting_limits=_ACCOUNTING_LIMITS, + contract=too_small, + capability=replace(exact_capability, retention=_RetentionPosture.ENABLED), + ) == _ContextRejected(_ContextAdmissionCode.CONTEXT_BYTES_EXCEEDED) + + +def test_retention_enabled_capability_rejects_a_valid_projection() -> None: + contract, capability = _contract_and_capability() + + assert _compile_context_plan( + _context_graph(), + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=replace(capability, retention=_RetentionPosture.ENABLED), + ) == _ContextRejected(_ContextAdmissionCode.BACKEND_INCOMPATIBLE) + + +@pytest.mark.parametrize( + "mutation", + [ + lambda capability: replace(capability, profile=cast(_ContextProfile, "future")), + lambda capability: replace(capability, schema_version=cast(_ContextSchemaVersion, "future")), + lambda capability: replace(capability, ordering=cast(_ContextOrdering, "implicit")), + lambda capability: replace(capability, artifact_classes=()), + lambda capability: replace(capability, allow_target_as_context=False), + lambda capability: replace( + capability, + limits=replace(capability.limits, max_total_context_references=2), + ), + ], +) +def test_each_capability_dimension_fails_closed( + mutation: Callable[[_ContextBackendCapability], _ContextBackendCapability], +) -> None: + contract, capability = _contract_and_capability() + + assert _compile_context_plan( + _context_graph(), + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=mutation(capability), + ) == _ContextRejected(_ContextAdmissionCode.BACKEND_INCOMPATIBLE) + + +@pytest.mark.parametrize( + ("limits", "code"), + [ + (_ContextLimits(1, 32, 4, 128), _ContextAdmissionCode.CONTEXT_MEMBERS_EXCEEDED), + (_ContextLimits(2, 8, 4, 128), _ContextAdmissionCode.CONTEXT_BYTES_EXCEEDED), + (_ContextLimits(2, 32, 2, 128), _ContextAdmissionCode.TOTAL_CONTEXT_REFERENCES_EXCEEDED), + (_ContextLimits(2, 32, 4, 22), _ContextAdmissionCode.EXPANDED_FRAME_BYTES_EXCEEDED), + ], +) +def test_each_context_limit_has_a_closed_rejection_code( + limits: _ContextLimits, + code: _ContextAdmissionCode, +) -> None: + contract, capability = _contract_and_capability() + + assert _compile_context_plan( + _context_graph(), + accounting_limits=_ACCOUNTING_LIMITS, + contract=replace(contract, limits=limits), + capability=replace(capability, limits=limits), + ) == _ContextRejected(code) + + +def test_unknown_context_references_and_context_only_targets_are_distinct() -> None: + graph = _context_graph() + contract, capability = _contract_and_capability() + unknown = _DatumId("unknown") + + unknown_target = replace( + graph, + context_scopes=(replace(graph.context_scopes[0], target=unknown), graph.context_scopes[1]), + ) + unknown_member = replace( + graph, + context_scopes=(replace(graph.context_scopes[0], context=(unknown,)), graph.context_scopes[1]), + ) + context_only_target = replace( + graph, + context_scopes=(*graph.context_scopes, _ContextScope(graph.datums[2].id)), + ) + + assert _compile_context_plan( + unknown_target, + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=capability, + ) == _ContextRejected(_ContextAdmissionCode.UNKNOWN_CONTEXT_TARGET) + assert _compile_context_plan( + unknown_member, + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=capability, + ) == _ContextRejected(_ContextAdmissionCode.UNKNOWN_CONTEXT_DATUM) + assert _compile_context_plan( + context_only_target, + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=capability, + ) == _ContextRejected(_ContextAdmissionCode.CONTEXT_ONLY_TARGET) + + +def test_target_as_context_requires_explicit_contract_and_capability_support() -> None: + contract, capability = _contract_and_capability() + + assert _compile_context_plan( + _context_graph(), + accounting_limits=_ACCOUNTING_LIMITS, + contract=replace(contract, allow_target_as_context=False), + capability=replace(capability, allow_target_as_context=False), + ) == _ContextRejected(_ContextAdmissionCode.TARGET_CONTEXT_DISABLED) + + +def test_missing_capability_and_unsupported_artifact_contract_fail_closed() -> None: + contract, capability = _contract_and_capability() + + assert _compile_context_plan( + _context_graph(), + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=None, + ) == _ContextRejected(_ContextAdmissionCode.BACKEND_INCOMPATIBLE) + assert _compile_context_plan( + _context_graph(), + accounting_limits=_ACCOUNTING_LIMITS, + contract=replace(contract, required_artifacts=()), + capability=capability, + ) == _ContextRejected(_ContextAdmissionCode.UNSUPPORTED_CONTEXT_CONTRACT) + + +def test_context_only_datum_cannot_become_an_atomic_output() -> None: + graph = _context_graph() + contract, capability = _contract_and_capability() + graph = replace( + graph, + atomic_groups=(*graph.atomic_groups, _AtomicGroup((graph.datums[2].id,))), + ) + + assert _compile_context_plan( + graph, + accounting_limits=_ACCOUNTING_LIMITS, + contract=contract, + capability=capability, + ) == _ContextRejected(_AccountingAdmissionCode.DANGLING_ATOMIC_MEMBER) + + +def test_malformed_contract_fails_closed_after_structural_validation() -> None: + _contract, capability = _contract_and_capability() + + assert _compile_context_plan( + _context_graph(), + accounting_limits=_ACCOUNTING_LIMITS, + contract=object(), + capability=capability, + ) == _ContextRejected(_ContextAdmissionCode.UNSUPPORTED_CONTEXT_CONTRACT) + + +def _contract_and_capability() -> tuple[_ContextExecutionContract, _ContextBackendCapability]: + limits = _ContextLimits( + max_context_members_per_target=2, + max_context_bytes_per_target=32, + max_total_context_references=4, + max_expanded_frame_bytes=128, + ) + contract = _ContextExecutionContract( + profile=_ContextProfile.TARGET_CONTEXT_V1, + schema_version=_ContextSchemaVersion.V1, + limits=limits, + allow_target_as_context=True, + ordering=_ContextOrdering.DECLARED, + required_artifacts=(_BackendArtifactClass.CONTEXT_REQUEST,), + ) + capability = _ContextBackendCapability( + profile=contract.profile, + schema_version=contract.schema_version, + limits=limits, + allow_target_as_context=True, + ordering=contract.ordering, + artifact_classes=contract.required_artifacts, + retention=_RetentionPosture.DISABLED, + ) + return contract, capability + + +def _context_graph() -> _ProtectionGraph: + target_a = _TextDatum(_DatumId("target-a"), "alpha", _DatumPurpose.TARGET) + target_b = _TextDatum(_DatumId("target-b"), "beta", _DatumPurpose.TARGET) + context = _TextDatum(_DatumId("context-c"), "gamma", _DatumPurpose.CONTEXT_ONLY) + return _ProtectionGraph( + datums=(target_a, target_b, context), + links=(), + context_scopes=( + _ContextScope(target_a.id, (context.id, target_b.id)), + _ContextScope(target_b.id, (target_a.id,)), + ), + coherence_scopes=(_CoherenceScope((target_a.id,)), _CoherenceScope((target_b.id,))), + atomic_groups=(_AtomicGroup((target_a.id,)), _AtomicGroup((target_b.id,))), + ) + + +_ACCOUNTING_LIMITS = _AccountingLimits( + max_datums=8, + max_datum_bytes=64, + max_graph_bytes=256, +) diff --git a/tests/engine/execution/test_context_runtime.py b/tests/engine/execution/test_context_runtime.py new file mode 100644 index 00000000..b56e23a1 --- /dev/null +++ b/tests/engine/execution/test_context_runtime.py @@ -0,0 +1,1111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass, replace +from enum import Enum +from typing import Any, cast + +import pandas as pd +import pytest + +import anonymizer.engine.constants as execution_constants +import anonymizer.engine.execution.graph_runtime as graph_runtime +from anonymizer.config.anonymizer_config import AnonymizerConfig +from anonymizer.config.models import ModelSelection +from anonymizer.config.replace_strategies import Redact +from anonymizer.engine.constants import ( + COL_CONTEXT_BINDING_ID, + COL_CONTEXT_ORDINAL, + COL_CONTEXT_OWNER_WORK_ID, + COL_CONTEXT_TEXT, + COL_FINAL_ENTITIES, + COL_TARGET_WORK_ID, + COL_TEXT, +) +from anonymizer.engine.execution.accounting_ledger import _AccountingLedger +from anonymizer.engine.execution.accounting_outcomes import ( + _GroupReleased, + _GroupWithheld, + _InvocationCancelled, + _InvocationCompleted, + _InvocationFailed, + _InvocationInconsistent, + _InvocationLost, + _TaskCancelled, + _TaskInconsistent, + _TaskLost, + _TaskSucceeded, +) +from anonymizer.engine.execution.accounting_plan import ( + _AccountingLimits, + _AccountingPlan, + _DatumTaskSubject, + _TaskKey, +) +from anonymizer.engine.execution.context_admission import ( + _compile_context_plan, + _ContextAdmissionCode, + _ContextPlan, + _ContextRejected, +) +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextBackendCapability, + _ContextExecutionContract, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.context_workframes import ( + _BackendArtifactId, + _BackendClosureAttestation, + _ContextBindingEvidence, + _ContextWorkframes, + _lower_context_workframes, + _make_context_binding_evidence, + _WorkframeClosedError, + _WorkframeConstructionError, +) +from anonymizer.engine.execution.graph import ( + _AtomicGroup, + _CoherenceScope, + _ContextScope, + _DatumDependency, + _DatumId, + _DatumPurpose, + _ProtectionGraph, + _TextDatum, +) +from anonymizer.engine.execution.graph_runtime import ( + _AccountingGraphExecution, + _AccountingGraphRuntime, + _ContextGraphAdmissionError, + _ExecutionFrontier, + _PreparedRuntimePlan, +) +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.execution.pandas_runtime import _PandasExecutionResult +from anonymizer.engine.execution.protection_service import _RedactProtectionService +from anonymizer.engine.private_row_verification import _InvocationRowVerifier, _TerminalOutcome +from tests.engine.execution.phase5_reference_model import ( + ReferenceAdmission, + ReferenceCase, + ReferenceEventKind, + ReferenceInvocation, + ReferenceLimits, + ReferenceResult, + ReferenceScope, + evaluate, + reference_cases, + schedule_for, +) + + +@dataclass(frozen=True) +class _BackendSchedule: + """Translate one frozen oracle trace into private backend evidence.""" + + binding_evidence: str = "exact" + cleanup: str = "verified" + task_schedule: str = "success" + terminal_evidence: _TerminalEvidence | None = None + + @classmethod + def from_reference(cls, reference: ReferenceCase) -> _BackendSchedule: + corruption = next( + (event.outcome for event in reference.events if event.kind is ReferenceEventKind.BINDING_CORRUPTION), + "exact", + ) + constructed = { + event.subject for event in reference.events if event.kind is ReferenceEventKind.BINDING_CONSTRUCTION + } + consumed = {event.subject for event in reference.events if event.kind is ReferenceEventKind.BINDING_CONSUMPTION} + if constructed - consumed: + corruption = "missing" + cleanup = tuple( + event + for event in reference.events + if event.kind in {ReferenceEventKind.CLEANUP_PRIMARY, ReferenceEventKind.CLEANUP_COMPETING} + ) + cleanup_evidence = cleanup[0].outcome if len(cleanup) == 1 else "unconfirmed" + if any(event.kind is ReferenceEventKind.TRUSTED_STOP for event in reference.events): + task_schedule = "trusted_stop" + elif any(event.kind is ReferenceEventKind.TRANSPORT_LOSS for event in reference.events): + task_schedule = "transport_loss" + elif any(event.kind is ReferenceEventKind.CANCELLATION for event in reference.events): + task_schedule = ( + "cancel_after_terminal" + if any(event.kind is ReferenceEventKind.TASK_TERMINAL for event in reference.events) + else "cancel_before_dispatch" + ) + elif any(event.kind is ReferenceEventKind.TASK_CORRUPTION for event in reference.events): + task_schedule = "corrupt" + elif {event.subject for event in reference.events if event.kind is ReferenceEventKind.TASK_TERMINAL} != { + identifier for identifier, _text in reference.targets + }: + task_schedule = "missing" + elif any(event.kind is ReferenceEventKind.TASK_TERMINAL for event in reference.events): + task_schedule = "success" + else: + task_schedule = "missing" + terminal_evidence = next( + ( + _TerminalEvidence(event.outcome) + for event in reference.events + if event.kind is ReferenceEventKind.TASK_CORRUPTION + ), + None, + ) + return cls(corruption, cleanup_evidence, task_schedule, terminal_evidence) + + @classmethod + def from_legacy_mode(cls, mode: str) -> _BackendSchedule: + """Keep focused fixture modes on the same evidence boundary.""" + match mode: + case "missing" | "duplicate" | "wrong_ordinal": + return cls(binding_evidence=mode) + case "cross": + return cls(binding_evidence="cross_target") + case "unconfirmed_cleanup": + return cls(cleanup="missing") + case "failed_cleanup": + return cls(cleanup="failed") + case "trusted_stop" | "transport_loss": + return cls(task_schedule=mode) + case _: + return cls() + + +class _ContextBackend: + def __init__( + self, + capability: _ContextBackendCapability | None, + *, + evidence_mode: str = "exact", + schedule: _BackendSchedule | None = None, + ) -> None: + self._capability = capability + self.evidence_mode = evidence_mode + self.schedule = schedule + self.calls = 0 + self.target_frame: pd.DataFrame | None = None + self.context_frame: pd.DataFrame | None = None + + def context_capability(self) -> _ContextBackendCapability | None: + return self._capability + + def run_context( + self, + dataframe: pd.DataFrame, + *, + context_dataframe: pd.DataFrame, + artifact_id: _BackendArtifactId, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + self.calls += 1 + if self.evidence_mode == "permuted": + dataframe = dataframe.iloc[::-1].copy() + dataframe.index = pd.Index([None] * len(dataframe)) + context_dataframe = context_dataframe.iloc[::-1].copy() + context_dataframe.index = pd.Index(["duplicate"] * len(context_dataframe)) + self.target_frame = dataframe.copy() + self.context_frame = context_dataframe.copy() + schedule = self.schedule or _BackendSchedule.from_legacy_mode(self.evidence_mode) + if schedule.task_schedule in {"trusted_stop", "transport_loss"}: + verifier.abort(cancelled=True) + final = dataframe.iloc[0:0].drop(columns=[COL_TARGET_WORK_ID], errors="ignore") + else: + detected = dataframe.assign(**{COL_FINAL_ENTITIES: [{"entities": []} for _ in range(len(dataframe))]}) + verifier.freeze_accepted_detections(detected) + final = verifier.finish(detected.iloc[1:] if schedule.task_schedule == "missing" else detected) + evidence = tuple( + _make_context_binding_evidence( + row[COL_CONTEXT_BINDING_ID], + row[COL_CONTEXT_OWNER_WORK_ID], + row[COL_CONTEXT_ORDINAL], + row[COL_CONTEXT_TEXT], + ) + for _index, row in context_dataframe.iterrows() + ) + if schedule.binding_evidence == "missing" and evidence: + evidence = evidence[1:] + elif schedule.binding_evidence == "duplicate" and evidence: + evidence = (evidence[0], *evidence) + elif schedule.binding_evidence == "wrong_ordinal" and evidence: + evidence = (replace(evidence[0], ordinal=evidence[0].ordinal + 1), *evidence[1:]) + elif schedule.binding_evidence in {"cross_target", "foreign", "contradictory"}: + evidence = cast(tuple[_ContextBindingEvidence, ...], (object(),)) + terminal_outcomes = verifier.take_terminal_outcomes() + result_row_tokens = verifier.take_result_order() + if schedule.task_schedule == "missing": + terminal_outcomes = terminal_outcomes[1:] + elif schedule.terminal_evidence is not None: + terminal_outcomes = _corrupt_terminal_evidence(terminal_outcomes, schedule.terminal_evidence) + closures = ( + () + if schedule.cleanup != "verified" + else ( + _BackendClosureAttestation( + artifact_id, + _BackendArtifactClass.CONTEXT_REQUEST, + True, + ), + ) + ) + if schedule.cleanup == "failed": + closures = (_BackendClosureAttestation(artifact_id, _BackendArtifactClass.CONTEXT_REQUEST, False),) + return _PandasExecutionResult( + dataframe=final, + failed_records=[], + terminal_outcomes=terminal_outcomes, + result_row_tokens=result_row_tokens, + trusted_stop_tokens=( + tuple(dataframe[COL_TARGET_WORK_ID]) if schedule.task_schedule == "trusted_stop" else () + ), + context_binding_evidence=evidence, + closure_attestations=closures, + ) + + def run(self, *_args: object, **_kwargs: object) -> _PandasExecutionResult: + raise AssertionError("context plans must not fall back to independent-row execution") + + +def _datum_id(task: _TaskKey) -> _DatumId: + assert isinstance(task.subject, _DatumTaskSubject) + return task.subject.datum_id + + +def _corrupt_terminal_evidence( + terminal_outcomes: tuple[tuple[str, _TerminalOutcome], ...], + evidence: _TerminalEvidence, +) -> tuple[tuple[str, _TerminalOutcome], ...]: + """Emit one frozen terminal-evidence class at the private backend boundary.""" + first = terminal_outcomes[0] + match evidence: + case _TerminalEvidence.DUPLICATE | _TerminalEvidence.CONTRADICTORY: + return (*terminal_outcomes, first) + case _TerminalEvidence.FOREIGN: + return (*terminal_outcomes, ("foreign-terminal", first[1])) + case _TerminalEvidence.STALE: + return (*terminal_outcomes, ("stale-terminal", first[1])) + case _TerminalEvidence.CROSS_TARGET: + return (*terminal_outcomes, ("cross-target-terminal", first[1])) + case _TerminalEvidence.PLAN_MISMATCH: + return (*terminal_outcomes, ("mismatched-plan-terminal", first[1])) + + +class _TerminalEvidence(str, Enum): + """Frozen malformed terminal-evidence classes accepted by the corpus adapter.""" + + DUPLICATE = "duplicate" + FOREIGN = "foreign" + STALE = "stale" + CROSS_TARGET = "cross_target" + PLAN_MISMATCH = "plan_mismatch" + CONTRADICTORY = "contradictory" + + +class _ScheduledContextRuntime(_AccountingGraphRuntime): + """Inject cancellation linearization at the test runtime boundary only.""" + + def __init__(self, backend: _ContextBackend, schedule: _BackendSchedule | None) -> None: + super().__init__(backend) + self._schedule = schedule + + def _build_frontier( + self, + ledger: _AccountingLedger[tuple[str, str]], + prepared: _PreparedRuntimePlan, + ready: tuple[_TaskKey, ...], + datum_by_id: dict[_DatumId, _TextDatum], + ) -> _ExecutionFrontier | None: + if self._schedule is not None and self._schedule.task_schedule == "cancel_before_dispatch": + ledger.request_cancellation() + return super()._build_frontier(ledger, prepared, ready, datum_by_id) + + def _accept_frontier( + self, + ledger: _AccountingLedger[tuple[str, str]], + plan: _AccountingPlan, + frontier: _ExecutionFrontier, + value: object, + hydrate: Callable[[_TextDatum, pd.Series], tuple[str, str]], + ) -> bool: + accepted = super()._accept_frontier(ledger, plan, frontier, value, hydrate) + if self._schedule is not None and self._schedule.task_schedule == "cancel_after_terminal": + ledger.request_cancellation() + return accepted + + +@pytest.mark.parametrize( + "mutation", + [ + lambda capability: replace(capability, retention=_RetentionPosture.ENABLED), + lambda capability: replace(capability, profile=cast(_ContextProfile, "future")), + lambda capability: replace(capability, schema_version=cast(_ContextSchemaVersion, "future")), + lambda capability: replace(capability, ordering=cast(_ContextOrdering, "implicit")), + lambda capability: replace(capability, artifact_classes=()), + lambda capability: replace( + capability, + limits=replace(capability.limits, max_context_bytes_per_target=1), + ), + ], +) +def test_runtime_rechecks_complete_capability_before_open_or_dispatch( + mutation: Callable[[_ContextBackendCapability], _ContextBackendCapability], + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + backend = _ContextBackend(mutation(capability)) + + with pytest.raises(_ContextGraphAdmissionError) as raised: + _run(plan, backend, stub_slim_model_selection) + + assert raised.value.code is _ContextAdmissionCode.BACKEND_INCOMPATIBLE + assert backend.calls == 0 + + +def test_runtime_rejects_a_capable_backend_without_context_execution_before_open( + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + + class _CapabilityOnlyBackend: + def context_capability(self) -> _ContextBackendCapability: + return capability + + def run(self, *_args: object, **_kwargs: object) -> _PandasExecutionResult: + raise AssertionError("context plans must not use the row backend") + + with pytest.raises(_ContextGraphAdmissionError) as raised: + _run(plan, cast(_ContextBackend, _CapabilityOnlyBackend()), stub_slim_model_selection) + + assert raised.value.code is _ContextAdmissionCode.BACKEND_INCOMPATIBLE + + +@pytest.mark.parametrize( + "runner", + [None, object(), lambda dataframe: dataframe], +) +def test_runtime_rejects_unusable_context_runner_before_ledger_or_dispatch( + monkeypatch: pytest.MonkeyPatch, + stub_slim_model_selection: ModelSelection, + runner: object, +) -> None: + plan, capability = _plan() + backend = _ContextBackend(capability) + monkeypatch.setattr(backend, "run_context", runner, raising=False) + + def reject_open(*_args: object, **_kwargs: object) -> None: + raise AssertionError("ledger must not open") + + def reject_lowering(*_args: object, **_kwargs: object) -> None: + raise AssertionError("workframes must not be constructed") + + monkeypatch.setattr(_AccountingLedger, "open", reject_open) + monkeypatch.setattr(graph_runtime, "_lower_context_workframes", reject_lowering) + + with pytest.raises(_ContextGraphAdmissionError) as raised: + _run(plan, backend, stub_slim_model_selection) + + assert raised.value.code is _ContextAdmissionCode.BACKEND_INCOMPATIBLE + + +def test_runtime_recheck_handles_a_raising_capability_snapshot_before_open( + monkeypatch: pytest.MonkeyPatch, + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + backend = _ContextBackend(capability) + + def raising_snapshot() -> _ContextBackendCapability: + raise RuntimeError("backend unavailable") + + def reject_open(*_args: object, **_kwargs: object) -> None: + raise AssertionError("ledger must not open") + + monkeypatch.setattr(backend, "context_capability", raising_snapshot) + monkeypatch.setattr(_AccountingLedger, "open", reject_open) + + with pytest.raises(_ContextGraphAdmissionError) as raised: + _run(plan, backend, stub_slim_model_selection) + + assert raised.value.code is _ContextAdmissionCode.BACKEND_INCOMPATIBLE + + +def test_context_target_frame_carries_exact_private_task_and_attempt_identities( + monkeypatch: pytest.MonkeyPatch, + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + backend = _ContextBackend(capability) + dispatched = [] + real_dispatch = _AccountingLedger.dispatch_batch + + def capture_dispatch( + ledger: _AccountingLedger[object], + tasks: tuple[_TaskKey, ...], + *, + row_token_values: tuple[str, ...], + ): + values = real_dispatch(ledger, tasks, row_token_values=row_token_values) + dispatched.extend(values) + return values + + monkeypatch.setattr(_AccountingLedger, "dispatch_batch", capture_dispatch) + + execution = _run(plan, backend, stub_slim_model_selection) + + assert isinstance(execution.accounting.invocation, _InvocationCompleted) + assert backend.target_frame is not None + task_column = getattr(execution_constants, "COL_TASK_ID") + attempt_column = getattr(execution_constants, "COL_ATTEMPT_ID") + assert {task_column, attempt_column}.issubset(backend.target_frame.columns) + assert tuple(backend.target_frame[task_column]) == tuple(dispatch.task for dispatch in dispatched) + assert tuple(backend.target_frame[attempt_column]) == tuple(dispatch.attempt_id for dispatch in dispatched) + rendered = backend.target_frame.to_string() + assert "target-a" not in rendered + assert "target-b" not in rendered + + +def test_context_construction_failure_occurs_before_dispatch( + monkeypatch: pytest.MonkeyPatch, + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + backend = _ContextBackend(capability) + + def fail_lowering(*_args: object, **_kwargs: object) -> None: + raise _WorkframeConstructionError + + def reject_dispatch(*_args: object, **_kwargs: object) -> None: + raise AssertionError("dispatch must not occur after construction failed") + + monkeypatch.setattr(graph_runtime, "_lower_context_workframes", fail_lowering) + monkeypatch.setattr(_AccountingLedger, "dispatch_batch", reject_dispatch) + + execution = _run(plan, backend, stub_slim_model_selection) + + assert all(not isinstance(task, _TaskSucceeded) for task in execution.accounting.tasks) + assert backend.calls == 0 + + +def test_dispatch_failure_closes_constructed_workframes( + monkeypatch: pytest.MonkeyPatch, + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + backend = _ContextBackend(capability) + lowered: list[_ContextWorkframes] = [] + real_lowering = graph_runtime._lower_context_workframes + + def capture_lowering(*args: Any, **kwargs: Any) -> _ContextWorkframes: + frames = real_lowering(*args, **kwargs) + lowered.append(frames) + return frames + + def fail_dispatch(*_args: object, **_kwargs: object) -> None: + raise RuntimeError("dispatch unavailable") + + monkeypatch.setattr(graph_runtime, "_lower_context_workframes", capture_lowering) + monkeypatch.setattr(_AccountingLedger, "dispatch_batch", fail_dispatch) + + execution = _run(plan, backend, stub_slim_model_selection) + + assert all(not isinstance(task, _TaskSucceeded) for task in execution.accounting.tasks) + assert backend.calls == 0 + assert len(lowered) == 1 + assert lowered[0].target_frame.empty + assert lowered[0].context_frame.empty + rendered = repr(vars(lowered[0])) + assert all(value not in rendered for value in ("alpha", "beta", "gamma", "target-a", "context-c")) + with pytest.raises(_WorkframeClosedError): + lowered[0].artifact_id + + +def test_discard_failure_after_uncommitted_dispatch_is_contained_and_embargoed( + monkeypatch: pytest.MonkeyPatch, + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + backend = _ContextBackend(capability) + lowered: list[_ContextWorkframes] = [] + real_lowering = graph_runtime._lower_context_workframes + + def capture_lowering(*args: Any, **kwargs: Any) -> _ContextWorkframes: + frames = real_lowering(*args, **kwargs) + lowered.append(frames) + return frames + + def fail_dispatch(*_args: object, **_kwargs: object) -> None: + raise RuntimeError("dispatch unavailable") + + def fail_discard(self: _ContextWorkframes) -> None: + raise RuntimeError("discard unavailable") + + monkeypatch.setattr(graph_runtime, "_lower_context_workframes", capture_lowering) + monkeypatch.setattr(_AccountingLedger, "dispatch_batch", fail_dispatch) + monkeypatch.setattr(_ContextWorkframes, "discard_before_dispatch", fail_discard) + + execution = _run(plan, backend, stub_slim_model_selection) + + assert isinstance(execution.accounting.invocation, _InvocationFailed) + assert "cleanup_failed" in _invocation_causes(execution) + assert all(isinstance(group, _GroupWithheld) for group in execution.accounting.groups) + assert backend.calls == 0 + assert len(lowered) == 1 + assert lowered[0].target_frame.empty + assert lowered[0].context_frame.empty + with pytest.raises(_WorkframeClosedError): + lowered[0].artifact_id + + +def test_cancellation_winning_before_atomic_dispatch_closes_frames( + monkeypatch: pytest.MonkeyPatch, + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + backend = _ContextBackend(capability) + lowered: list[_ContextWorkframes] = [] + real_lowering = graph_runtime._lower_context_workframes + real_dispatch = _AccountingLedger.dispatch_batch + + def capture_lowering(*args: Any, **kwargs: Any) -> _ContextWorkframes: + frames = real_lowering(*args, **kwargs) + lowered.append(frames) + return frames + + def cancel_before_dispatch( + ledger: _AccountingLedger[object], + tasks: tuple[_TaskKey, ...], + *, + row_token_values: tuple[str, ...], + ): + ledger.request_cancellation() + return real_dispatch(ledger, tasks, row_token_values=row_token_values) + + monkeypatch.setattr(graph_runtime, "_lower_context_workframes", capture_lowering) + monkeypatch.setattr(_AccountingLedger, "dispatch_batch", cancel_before_dispatch) + + execution = _run(plan, backend, stub_slim_model_selection) + + assert isinstance(execution.accounting.invocation, _InvocationCancelled) + assert backend.calls == 0 + assert len(lowered) == 1 + assert lowered[0].target_frame.empty + assert lowered[0].context_frame.empty + + +def test_late_cancellation_preserves_private_success_but_embargos_release() -> None: + plan, _capability = _plan() + ledger: _AccountingLedger[str] = _AccountingLedger(plan.accounting) + ledger.open() + ready = ledger.ready_tasks() + frames = _lower_context_workframes(plan, ready) + correlations = tuple(work_id.value for work_id in frames.target_work_ids()) + dispatches = ledger.dispatch_batch(ready, row_token_values=correlations) + frames.bind_dispatches(dispatches) + for dispatch in dispatches: + ledger.accept_success(dispatch, _datum_id(dispatch.task).value) + evidence = tuple( + _make_context_binding_evidence( + row[COL_CONTEXT_BINDING_ID], + row[COL_CONTEXT_OWNER_WORK_ID], + row[COL_CONTEXT_ORDINAL], + row[COL_CONTEXT_TEXT], + ) + for _index, row in frames.context_frame.iterrows() + ) + assert frames.reconcile(evidence).status.value == "verified" + artifact_id = frames.artifact_id + assert ( + frames.close( + (_BackendClosureAttestation(artifact_id, _BackendArtifactClass.CONTEXT_REQUEST, True),) + ).status.value + == "verified" + ) + + ledger.request_cancellation() + result = ledger.finish() + + assert all(isinstance(task, _TaskSucceeded) for task in result.tasks) + assert isinstance(result.invocation, _InvocationCancelled) + assert all(isinstance(group, _GroupWithheld) for group in result.groups) + + +def test_protection_service_preflights_context_with_the_selected_backend() -> None: + plan, capability = _plan() + backend = _ContextBackend(capability) + service = _RedactProtectionService(_AccountingGraphRuntime(backend)) + + admitted = service.admit_context( + _graph(), + accounting_limits=_AccountingLimits(8, 64, 256), + contract=plan.contract, + ) + + assert isinstance(admitted, _ContextPlan) + + +def test_context_runtime_reconciles_exact_bindings_and_releases_targets_only( + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + backend = _ContextBackend(capability) + + execution = _run(plan, backend, stub_slim_model_selection) + + assert isinstance(execution.accounting.invocation, _InvocationCompleted) + assert all(isinstance(group, _GroupReleased) for group in execution.accounting.groups) + assert backend.calls == 1 + assert backend.target_frame is not None + assert backend.context_frame is not None + assert COL_TEXT in backend.target_frame + assert COL_TEXT not in backend.context_frame + assert set(backend.context_frame[COL_CONTEXT_ORDINAL]) == {0, 1} + assert "context-c" not in backend.context_frame.to_string() + + +def test_context_runtime_is_invariant_to_equal_text_row_order_and_duplicate_indices( + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan(equal_text=True) + backend = _ContextBackend(capability, evidence_mode="permuted") + + execution = _run(plan, backend, stub_slim_model_selection) + + assert all(isinstance(group, _GroupReleased) for group in execution.accounting.groups) + assert backend.target_frame is not None + assert backend.context_frame is not None + assert backend.target_frame.index.tolist() == [None, None] + assert backend.context_frame.index.tolist() == ["duplicate", "duplicate", "duplicate"] + assert backend.target_frame[COL_TEXT].tolist() == ["same-text", "same-text"] + assert backend.context_frame[COL_CONTEXT_TEXT].tolist() == ["same-text", "same-text", "same-text"] + + +def test_missing_binding_evidence_withholds_only_its_owner_group( + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + + execution = _run(plan, _ContextBackend(capability, evidence_mode="missing"), stub_slim_model_selection) + + assert isinstance(execution.accounting.groups[0], _GroupWithheld) + assert isinstance(execution.accounting.groups[1], _GroupReleased) + + +@pytest.mark.parametrize("mode", ["cross", "unconfirmed_cleanup"]) +def test_global_binding_or_cleanup_uncertainty_embargos_every_group( + mode: str, + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + + execution = _run(plan, _ContextBackend(capability, evidence_mode=mode), stub_slim_model_selection) + + assert isinstance(execution.accounting.invocation, _InvocationInconsistent) + assert all(isinstance(group, _GroupWithheld) for group in execution.accounting.groups) + + +def test_definitive_cleanup_failure_preserves_private_tasks_but_fails_public_release( + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + + execution = _run(plan, _ContextBackend(capability, evidence_mode="failed_cleanup"), stub_slim_model_selection) + + assert isinstance(execution.accounting.invocation, _InvocationFailed) + assert all(isinstance(task, _TaskSucceeded) for task in execution.accounting.tasks) + assert all(isinstance(group, _GroupWithheld) for group in execution.accounting.groups) + + +def test_production_admission_matches_every_frozen_reference_case() -> None: + """Pair every frozen oracle dimension with the production admission boundary.""" + for reference in reference_cases(): + expected = evaluate(reference) + actual = _compile_reference_case(reference) + + assert isinstance(actual, _ContextPlan) is (expected.admission is ReferenceAdmission.ADMITTED), ( + reference.case_id + ) + + +def _runtime_reference_cases() -> tuple[ReferenceCase, ...]: + """Cover every frozen graph, schedule, and runtime-capability class.""" + return tuple(reference_cases()) + + +@pytest.mark.parametrize("reference", _runtime_reference_cases(), ids=lambda reference: reference.case_id) +def test_production_runtime_and_release_match_every_admitted_frozen_graph_and_runtime_capability( + reference: ReferenceCase, + stub_slim_model_selection: ModelSelection, +) -> None: + """Pair every admitted frozen graph and runtime capability with runtime release.""" + expected = evaluate(reference) + if expected.admission is not ReferenceAdmission.ADMITTED: + return + + plan = _compile_reference_case(reference) + assert isinstance(plan, _ContextPlan) + backend = _ContextBackend( + _reference_runtime_capability(reference, plan.contract), + schedule=_BackendSchedule.from_reference(reference), + ) + if expected.invocation is ReferenceInvocation.NOT_OPENED: + with pytest.raises(_ContextGraphAdmissionError) as raised: + _run(plan, backend, stub_slim_model_selection) + assert raised.value.code is _ContextAdmissionCode.BACKEND_INCOMPATIBLE + assert backend.calls == 0 + return + + execution = _run(plan, backend, stub_slim_model_selection, schedule=backend.schedule) + + _assert_execution_matches_reference(execution, plan, expected, reference=reference, backend=backend) + + +def _compile_reference_case(reference: ReferenceCase) -> _ContextPlan | _ContextRejected: + graph = _reference_graph(reference) + contract = _reference_contract(reference) + capability = _reference_capability(reference, contract) + return _compile_context_plan( + graph, + accounting_limits=_AccountingLimits( + len(graph.datums), + reference.limits.datum_bytes, + sum(len(datum.text.encode()) for datum in graph.datums), + reference.limits.id_bytes, + ), + contract=contract, + capability=capability, + ) + + +def _reference_graph(reference: ReferenceCase) -> _ProtectionGraph: + target_datums = tuple( + _TextDatum(_DatumId(identifier), text, _DatumPurpose.TARGET) for identifier, text in reference.targets + ) + context_datums = tuple( + _TextDatum(_DatumId(identifier), text, _DatumPurpose.CONTEXT_ONLY) + for identifier, text in reference.context_only + ) + return _ProtectionGraph( + datums=(*target_datums, *context_datums), + links=(), + context_scopes=tuple( + _ContextScope(_DatumId(scope.target), tuple(_DatumId(member) for member in scope.context)) + for scope in reference.scopes + ), + coherence_scopes=tuple(_CoherenceScope((_DatumId(identifier),)) for identifier, _text in reference.targets), + atomic_groups=tuple(_AtomicGroup(tuple(_DatumId(member) for member in group)) for group in reference.groups), + dependencies=tuple( + _DatumDependency(_DatumId(before), _DatumId(after)) for before, after in reference.dependencies + ), + ) + + +def _reference_contract(reference: ReferenceCase) -> _ContextExecutionContract: + return _ContextExecutionContract( + _ContextProfile.TARGET_CONTEXT_V1 + if reference.profile == "target-context-v1" and reference.relation == "bounded_context" + else cast(_ContextProfile, "unsupported"), + _ContextSchemaVersion.V1 + if reference.schema == "context-workframe-v1" + else cast(_ContextSchemaVersion, "unsupported"), + _ContextLimits( + reference.limits.members, + reference.limits.context_bytes, + reference.limits.references, + reference.limits.expanded_bytes, + ), + reference.allow_target_as_context, + _ContextOrdering.DECLARED if reference.ordering == "declared" else cast(_ContextOrdering, "unsupported"), + (_BackendArtifactClass.CONTEXT_REQUEST,), + ) + + +def _reference_capability(reference: ReferenceCase, contract: _ContextExecutionContract) -> _ContextBackendCapability: + return _ContextBackendCapability( + _ContextProfile.TARGET_CONTEXT_V1, + _ContextSchemaVersion.V1, + contract.limits, + reference.allow_target_as_context, + _ContextOrdering.DECLARED, + (_BackendArtifactClass.CONTEXT_REQUEST,), + _RetentionPosture.DISABLED if reference.preflight_capability == "compatible" else _RetentionPosture.ENABLED, + ) + + +def _reference_runtime_capability( + reference: ReferenceCase, + contract: _ContextExecutionContract, +) -> _ContextBackendCapability | None: + capability = _reference_capability(reference, contract) + match reference.runtime_capability: + case "compatible": + return capability + case "missing": + return None + case "incompatible": + return replace(capability, retention=_RetentionPosture.ENABLED) + case "weakened": + return replace(capability, limits=replace(capability.limits, max_expanded_frame_bytes=0)) + case "retention_enabled": + return replace(capability, retention=_RetentionPosture.ENABLED) + case "profile": + return replace(capability, profile=cast(_ContextProfile, "future")) + case "schema": + return replace(capability, schema_version=cast(_ContextSchemaVersion, "future")) + case "ordering": + return replace(capability, ordering=cast(_ContextOrdering, "implicit")) + case unexpected: + raise AssertionError(f"unexpected frozen runtime capability: {unexpected}") + + +@pytest.mark.parametrize( + ("backend_mode", "reference_evidence", "reference_cleanup", "task_schedule"), + [ + ("exact", "exact", "verified", "success"), + ("missing", "missing", "verified", "success"), + ("duplicate", "duplicate", "verified", "success"), + ("wrong_ordinal", "wrong_ordinal", "verified", "success"), + ("cross", "cross_target", "verified", "success"), + ("unconfirmed_cleanup", "exact", "missing", "success"), + ("failed_cleanup", "exact", "failed", "success"), + ("trusted_stop", "exact", "verified", "trusted_stop"), + ("transport_loss", "exact", "verified", "transport_loss"), + ], +) +def test_production_release_matches_the_independent_reference_model( + backend_mode: str, + reference_evidence: str, + reference_cleanup: str, + task_schedule: str, + stub_slim_model_selection: ModelSelection, +) -> None: + plan, capability = _plan() + reference = ReferenceCase( + case_id="production-pair", + targets=(("target-a", "alpha"), ("target-b", "beta")), + context_only=(("context-c", "gamma"),), + scopes=( + ReferenceScope("target-a", ("context-c", "target-b")), + ReferenceScope("target-b", ("target-a",)), + ), + limits=ReferenceLimits(5, 9, 2, 9, 3, 23), + events=(), + groups=(("target-a",), ("target-b",)), + ) + reference = replace( + reference, + events=schedule_for( + reference, + binding_evidence=reference_evidence, + cleanup=reference_cleanup, + task_schedule=task_schedule, + ), + ) + + expected = evaluate(reference) + actual = _run(plan, _ContextBackend(capability, evidence_mode=backend_mode), stub_slim_model_selection) + + assert expected.admission is ReferenceAdmission.ADMITTED + assert expected.binding_count == sum(len(projection.bindings) for projection in plan.projections) + assert expected.event_count <= expected.event_max + _assert_execution_matches_reference(actual, plan, expected) + + +def _run( + plan: _ContextPlan, + backend: _ContextBackend, + model_selection: ModelSelection, + *, + schedule: _BackendSchedule | None = None, +) -> _AccountingGraphExecution[tuple[str, str]]: + runtime = _AccountingGraphRuntime(backend) if schedule is None else _ScheduledContextRuntime(backend, schedule) + return runtime.run( + plan, + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + ) + + +def _assert_execution_matches_reference( + execution: _AccountingGraphExecution[tuple[str, str]], + plan: _ContextPlan, + expected: ReferenceResult, + *, + reference: ReferenceCase | None = None, + backend: _ContextBackend | None = None, +) -> None: + released = tuple( + datum_id.value + for group in execution.accounting.groups + if isinstance(group, _GroupReleased) + for datum_id, _candidate in group.outputs + ) + + assert released == expected.released + _assert_invocation_matches_reference(execution, expected, reference) + assert _task_outcomes(execution, plan) == _runtime_task_outcomes(expected, reference) + if backend is not None: + assert _runtime_cleanup_outcome(execution, backend) == expected.cleanup + + +def _assert_invocation_matches_reference( + execution: _AccountingGraphExecution[tuple[str, str]], + expected: ReferenceResult, + reference: ReferenceCase | None, +) -> None: + assert isinstance(execution.accounting.invocation, _InvocationCompleted) == ( + expected.invocation is ReferenceInvocation.COMPLETED + ) + assert isinstance(execution.accounting.invocation, _InvocationInconsistent) == ( + expected.invocation is ReferenceInvocation.INCONSISTENT + ) + assert isinstance(execution.accounting.invocation, _InvocationFailed) == ( + expected.invocation is ReferenceInvocation.FAILED + ) + assert isinstance(execution.accounting.invocation, _InvocationCancelled) == ( + expected.invocation is ReferenceInvocation.CANCELLED + ) + assert isinstance(execution.accounting.invocation, _InvocationLost) == ( + expected.invocation is ReferenceInvocation.LOST + ) + expected_cause = _runtime_invocation_cause(expected, reference) + if expected_cause != "none": + assert expected_cause in _invocation_causes(execution) + + +def _runtime_task_outcomes( + expected: ReferenceResult, + reference: ReferenceCase | None, +) -> tuple[tuple[str, str, str], ...]: + """Project oracle schedules onto the private runtime's terminal evidence vocabulary.""" + if reference is None: + return expected.task_outcomes + schedule = _BackendSchedule.from_reference(reference) + if schedule.terminal_evidence is not None: + return tuple((target, "inconsistent", "foreign") for target, _state, _reason in expected.task_outcomes) + if schedule.task_schedule == "missing": + return tuple( + (target, state, "missing" if reason == "terminal_missing" else reason) + for target, state, reason in expected.task_outcomes + ) + if schedule.task_schedule == "cancel_before_dispatch": + return tuple((target, state, "stop_acknowledged") for target, state, _reason in expected.task_outcomes) + if schedule.binding_evidence == "wrong_ordinal": + return tuple( + (target, state, "contradictory" if reason == "wrong_ordinal" else reason) + for target, state, reason in expected.task_outcomes + ) + return expected.task_outcomes + + +def _runtime_invocation_cause(expected: ReferenceResult, reference: ReferenceCase | None) -> str: + if reference is not None and _BackendSchedule.from_reference(reference).terminal_evidence is not None: + return "foreign" + return {"terminal_attribution_invalid": "foreign"}.get(expected.reason, expected.reason) + + +def _runtime_cleanup_outcome( + execution: _AccountingGraphExecution[tuple[str, str]], + backend: _ContextBackend, +) -> str: + if backend.calls == 0: + return "not_entered" + causes = _invocation_causes(execution) + if "cleanup_failed" in causes: + return "failed" + if "cleanup_unconfirmed" in causes: + return "unconfirmed" + return "verified" + + +def _invocation_causes(execution: _AccountingGraphExecution[tuple[str, str]]) -> tuple[str, ...]: + causes = getattr(execution.accounting.invocation, "causes", ()) + return tuple(cause.code.value for cause in causes) + + +def _task_outcomes( + execution: _AccountingGraphExecution[tuple[str, str]], + plan: _ContextPlan, +) -> tuple[tuple[str, str, str], ...]: + return tuple( + ( + datum.id.value, + "succeeded" + if isinstance(task, _TaskSucceeded) + else "inconsistent" + if isinstance(task, _TaskInconsistent) + else "cancelled" + if isinstance(task, _TaskCancelled) + else "lost", + "none" + if isinstance(task, _TaskSucceeded) + else "stop_acknowledged" + if isinstance(task, _TaskCancelled) + else next(cause.code.value for cause in task.causes) + if isinstance(task, (_TaskInconsistent, _TaskLost)) + else "unexpected", + ) + for datum, task in zip(plan.accounting.datums, execution.accounting.tasks, strict=True) + ) + + +def _plan(*, equal_text: bool = False) -> tuple[_ContextPlan, _ContextBackendCapability]: + graph = _graph() + if equal_text: + graph = replace(graph, datums=tuple(replace(datum, text="same-text") for datum in graph.datums)) + limits = _ContextLimits(2, 32, 4, 128) + contract = _ContextExecutionContract( + _ContextProfile.TARGET_CONTEXT_V1, + _ContextSchemaVersion.V1, + limits, + True, + _ContextOrdering.DECLARED, + (_BackendArtifactClass.CONTEXT_REQUEST,), + ) + capability = _ContextBackendCapability( + contract.profile, + contract.schema_version, + limits, + True, + contract.ordering, + contract.required_artifacts, + _RetentionPosture.DISABLED, + ) + compiled = _compile_context_plan( + graph, + accounting_limits=_AccountingLimits(8, 64, 256), + contract=contract, + capability=capability, + ) + assert isinstance(compiled, _ContextPlan) + return compiled, capability + + +def _graph() -> _ProtectionGraph: + target_a = _TextDatum(_DatumId("target-a"), "alpha", _DatumPurpose.TARGET) + target_b = _TextDatum(_DatumId("target-b"), "beta", _DatumPurpose.TARGET) + context = _TextDatum(_DatumId("context-c"), "gamma", _DatumPurpose.CONTEXT_ONLY) + return _ProtectionGraph( + datums=(target_a, target_b, context), + links=(), + context_scopes=( + _ContextScope(target_a.id, (context.id, target_b.id)), + _ContextScope(target_b.id, (target_a.id,)), + ), + coherence_scopes=(_CoherenceScope((target_a.id,)), _CoherenceScope((target_b.id,))), + atomic_groups=(_AtomicGroup((target_a.id,)), _AtomicGroup((target_b.id,))), + ) diff --git a/tests/engine/execution/test_context_workframes.py b/tests/engine/execution/test_context_workframes.py new file mode 100644 index 00000000..2ee14d2c --- /dev/null +++ b/tests/engine/execution/test_context_workframes.py @@ -0,0 +1,381 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +from collections.abc import Iterator +from dataclasses import replace +from hashlib import sha256 +from typing import cast + +import pytest + +import anonymizer.engine.constants as execution_constants +from anonymizer.engine.constants import ( + COL_CONTEXT_BINDING_ID, + COL_CONTEXT_ORDINAL, + COL_CONTEXT_OWNER_WORK_ID, + COL_CONTEXT_TEXT, + COL_TARGET_WORK_ID, + COL_TEXT, +) +from anonymizer.engine.execution.accounting_evidence import _AttemptId, _Dispatch, _InvocationId, _RowToken +from anonymizer.engine.execution.accounting_plan import _AccountingLimits +from anonymizer.engine.execution.context_admission import _compile_context_plan, _ContextPlan +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextBackendCapability, + _ContextExecutionContract, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.context_workframes import ( + _BackendArtifactId, + _BackendClosureAttestation, + _ContextBindingEvidence, + _ContextBindingId, + _ContextCleanupStatus, + _ContextPayloadToken, + _ContextReconciliationStatus, + _lower_context_workframes, + _make_context_binding_evidence, + _TargetWorkId, + _WorkframeClosedError, + _WorkframeStateError, +) +from anonymizer.engine.execution.graph import ( + _AtomicGroup, + _CoherenceScope, + _ContextScope, + _DatumId, + _DatumPurpose, + _ProtectionGraph, + _TextDatum, +) + + +def test_lowering_keeps_target_and_ordered_context_frames_separate() -> None: + plan = _plan() + frames = _lower_context_workframes( + plan, + tuple(projection.owner_task for projection in plan.projections), + target_work_ids=("target-work-a", "target-work-b"), + identity_factory=_identities("binding-a0", "binding-a1", "binding-b0", "artifact-a"), + ) + + task_column = getattr(execution_constants, "COL_TASK_ID") + attempt_column = getattr(execution_constants, "COL_ATTEMPT_ID") + assert list(frames.target_frame.columns) == [COL_TARGET_WORK_ID, task_column, attempt_column, COL_TEXT] + assert list(frames.context_frame.columns) == [ + COL_CONTEXT_BINDING_ID, + COL_CONTEXT_OWNER_WORK_ID, + COL_CONTEXT_ORDINAL, + COL_CONTEXT_TEXT, + ] + assert frames.target_frame.to_dict("records") == [ + { + COL_TARGET_WORK_ID: "target-work-a", + task_column: plan.projections[0].owner_task, + attempt_column: None, + COL_TEXT: "alpha", + }, + { + COL_TARGET_WORK_ID: "target-work-b", + task_column: plan.projections[1].owner_task, + attempt_column: None, + COL_TEXT: "beta", + }, + ] + assert frames.context_frame[COL_CONTEXT_TEXT].tolist() == ["gamma", "beta", "alpha"] + assert frames.context_frame[COL_CONTEXT_ORDINAL].tolist() == [0, 1, 0] + serialized = f"{frames.target_frame.columns.tolist()}{frames.context_frame.columns.tolist()}" + assert "target-a" not in serialized + assert "context-c" not in serialized + + +def test_exact_binding_and_closure_evidence_verifies_then_closes_all_state() -> None: + frames = _frames() + _bind_dispatches(frames) + evidence = _exact_evidence(frames) + + reconciliation = frames.reconcile(evidence) + cleanup = frames.close( + (_BackendClosureAttestation(frames.artifact_id, _BackendArtifactClass.CONTEXT_REQUEST, True),) + ) + + assert reconciliation.status is _ContextReconciliationStatus.VERIFIED + assert cleanup.status is _ContextCleanupStatus.VERIFIED + assert frames.target_frame.empty + assert frames.context_frame.empty + rendered = repr(vars(frames)) + for canary in ("alpha", "beta", "gamma", "binding-a0", "target-work-a", "artifact-a"): + assert canary not in rendered + assert sha256(canary.encode()).hexdigest() not in rendered + with pytest.raises(_WorkframeClosedError): + frames.expected_bindings() + with pytest.raises(_WorkframeClosedError): + _ = frames.artifact_id + + +def test_dispatch_binding_is_an_absorbing_one_shot_transition() -> None: + frames = _frames() + dispatches = _dispatches_for(frames) + + frames.bind_dispatches(dispatches) + original_attempts = tuple(frames.target_frame[getattr(execution_constants, "COL_ATTEMPT_ID")]) + + with pytest.raises(_WorkframeStateError): + frames.bind_dispatches(dispatches) + with pytest.raises(_WorkframeStateError): + frames.bind_dispatches(tuple(reversed(dispatches))) + + assert tuple(frames.target_frame[getattr(execution_constants, "COL_ATTEMPT_ID")]) == original_attempts + + +def test_missing_binding_is_local_to_its_compiled_owner() -> None: + frames = _frames() + _bind_dispatches(frames) + + result = frames.reconcile(_exact_evidence(frames)[1:]) + + assert result.status is _ContextReconciliationStatus.LOCAL_INVALID + assert result.affected_tasks == (frames.tasks[0],) + + +def test_cross_target_binding_is_a_global_attribution_failure() -> None: + frames = _frames() + _bind_dispatches(frames) + expected = frames.expected_bindings() + _binding_id, _owner_work_id, _ordinal = expected[0] + exact = _exact_evidence(frames) + + result = frames.reconcile( + ( + replace(exact[0], owner_target_work_id=_TargetWorkId("target-work-b")), + *exact[1:], + ) + ) + + assert result.status is _ContextReconciliationStatus.GLOBAL_INVALID + assert result.affected_tasks == () + + +def test_reordered_evidence_is_transport_only_and_known_duplicate_is_local() -> None: + reordered = _frames() + _bind_dispatches(reordered) + exact = tuple(reversed(_exact_evidence(reordered))) + assert reordered.reconcile(exact).status is _ContextReconciliationStatus.VERIFIED + + duplicated = _frames() + _bind_dispatches(duplicated) + exact_duplicate = _exact_evidence(duplicated) + duplicate_evidence = (*exact_duplicate, exact_duplicate[0]) + result = duplicated.reconcile(duplicate_evidence) + + assert result.status is _ContextReconciliationStatus.LOCAL_INVALID + assert result.affected_tasks == (duplicated.tasks[0],) + + +@pytest.mark.parametrize( + ("attestations", "status"), + [ + ((), _ContextCleanupStatus.UNCONFIRMED), + ((False,), _ContextCleanupStatus.FAILED), + ((True, True), _ContextCleanupStatus.UNCONFIRMED), + ], +) +def test_cleanup_requires_one_exact_trusted_artifact_attestation( + attestations: tuple[bool, ...], + status: _ContextCleanupStatus, +) -> None: + frames = _frames() + _bind_dispatches(frames) + assert frames.reconcile(_exact_evidence(frames)).status is _ContextReconciliationStatus.VERIFIED + + result = frames.close( + tuple( + _BackendClosureAttestation(frames.artifact_id, _BackendArtifactClass.CONTEXT_REQUEST, closed) + for closed in attestations + ) + ) + + assert result.status is status + assert frames.target_frame.empty + assert frames.context_frame.empty + + +def test_pre_bind_evidence_and_attestation_fail_closed() -> None: + frames = _frames() + + reconciliation = frames.reconcile(_exact_evidence(frames)) + cleanup = frames.close( + (_BackendClosureAttestation(frames.artifact_id, _BackendArtifactClass.CONTEXT_REQUEST, True),) + ) + + assert reconciliation.status is _ContextReconciliationStatus.GLOBAL_INVALID + assert cleanup.status is _ContextCleanupStatus.UNCONFIRMED + + +def test_discard_failure_containment_closes_before_best_effort_erasure( + monkeypatch: pytest.MonkeyPatch, +) -> None: + frames = _frames() + + def fail_erasure(self: object) -> None: + raise RuntimeError("erasure unavailable") + + monkeypatch.setattr(type(frames), "_erase_owned_state", fail_erasure) + + frames.contain_discard_failure() + + assert frames.target_frame.empty + assert frames.context_frame.empty + with pytest.raises(_WorkframeClosedError): + frames.expected_bindings() + with pytest.raises(_WorkframeClosedError): + _ = frames.artifact_id + + +@pytest.mark.parametrize( + "attestation", + [ + _BackendClosureAttestation( + _BackendArtifactId("foreign-artifact"), + _BackendArtifactClass.CONTEXT_REQUEST, + True, + ), + _BackendClosureAttestation( + _BackendArtifactId("artifact-a"), + _BackendArtifactClass.CONTEXT_REQUEST, + True, + schema_version=cast(_ContextSchemaVersion, "future-schema"), + ), + ], +) +def test_foreign_or_incompatible_cleanup_evidence_is_unconfirmed( + attestation: _BackendClosureAttestation, +) -> None: + frames = _frames() + _bind_dispatches(frames) + evidence = _exact_evidence(frames) + assert frames.reconcile(evidence).status is _ContextReconciliationStatus.VERIFIED + + assert frames.close((attestation,)).status is _ContextCleanupStatus.UNCONFIRMED + + +def test_mutated_context_payload_cannot_satisfy_a_compiled_binding() -> None: + frames = _frames() + _bind_dispatches(frames) + evidence = list(_exact_evidence(frames)) + evidence[0] = replace(evidence[0], payload_token=_ContextPayloadToken("foreign-payload")) + + result = frames.reconcile(tuple(evidence)) + + assert result.status is _ContextReconciliationStatus.LOCAL_INVALID + assert result.affected_tasks == (frames.tasks[0],) + + with pytest.raises(_WorkframeStateError): + _make_context_binding_evidence("binding", "owner", 0, "mutated-context") + + +def test_malformed_unhashable_binding_evidence_fails_closed_and_can_cleanup() -> None: + frames = _frames() + _bind_dispatches(frames) + evidence = list(_exact_evidence(frames)) + object.__setattr__(evidence[0], "binding_id", cast(_ContextBindingId, [])) + + result = frames.reconcile(tuple(evidence)) + cleanup = frames.close( + (_BackendClosureAttestation(frames.artifact_id, _BackendArtifactClass.CONTEXT_REQUEST, True),) + ) + + assert result.status is _ContextReconciliationStatus.GLOBAL_INVALID + assert cleanup.status is _ContextCleanupStatus.VERIFIED + + +def _frames(): + plan = _plan() + return _lower_context_workframes( + plan, + tuple(projection.owner_task for projection in plan.projections), + target_work_ids=("target-work-a", "target-work-b"), + identity_factory=_identities("binding-a0", "binding-a1", "binding-b0", "artifact-a"), + ) + + +def _exact_evidence(frames) -> tuple[_ContextBindingEvidence, ...]: + return tuple( + _make_context_binding_evidence( + row[COL_CONTEXT_BINDING_ID], + row[COL_CONTEXT_OWNER_WORK_ID], + row[COL_CONTEXT_ORDINAL], + row[COL_CONTEXT_TEXT], + ) + for _index, row in frames.context_frame.iterrows() + ) + + +def _dispatches_for(frames) -> tuple[_Dispatch, ...]: + return tuple( + _Dispatch( + _InvocationId("invocation"), + task, + _AttemptId(f"attempt-{index}"), + _RowToken(work_id.value), + ) + for index, (task, work_id) in enumerate(zip(frames.tasks, frames.target_work_ids(), strict=True)) + ) + + +def _bind_dispatches(frames) -> None: + frames.bind_dispatches(_dispatches_for(frames)) + + +def _identities(*values: str): + iterator: Iterator[str] = iter(values) + return lambda: next(iterator) + + +def _plan() -> _ContextPlan: + target_a = _TextDatum(_DatumId("target-a"), "alpha", _DatumPurpose.TARGET) + target_b = _TextDatum(_DatumId("target-b"), "beta", _DatumPurpose.TARGET) + context = _TextDatum(_DatumId("context-c"), "gamma", _DatumPurpose.CONTEXT_ONLY) + graph = _ProtectionGraph( + datums=(target_a, target_b, context), + links=(), + context_scopes=( + _ContextScope(target_a.id, (context.id, target_b.id)), + _ContextScope(target_b.id, (target_a.id,)), + ), + coherence_scopes=(_CoherenceScope((target_a.id,)), _CoherenceScope((target_b.id,))), + atomic_groups=(_AtomicGroup((target_a.id,)), _AtomicGroup((target_b.id,))), + ) + limits = _ContextLimits(2, 32, 4, 128) + contract = _ContextExecutionContract( + _ContextProfile.TARGET_CONTEXT_V1, + _ContextSchemaVersion.V1, + limits, + True, + _ContextOrdering.DECLARED, + (_BackendArtifactClass.CONTEXT_REQUEST,), + ) + capability = _ContextBackendCapability( + contract.profile, + contract.schema_version, + limits, + True, + contract.ordering, + contract.required_artifacts, + _RetentionPosture.DISABLED, + ) + result = _compile_context_plan( + graph, + accounting_limits=_AccountingLimits(8, 64, 256), + contract=contract, + capability=capability, + ) + assert isinstance(result, _ContextPlan) + return result diff --git a/tests/engine/execution/test_graph_runtime.py b/tests/engine/execution/test_graph_runtime.py new file mode 100644 index 00000000..9798a273 --- /dev/null +++ b/tests/engine/execution/test_graph_runtime.py @@ -0,0 +1,485 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import pickle +from dataclasses import FrozenInstanceError +from typing import Any, cast + +import pandas as pd +import pytest + +from anonymizer.config.anonymizer_config import AnonymizerConfig +from anonymizer.config.models import ModelSelection +from anonymizer.config.replace_strategies import Redact +from anonymizer.engine.constants import ( + COL_FINAL_ENTITIES, + COL_REPLACED_TEXT, + COL_REPLACEMENT_APPLICATION, + COL_TEXT, +) +from anonymizer.engine.execution.accounting_admission import _compile_accounting_plan +from anonymizer.engine.execution.accounting_outcomes import _GroupWithheld, _InvocationInconsistent, _InvocationLost +from anonymizer.engine.execution.accounting_plan import _AccountingLimits, _AccountingPlan +from anonymizer.engine.execution.graph import ( + _DatumId, + _ProtectionGraph, + _TextDatum, + _trivial_graph, +) +from anonymizer.engine.execution.graph_runtime import _AccountingGraphRuntime +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.execution.pandas_runtime import _PandasExecutionResult +from anonymizer.engine.execution.protection_service import ( + _GraphProtectionFailed, + _GraphProtectionResult, + _GraphProtectionSucceeded, + _RedactProtectionService, +) +from anonymizer.engine.private_row_verification import ( + _InvocationRowVerifier, +) + +_LIMITS = _AccountingLimits(max_datums=4, max_datum_bytes=64, max_graph_bytes=128) + + +def _graph(*texts: str) -> _ProtectionGraph: + return _trivial_graph(tuple(_TextDatum(_DatumId(f"datum-{index}"), text) for index, text in enumerate(texts))) + + +def _plan(*texts: str) -> _AccountingPlan: + compiled = _compile_accounting_plan(_graph(*texts), limits=_LIMITS) + assert isinstance(compiled, _AccountingPlan) + return compiled + + +class _SuccessfulBackend: + def __init__(self) -> None: + self.frame: pd.DataFrame | None = None + + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + self.frame = dataframe.copy() + detected = dataframe.assign(**{COL_FINAL_ENTITIES: [{"entities": []} for _ in range(len(dataframe))]}) + verifier.freeze_accepted_detections(detected) + final = verifier.finish(detected) + return _PandasExecutionResult( + dataframe=final, + failed_records=[], + terminal_outcomes=verifier.take_terminal_outcomes(), + result_row_tokens=verifier.take_result_order(), + ) + + +def _protect_release_row( + input_text: str, + row: dict[str, object], + model_selection: ModelSelection, +) -> tuple[_ProtectionGraph, _GraphProtectionResult]: + graph = _graph(input_text) + + class _StaticBackend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + detected = dataframe.assign(**{name: [value] for name, value in row.items()}) + verifier.freeze_accepted_detections(detected) + final = verifier.finish(detected) + return _PandasExecutionResult( + dataframe=final, + failed_records=[], + terminal_outcomes=verifier.take_terminal_outcomes(), + result_row_tokens=verifier.take_result_order(), + ) + + service = _RedactProtectionService(_AccountingGraphRuntime(_StaticBackend())) + plan = service.admit(graph, limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + result = service.protect( + plan, + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), model_selection), + ) + return graph, result + + +def test_trivial_graph_is_immutable_and_preserves_datum_order() -> None: + graph = _graph("first", "second") + + assert [datum.text for datum in graph.datums] == ["first", "second"] + with pytest.raises(FrozenInstanceError): + setattr(graph.datums[0], "text", "changed") + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(graph) + assert "first" not in repr(graph) + + +def test_graph_runtime_lowers_only_text_and_preserves_graph_identity( + stub_slim_model_selection: ModelSelection, +) -> None: + backend = _SuccessfulBackend() + graph = _graph("first", "second") + plan = _compile_accounting_plan(graph, limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + result = _AccountingGraphRuntime(backend).run( + plan, + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id, row[COL_TEXT]), + ) + + assert tuple(datum.id.value for datum in result.plan.datums) == ("datum-0", "datum-1") + assert backend.frame is not None + assert list(backend.frame.columns) == [COL_TEXT, "__anonymizer_private_row_correlation__"] + assert all(datum.id.value not in backend.frame.to_string() for datum in graph.datums) + + +def test_graph_runtime_rejects_rows_swapped_after_verification( + stub_slim_model_selection: ModelSelection, +) -> None: + class _PostVerificationSwapBackend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + detected = dataframe.assign(**{COL_FINAL_ENTITIES: [{"entities": []}, {"entities": []}]}) + verifier.freeze_accepted_detections(detected) + final = verifier.finish(detected) + return _PandasExecutionResult( + dataframe=final.iloc[::-1].reset_index(drop=True), + failed_records=[], + terminal_outcomes=verifier.take_terminal_outcomes(), + result_row_tokens=verifier.take_result_order(), + ) + + execution = _AccountingGraphRuntime(_PostVerificationSwapBackend()).run( + _plan("first", "second"), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + ) + + assert isinstance(execution.accounting.invocation, _InvocationInconsistent) + assert all(isinstance(group, _GroupWithheld) for group in execution.accounting.groups) + + +def test_graph_runtime_accounts_backend_failure_as_lost_without_content( + stub_slim_model_selection: ModelSelection, +) -> None: + secret = "backend-secret@example.test" + + class _FailingBackend: + def run(self, *_args: Any, **_kwargs: Any) -> _PandasExecutionResult: + raise RuntimeError(secret) + + execution = _AccountingGraphRuntime(_FailingBackend()).run( + _plan("input-secret@example.test"), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id, row[COL_TEXT]), + ) + + assert isinstance(execution.accounting.invocation, _InvocationLost) + assert secret not in repr(execution) + + +def test_graph_runtime_rejects_raw_graph_before_backend_effects( + stub_slim_model_selection: ModelSelection, +) -> None: + class _NeverRunsBackend: + def run(self, *_args: Any, **_kwargs: Any) -> _PandasExecutionResult: + raise AssertionError("raw graphs must not reach the backend") + + with pytest.raises(TypeError, match="private accounting plan"): + _AccountingGraphRuntime(_NeverRunsBackend()).run( + cast(_AccountingPlan, _graph("uncompiled")), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id, row[COL_TEXT]), + ) + + +def test_release_reconciles_reordered_terminal_outcomes_by_private_token( + stub_slim_model_selection: ModelSelection, +) -> None: + graph = _graph("first", "second") + + class _ReorderedBackend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + detected = dataframe.assign( + **{ + COL_REPLACED_TEXT: dataframe[COL_TEXT], + COL_FINAL_ENTITIES: [{"entities": []} for _ in range(len(dataframe))], + COL_REPLACEMENT_APPLICATION: [ + { + "targeted_span_count": 0, + "applied_span_count": 0, + "skipped_span_count": 0, + "skipped_span_label_counts": {}, + } + for _ in range(len(dataframe)) + ], + } + ) + verifier.freeze_accepted_detections(detected) + final = verifier.finish(detected).iloc[::-1].reset_index(drop=True) + tokens = tuple(reversed(verifier.take_result_order())) + return _PandasExecutionResult( + dataframe=final, + failed_records=[], + terminal_outcomes=tuple(reversed(verifier.take_terminal_outcomes())), + result_row_tokens=tokens, + ) + + service = _RedactProtectionService(_AccountingGraphRuntime(_ReorderedBackend())) + plan = service.admit(graph, limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + result = service.protect( + plan, + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + ) + + assert all(isinstance(outcome, _GraphProtectionSucceeded) for outcome in result.outcomes) + assert [outcome.output for outcome in result.outcomes if isinstance(outcome, _GraphProtectionSucceeded)] == [ + "first", + "second", + ] + + +def test_release_rejects_unchanged_authoritative_span_when_entity_value_case_differs( + stub_slim_model_selection: ModelSelection, +) -> None: + graph, result = _protect_release_row( + "Alice works here", + { + COL_REPLACED_TEXT: "Alice works here", + COL_FINAL_ENTITIES: { + "entities": [{"value": "alice", "label": "first_name", "start_position": 0, "end_position": 5}] + }, + COL_REPLACEMENT_APPLICATION: { + "targeted_span_count": 1, + "applied_span_count": 0, + "skipped_span_count": 1, + "skipped_span_label_counts": {"first_name": 1}, + }, + }, + stub_slim_model_selection, + ) + + assert result.outcomes == (_GraphProtectionFailed(graph.datums[0].id, "release", "datum"),) + + +@pytest.mark.parametrize( + ("entities", "output"), + [ + ( + [{"value": "alice", "label": "first_name", "start_position": 0, "end_position": 5}], + "[REDACTED_FIRST_NAME] works here", + ), + ( + [{"value": "Alice", "label": "first_name", "start_position": 0, "end_position": 5}], + "Alice works here", + ), + ], +) +def test_release_rejects_forged_success_accounting( + stub_slim_model_selection: ModelSelection, + entities: list[dict[str, object]], + output: str, +) -> None: + graph, result = _protect_release_row( + "Alice works here", + { + COL_REPLACED_TEXT: output, + COL_FINAL_ENTITIES: {"entities": entities}, + COL_REPLACEMENT_APPLICATION: { + "targeted_span_count": 1, + "applied_span_count": 1, + "skipped_span_count": 0, + "skipped_span_label_counts": {}, + }, + }, + stub_slim_model_selection, + ) + + assert result.outcomes == (_GraphProtectionFailed(graph.datums[0].id, "release", "datum"),) + + +@pytest.mark.parametrize( + "entities", + [ + [{"value": "Alice", "label": "first_name", "start_position": 0}], + [{"value": "Alice", "label": "first_name", "start_position": -1, "end_position": 5}], + [{"value": "Alice", "label": "first_name", "start_position": 0, "end_position": 99}], + [ + {"value": "Alice", "label": "first_name", "start_position": 0, "end_position": 5}, + {"value": "lice ", "label": "alias", "start_position": 1, "end_position": 6}, + ], + ], +) +def test_release_rejects_malformed_authoritative_spans( + stub_slim_model_selection: ModelSelection, + entities: list[dict[str, object]], +) -> None: + targeted = len(entities) + graph, result = _protect_release_row( + "Alice works here", + { + COL_REPLACED_TEXT: "[REDACTED] works here", + COL_FINAL_ENTITIES: {"entities": entities}, + COL_REPLACEMENT_APPLICATION: { + "targeted_span_count": targeted, + "applied_span_count": targeted, + "skipped_span_count": 0, + "skipped_span_label_counts": {}, + }, + }, + stub_slim_model_selection, + ) + + assert result.outcomes == (_GraphProtectionFailed(graph.datums[0].id, "release", "datum"),) + + +@pytest.mark.parametrize("entity_label", [None, "", 7]) +def test_release_rejects_malformed_accepted_entity_labels( + stub_slim_model_selection: ModelSelection, + entity_label: object, +) -> None: + graph, result = _protect_release_row( + "Alice works here", + { + COL_REPLACED_TEXT: "[REDACTED] works here", + COL_FINAL_ENTITIES: { + "entities": [{"value": "Alice", "label": entity_label, "start_position": 0, "end_position": 5}] + }, + COL_REPLACEMENT_APPLICATION: { + "targeted_span_count": 1, + "applied_span_count": 1, + "skipped_span_count": 0, + "skipped_span_label_counts": {}, + }, + }, + stub_slim_model_selection, + ) + + assert result.outcomes == (_GraphProtectionFailed(graph.datums[0].id, "release", "datum"),) + + +@pytest.mark.parametrize( + "application", + [ + None, + {"targeted_span_count": 1, "applied_span_count": 1, "skipped_span_count": 0}, + { + "targeted_span_count": True, + "applied_span_count": 1, + "skipped_span_count": 0, + "skipped_span_label_counts": {}, + }, + { + "targeted_span_count": 2, + "applied_span_count": 2, + "skipped_span_count": 0, + "skipped_span_label_counts": {}, + }, + { + "targeted_span_count": 1, + "applied_span_count": 0, + "skipped_span_count": 1, + "skipped_span_label_counts": {"first_name": 1}, + }, + ], +) +def test_release_rejects_malformed_or_incomplete_replacement_accounting( + stub_slim_model_selection: ModelSelection, + application: object, +) -> None: + graph, result = _protect_release_row( + "Alice works here", + { + COL_REPLACED_TEXT: "[REDACTED_FIRST_NAME] works here", + COL_FINAL_ENTITIES: { + "entities": [{"value": "Alice", "label": "first_name", "start_position": 0, "end_position": 5}] + }, + COL_REPLACEMENT_APPLICATION: application, + }, + stub_slim_model_selection, + ) + + assert result.outcomes == (_GraphProtectionFailed(graph.datums[0].id, "release", "datum"),) + + +def test_release_accepts_complete_exact_case_redaction(stub_slim_model_selection: ModelSelection) -> None: + graph, result = _protect_release_row( + "Alice works here", + { + COL_REPLACED_TEXT: "[REDACTED_FIRST_NAME] works here", + COL_FINAL_ENTITIES: { + "entities": [{"value": "Alice", "label": "first_name", "start_position": 0, "end_position": 5}] + }, + COL_REPLACEMENT_APPLICATION: { + "targeted_span_count": 1, + "applied_span_count": 1, + "skipped_span_count": 0, + "skipped_span_label_counts": {}, + }, + }, + stub_slim_model_selection, + ) + + assert result.outcomes == (_GraphProtectionSucceeded(graph.datums[0].id, "[REDACTED_FIRST_NAME] works here", True),) + + +def test_release_accepts_unchanged_no_detection_with_zero_accounting( + stub_slim_model_selection: ModelSelection, +) -> None: + graph, result = _protect_release_row( + "plain text", + { + COL_REPLACED_TEXT: "plain text", + COL_FINAL_ENTITIES: {"entities": []}, + COL_REPLACEMENT_APPLICATION: { + "targeted_span_count": 0, + "applied_span_count": 0, + "skipped_span_count": 0, + "skipped_span_label_counts": {}, + }, + }, + stub_slim_model_selection, + ) + + assert result.outcomes == (_GraphProtectionSucceeded(graph.datums[0].id, "plain text", False),) diff --git a/tests/engine/execution/test_hierarchical_accounting.py b/tests/engine/execution/test_hierarchical_accounting.py new file mode 100644 index 00000000..638ac2de --- /dev/null +++ b/tests/engine/execution/test_hierarchical_accounting.py @@ -0,0 +1,1949 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import os +import pickle +import random +from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor +from dataclasses import FrozenInstanceError, replace +from hashlib import sha256 +from inspect import signature +from itertools import count, product +from json import dumps, loads +from pathlib import Path +from threading import Barrier +from typing import Callable, cast + +import pandas as pd +import pytest + +from anonymizer.config.anonymizer_config import AnonymizerConfig +from anonymizer.config.models import ModelSelection +from anonymizer.config.replace_strategies import Redact +from anonymizer.engine.constants import COL_FINAL_ENTITIES, COL_TEXT +from anonymizer.engine.execution.accounting_admission import ( + _AccountingAdmissionCode, + _AccountingRejected, + _compile_accounting_plan, +) +from anonymizer.engine.execution.accounting_evidence import _AttemptId, _Dispatch, _RowToken, _SuccessRecord +from anonymizer.engine.execution.accounting_ledger import ( + _AccountingLedger, + _EvidenceAcceptance, + _LedgerClosedError, + _LedgerStateError, +) +from anonymizer.engine.execution.accounting_outcomes import ( + _AccountingResult, + _CauseCode, + _DatumFailed, + _DatumQualified, + _DependencySatisfied, + _GroupReleased, + _GroupWithheld, + _InvocationCancelled, + _InvocationCompleted, + _InvocationFailed, + _InvocationInconsistent, + _InvocationLost, + _StageFailed, + _StageSucceeded, + _TaskBlocked, + _TaskCancelled, + _TaskFailed, + _TaskInconsistent, + _TaskLost, + _TaskSucceeded, +) +from anonymizer.engine.execution.accounting_plan import ( + _AccountingLimits, + _AccountingPlan, + _DatumTaskSubject, + _TaskKey, +) +from anonymizer.engine.execution.accounting_release import _qualify_release +from anonymizer.engine.execution.graph import ( + _AtomicGroup, + _CoherenceScope, + _ContextScope, + _DatumDependency, + _DatumId, + _DatumLink, + _ProtectionGraph, + _RelationKind, + _TextDatum, + _trivial_graph, +) +from anonymizer.engine.execution.graph_runtime import ( + _AccountingGraphAdmissionError, + _AccountingGraphExecution, + _AccountingGraphRuntime, +) +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.execution.pandas_runtime import _PandasExecutionResult +from anonymizer.engine.execution.protection_service import _RedactCandidate, _RedactProtectionService +from anonymizer.engine.ndd.adapter import FailedRecord, _FailedRowEvidence +from anonymizer.engine.private_row_verification import ( + PRIVATE_CORRELATION_COLUMN, + _InvocationRowVerifier, + _TerminalOutcome, +) +from tests.engine.execution.phase4_reference_model import ( + ReferenceCancellationRequest, + ReferenceContradiction, + ReferenceCorruptEvidence, + ReferenceCorruption, + ReferenceDeclaration, + ReferenceDispatch, + ReferenceFailure, + ReferenceHierarchyResult, + ReferenceInvocationOutcome, + ReferenceObservation, + ReferenceResultConstructionFailure, + ReferenceStopAcknowledgement, + ReferenceSuccess, + ReferenceTaskOutcome, + ReferenceTransportLoss, + acyclic_dependencies, + flat_partitions, + reduce_observations, + reduce_reference, + streaming_conformance_cases, +) + +_SINGLETON = ReferenceDeclaration(("datum-a",), (), (("datum-a",),)) +_LIMITS = _AccountingLimits(max_datums=4, max_datum_bytes=64, max_graph_bytes=128) +_CONFORMANCE_MANIFEST = loads(Path(__file__).with_name("phase4_conformance_manifest.json").read_text()) + + +def _crash_backend_worker(marker: str) -> None: + """Run the test backend effect in a worker that dies before replying.""" + Path(marker).write_text("started") + os._exit(17) + + +def _compiled(graph: _ProtectionGraph) -> _AccountingPlan: + result = _compile_accounting_plan(graph, limits=_LIMITS) + assert isinstance(result, _AccountingPlan) + return result + + +def _datum_id(task: _TaskKey) -> _DatumId: + assert isinstance(task.subject, _DatumTaskSubject) + return task.subject.datum_id + + +def test_one_shot_ledger_reconciles_complete_singleton_invocation() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + identities = iter(("invocation", "attempt", "row-token")) + ledger = _AccountingLedger(plan, identity_factory=lambda: next(identities)) + + ledger.open() + ready = ledger.ready_tasks() + assert ready == plan.tasks + dispatch = ledger.dispatch(ready[0]) + ledger.accept_success(dispatch, "protected-a") + result = ledger.finish() + + assert type(result.invocation).__name__ == "_InvocationCompleted" + assert tuple(type(outcome).__name__ for outcome in result.tasks) == ("_TaskSucceeded",) + assert tuple(type(outcome).__name__ for outcome in result.datums) == ("_DatumQualified",) + assert tuple(type(outcome).__name__ for outcome in result.groups) == ("_GroupReleased",) + group = result.groups[0] + assert isinstance(group, _GroupReleased) + assert group.outputs == ((plan.datums[0].id, "protected-a"),) + with pytest.raises(_LedgerClosedError): + ledger.finish() + + +def test_synthetic_multistage_plan_pipelines_per_datum_without_global_stage_dispatch_barrier() -> None: + graph = _graph("a", "b") + graph = replace(graph, dependencies=(_dependency(graph, "a", "b"),)) + plan = _compile_accounting_plan(graph, limits=_LIMITS, stages=("detect", "protect")) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + + observed_frontiers: list[tuple[tuple[str, str], ...]] = [] + while ready := ledger.ready_tasks(): + observed_frontiers.append(tuple((task.stage.value, _datum_id(task).value) for task in ready)) + for task in ready: + ledger.accept_success(ledger.dispatch(task), f"{task.stage.value}-{_datum_id(task).value}") + result = ledger.finish() + + assert observed_frontiers == [ + (("detect", "a"),), + (("protect", "a"),), + (("detect", "b"),), + (("protect", "b"),), + ] + assert len(result.tasks) == len(plan.tasks) == 4 + assert len({outcome.task for outcome in result.tasks}) == 4 + assert len(result.datums) == len(plan.datums) == 2 + assert len(result.dependencies) == len(plan.dependencies) == 1 + assert len(result.stages) == len(plan.stages) == 2 + assert len(result.groups) == len(plan.atomic_groups) == 2 + assert all(isinstance(stage, _StageSucceeded) for stage in result.stages) + + +def test_one_shot_ledger_allows_exactly_one_concurrent_open() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + + ready = Barrier(3) + + def _open() -> str: + ready.wait() + try: + ledger.open() + except _LedgerStateError: + return "rejected" + return "opened" + + with ThreadPoolExecutor(max_workers=2) as executor: + futures = tuple(executor.submit(_open) for _ordinal in range(2)) + ready.wait() + outcomes = tuple(future.result() for future in futures) + + assert sorted(outcomes) == ["opened", "rejected"] + + +def test_one_shot_ledger_allows_exactly_one_concurrent_finish() -> None: + ledger = _ledger(_compiled(_graph("a"))) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + ledger.accept_success(dispatch, "protected") + ready = Barrier(3) + + def _finish() -> str: + ready.wait() + try: + ledger.finish() + except _LedgerClosedError: + return "rejected" + return "published" + + with ThreadPoolExecutor(max_workers=2) as executor: + futures = tuple(executor.submit(_finish) for _ordinal in range(2)) + ready.wait() + outcomes = tuple(future.result() for future in futures) + + assert sorted(outcomes) == ["published", "rejected"] + + +def test_private_plan_evidence_and_result_are_nonserializable_and_content_safe() -> None: + canary = "private-canary@example.test" + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + ledger.accept_success(dispatch, canary) + result = ledger.finish() + + for private_value in (plan, dispatch, result): + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(private_value) + assert canary not in repr(private_value) + + +def _ledger(plan: _AccountingPlan) -> _AccountingLedger[str]: + identities = count() + return _AccountingLedger(plan, identity_factory=lambda: f"private-{next(identities)}") + + +def test_failure_blocks_dependents_without_dispatch_or_retry_and_isolates_independent_group() -> None: + graph = _graph("a", "b", "c") + graph = replace(graph, dependencies=(_dependency(graph, "a", "b"),)) + plan = _compile_accounting_plan(graph, limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + ready = ledger.ready_tasks() + assert tuple(_datum_id(task).value for task in ready) == ("a", "c") + by_datum = {_datum_id(task).value: task for task in ready} + dispatch_a = ledger.dispatch(by_datum["a"]) + dispatch_c = ledger.dispatch(by_datum["c"]) + ledger.accept_failure(dispatch_a) + ledger.accept_success(dispatch_c, "protected-c") + + assert ledger.ready_tasks() == () + with pytest.raises(_LedgerStateError): + ledger.dispatch(by_datum["a"]) + result = ledger.finish() + + assert isinstance(result.invocation, _InvocationCompleted) + assert any(isinstance(outcome, _TaskBlocked) and _datum_id(outcome.task).value == "b" for outcome in result.tasks) + released = tuple(outcome for outcome in result.groups if isinstance(outcome, _GroupReleased)) + assert tuple(output[0].value for outcome in released for output in outcome.outputs) == ("c",) + + +def test_late_atomic_peer_failure_withholds_already_succeeded_dependent_at_fixed_point() -> None: + graph = _graph("a", "b", "c", "d", "e") + graph = replace( + graph, + dependencies=(_dependency(graph, "a", "c"),), + atomic_groups=( + _group(graph, "a", "b"), + _group(graph, "c", "d"), + _group(graph, "e"), + ), + ) + plan = _compile_accounting_plan(graph, limits=replace(_LIMITS, max_datums=5, max_graph_bytes=256)) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + initial = {_datum_id(task).value: task for task in ledger.ready_tasks()} + ledger.accept_success(ledger.dispatch(initial["a"]), "protected-a") + after_prerequisite = {_datum_id(task).value: task for task in ledger.ready_tasks()} + ledger.accept_success(ledger.dispatch(after_prerequisite["c"]), "protected-c") + ledger.accept_failure(ledger.dispatch(after_prerequisite["b"])) + ledger.accept_success(ledger.dispatch(after_prerequisite["d"]), "protected-d") + ledger.accept_success(ledger.dispatch(after_prerequisite["e"]), "protected-e") + + result = ledger.finish() + reference = reduce_observations( + ReferenceDeclaration( + ("a", "b", "c", "d", "e"), + (("a", "c"),), + (("a", "b"), ("c", "d"), ("e",)), + ), + ( + ReferenceDispatch(("protect", "a")), + ReferenceSuccess(("protect", "a")), + ReferenceDispatch(("protect", "c")), + ReferenceSuccess(("protect", "c")), + ReferenceDispatch(("protect", "b")), + ReferenceFailure(("protect", "b")), + ReferenceDispatch(("protect", "d")), + ReferenceSuccess(("protect", "d")), + ReferenceDispatch(("protect", "e")), + ReferenceSuccess(("protect", "e")), + ), + ) + released_ids = tuple( + datum_id.value + for group in result.groups + if isinstance(group, _GroupReleased) + for datum_id, _candidate in group.outputs + ) + + assert any(isinstance(outcome, _TaskSucceeded) and _datum_id(outcome.task).value == "c" for outcome in result.tasks) + assert released_ids == ("e",) + assert ( + tuple( + ((outcome.task.stage.value, _datum_id(outcome.task).value), ReferenceTaskOutcome.SUCCEEDED) + if isinstance(outcome, _TaskSucceeded) + else ((outcome.task.stage.value, _datum_id(outcome.task).value), ReferenceTaskOutcome.FAILED) + for outcome in result.tasks + if isinstance(outcome, (_TaskSucceeded, _TaskFailed)) + ) + == reference.tasks + ) + assert ( + tuple( + (outcome.datum_id.value, ReferenceTaskOutcome.SUCCEEDED) + if isinstance(outcome, _DatumQualified) + else (outcome.datum_id.value, ReferenceTaskOutcome.FAILED) + for outcome in result.datums + if isinstance(outcome, (_DatumQualified, _DatumFailed)) + ) + == reference.datums + ) + assert ( + tuple( + ((outcome.dependency.prerequisite.value, outcome.dependency.dependent.value), True) + for outcome in result.dependencies + if isinstance(outcome, _DependencySatisfied) + ) + == reference.dependencies + ) + assert isinstance(result.stages[0], _StageFailed) + assert reference.stages == (("protect", ReferenceTaskOutcome.FAILED),) + assert reference.released_groups == frozenset((frozenset(("e",)),)) + assert reference.invocation is ReferenceInvocationOutcome.COMPLETED + + +def test_post_dispatch_cancellation_without_stop_evidence_is_lost() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + + ledger.request_cancellation() + result = ledger.finish() + reference = reduce_observations( + ReferenceDeclaration(("a",), (), (("a",),)), + ( + ReferenceDispatch(("protect", "a")), + ReferenceCancellationRequest(), + ), + ) + + assert isinstance(result.invocation, _InvocationLost) + assert isinstance(result.tasks[0], _TaskLost) + assert isinstance(result.groups[0], _GroupWithheld) + assert reference.invocation is ReferenceInvocationOutcome.LOST + assert reference.tasks[0][1] is ReferenceTaskOutcome.LOST + assert ledger.acknowledge_stop(dispatch) is _EvidenceAcceptance.REJECTED_STALE + + +def test_pre_dispatch_cancellation_closes_without_dispatch() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + assert ledger.ready_tasks() == plan.tasks + + ledger.request_cancellation() + result = ledger.finish() + + assert isinstance(result.invocation, _InvocationCancelled) + assert isinstance(result.tasks[0], _TaskCancelled) + assert isinstance(result.groups[0], _GroupWithheld) + + +def test_cancellation_after_success_but_before_publication_embargoes_output() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + ledger.accept_success(dispatch, "protected") + + ledger.request_cancellation() + result = ledger.finish() + + assert isinstance(result.invocation, _InvocationCancelled) + assert isinstance(result.tasks[0], _TaskSucceeded) + assert isinstance(result.groups[0], _GroupWithheld) + + +def test_terminal_success_precedes_late_stop_and_post_publication_cancellation() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + ledger.accept_success(dispatch, "protected") + + assert ledger.acknowledge_stop(dispatch) is _EvidenceAcceptance.REJECTED_STALE + result = ledger.finish() + ledger.request_cancellation() + + assert isinstance(result.invocation, _InvocationCompleted) + assert isinstance(result.groups[0], _GroupReleased) + + +def test_trusted_stop_acknowledgement_before_late_success_is_cancelled() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + ledger.request_cancellation() + + assert ledger.acknowledge_stop(dispatch) is _EvidenceAcceptance.ACCEPTED + assert ledger.accept_success(dispatch, "late") is _EvidenceAcceptance.REJECTED_STALE + result = ledger.finish() + reference = reduce_observations( + ReferenceDeclaration(("a",), (), (("a",),)), + ( + ReferenceDispatch(("protect", "a")), + ReferenceCancellationRequest(), + ReferenceStopAcknowledgement(("protect", "a")), + ReferenceSuccess(("protect", "a")), + ), + ) + + assert isinstance(result.invocation, _InvocationCancelled) + assert isinstance(result.tasks[0], _TaskCancelled) + assert isinstance(result.groups[0], _GroupWithheld) + assert reference.invocation is ReferenceInvocationOutcome.CANCELLED + assert reference.tasks[0][1] is ReferenceTaskOutcome.CANCELLED + + +def test_terminal_replay_is_idempotent_but_conflicting_late_evidence_is_stale() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + + assert ledger.accept_success(dispatch, "protected") is _EvidenceAcceptance.ACCEPTED + assert ledger.accept_success(dispatch, "protected") is _EvidenceAcceptance.IDEMPOTENT_STALE + assert ledger.accept_failure(dispatch) is _EvidenceAcceptance.REJECTED_STALE + result = ledger.finish() + + assert isinstance(result.invocation, _InvocationCompleted) + assert isinstance(result.groups[0], _GroupReleased) + + +def test_result_construction_failure_closes_exhaustively_without_group_output() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + ledger.accept_success(dispatch, "protected") + + def _fail_release(_datum_id: _DatumId, _candidate: str) -> bool: + raise RuntimeError("private-result-content") + + result = ledger.finish(datum_release_predicate=_fail_release) + + assert isinstance(result.invocation, _InvocationFailed) + assert isinstance(result.groups[0], _GroupWithheld) + assert "private-result-content" not in repr(result) + with pytest.raises(_LedgerClosedError): + ledger.finish() + + +def test_trusted_batch_missing_record_is_local_inconsistency() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + + ledger.reconcile((dispatch,), (), trusted_run_record=True) + result = ledger.finish() + + assert isinstance(result.invocation, _InvocationCompleted) + assert isinstance(result.tasks[0], _TaskInconsistent) + assert tuple(cause.code for cause in result.tasks[0].causes) == (_CauseCode.MISSING,) + assert isinstance(result.groups[0], _GroupWithheld) + + +def test_foreign_row_token_is_invocation_global_inconsistency() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + foreign = replace(dispatch, row_token=_RowToken("foreign")) + + ledger.reconcile((dispatch,), (_SuccessRecord(foreign, "protected"),), trusted_run_record=True) + result = ledger.finish() + + assert isinstance(result.invocation, _InvocationInconsistent) + assert isinstance(result.tasks[0], _TaskInconsistent) + assert tuple(cause.code for cause in result.tasks[0].causes) == (_CauseCode.FOREIGN,) + assert isinstance(result.groups[0], _GroupWithheld) + + +def test_direct_foreign_evidence_before_terminal_acceptance_is_global_inconsistency() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + foreign = replace(dispatch, row_token=_RowToken("foreign")) + + assert ledger.accept_success(foreign, "protected") is _EvidenceAcceptance.REJECTED_STALE + result = ledger.finish() + + assert isinstance(result.invocation, _InvocationInconsistent) + assert isinstance(result.tasks[0], _TaskInconsistent) + assert tuple(cause.code for cause in result.tasks[0].causes) == (_CauseCode.FOREIGN,) + + +def test_duplicate_expected_dispatch_identity_is_global_inconsistency() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + + ledger.reconcile((dispatch, dispatch), (_SuccessRecord(dispatch, "protected"),), trusted_run_record=True) + result = ledger.finish() + + assert isinstance(result.invocation, _InvocationInconsistent) + assert isinstance(result.tasks[0], _TaskInconsistent) + assert tuple(cause.code for cause in result.tasks[0].causes) == (_CauseCode.DUPLICATE,) + assert isinstance(result.groups[0], _GroupWithheld) + + +@pytest.mark.parametrize( + ("mutation", "expected"), + [ + ( + lambda dispatch: replace(dispatch, attempt_id=_AttemptId("unknown"), row_token=_RowToken("unknown")), + "unknown", + ), + (lambda dispatch: replace(dispatch, attempt_id=_AttemptId("stale")), "stale"), + ], +) +def test_unknown_and_stale_attempt_evidence_keep_exact_causes( + mutation: Callable[[_Dispatch], _Dispatch], + expected: str, +) -> None: + plan = _compiled(_graph("a")) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + + ledger.reconcile((dispatch,), (_SuccessRecord(mutation(dispatch), "candidate"),), trusted_run_record=True) + result = ledger.finish() + + assert isinstance(result.tasks[0], _TaskInconsistent) + assert tuple(cause.code.value for cause in result.tasks[0].causes) == (expected,) + + +def test_swapped_valid_dispatch_evidence_keeps_exact_cause() -> None: + plan = _compiled(_graph("a", "b")) + ledger = _ledger(plan) + ledger.open() + first, second = tuple(ledger.dispatch(task) for task in ledger.ready_tasks()) + swapped = replace(first, task=second.task, row_token=second.row_token) + + ledger.reconcile((first, second), (_SuccessRecord(swapped, "candidate"),), trusted_run_record=True) + result = ledger.finish() + + assert all(isinstance(task, _TaskInconsistent) for task in result.tasks) + for task in result.tasks: + assert isinstance(task, _TaskInconsistent) + assert tuple(cause.code.value for cause in task.causes) == ("swapped",) + + +def test_ledger_rejects_opaque_identity_collision_before_dispatch() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger: _AccountingLedger[str] = _AccountingLedger(plan, identity_factory=lambda: "collision") + ledger.open() + + with pytest.raises(_LedgerStateError): + ledger.dispatch(ledger.ready_tasks()[0]) + + assert ledger.ready_tasks() == plan.tasks + + +def test_plan_mismatch_is_invocation_global_inconsistency() -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + ledger.dispatch(ledger.ready_tasks()[0]) + + ledger.mark_inconsistent(_CauseCode.PLAN_MISMATCH) + result = ledger.finish() + + assert isinstance(result.invocation, _InvocationInconsistent) + assert isinstance(result.tasks[0], _TaskInconsistent) + assert isinstance(result.groups[0], _GroupWithheld) + + +def test_row_verifier_binds_ledger_owned_tokens_without_regenerating_identity() -> None: + assert "correlations" in signature(_InvocationRowVerifier).parameters + frame = pd.DataFrame({COL_TEXT: ["first", "second"]}) + + verifier = _InvocationRowVerifier(frame, correlations=("ledger-a", "ledger-b")) + bound = verifier.bind(frame) + + assert tuple(bound[PRIVATE_CORRELATION_COLUMN]) == ("ledger-a", "ledger-b") + + +def test_accounting_runtime_executes_ready_frontiers_and_preserves_datum_identity( + stub_slim_model_selection: ModelSelection, +) -> None: + frames: list[tuple[str, ...]] = [] + + class _Backend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + frames.append(tuple(dataframe[COL_TEXT])) + detected = dataframe.assign(final_entities=[{"entities": []} for _ in range(len(dataframe))]) + verifier.freeze_accepted_detections(detected) + final = verifier.finish(detected) + return _PandasExecutionResult( + dataframe=final, + failed_records=[], + terminal_outcomes=verifier.take_terminal_outcomes(), + result_row_tokens=verifier.take_result_order(), + ) + + graph = _graph("a", "b", "c") + graph = replace(graph, dependencies=(_dependency(graph, "a", "b"),)) + plan = _compile_accounting_plan(graph, limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + runtime = _AccountingGraphRuntime(_Backend()) + + execution = runtime.run( + plan, + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + ) + + assert frames == [("synthetic a", "synthetic c"), ("synthetic b",)] + assert isinstance(execution.accounting.invocation, _InvocationCompleted) + assert tuple( + output[1][0] + for group in execution.accounting.groups + if isinstance(group, _GroupReleased) + for output in group.outputs + ) == ("a", "b", "c") + + +def test_accounting_runtime_rejects_direct_or_tampered_plan_before_effects( + stub_slim_model_selection: ModelSelection, +) -> None: + calls = 0 + + class _Backend: + def run(self, *_args: object, **_kwargs: object) -> _PandasExecutionResult: + nonlocal calls + calls += 1 + raise AssertionError("backend must not run") + + compiled = _compiled(_graph("a")) + direct = _AccountingPlan( + compiled.datums, + compiled.stages, + compiled.tasks, + compiled.dependencies, + compiled.atomic_groups, + compiled.topological_datums, + ) + tampered = _compiled(_graph("a")) + object.__setattr__(tampered, "topological_datums", ()) + nested_tampered = _compiled(_graph("a")) + object.__setattr__(nested_tampered.datums[0], "text", "nested-tamper-canary") + runtime = _AccountingGraphRuntime(_Backend()) + invocation = _CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection) + + for plan in (direct, tampered, nested_tampered): + with pytest.raises(_AccountingGraphAdmissionError): + runtime.run( + plan, + invocation=invocation, + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + ) + + assert calls == 0 + + +def test_accounting_runtime_classifies_verifier_corruption_as_inconsistent( + stub_slim_model_selection: ModelSelection, +) -> None: + class _CorruptingBackend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + corrupted = dataframe.assign(**{PRIVATE_CORRELATION_COLUMN: ["foreign-token"]}) + verifier.freeze_accepted_detections(corrupted.assign(**{COL_FINAL_ENTITIES: [{"entities": []}]})) + raise AssertionError("verifier must reject foreign correlation") + + execution = _AccountingGraphRuntime(_CorruptingBackend()).run( + _compiled(_graph("a")), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + ) + + assert isinstance(execution.accounting.invocation, _InvocationInconsistent) + assert isinstance(execution.accounting.tasks[0], _TaskInconsistent) + + +def test_accounting_runtime_accepts_only_explicit_trusted_stop_evidence( + stub_slim_model_selection: ModelSelection, +) -> None: + class _StoppedBackend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + verifier.abort(cancelled=True) + terminal_outcomes = verifier.take_terminal_outcomes() + token = terminal_outcomes[0][0] + return _PandasExecutionResult( + dataframe=pd.DataFrame(), + failed_records=[], + terminal_outcomes=terminal_outcomes, + trusted_stop_tokens=(token,), + ) + + execution = _AccountingGraphRuntime(_StoppedBackend()).run( + _compiled(_graph("a")), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + ) + + assert isinstance(execution.accounting.invocation, _InvocationCancelled) + assert isinstance(execution.accounting.tasks[0], _TaskCancelled) + assert isinstance(execution.accounting.groups[0], _GroupWithheld) + + +def test_accounting_runtime_attributes_failure_by_opaque_token_and_releases_independent_group( + stub_slim_model_selection: ModelSelection, +) -> None: + public_failure = FailedRecord("content-derived-public-id", "replace", "dropped") + + class _PartialBackend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + first = dataframe[PRIVATE_CORRELATION_COLUMN].iloc[0] + surviving = dataframe.iloc[[1]].assign(**{COL_FINAL_ENTITIES: [{"entities": []}]}) + verifier.freeze_accepted_detections(surviving) + final = verifier.finish(surviving) + return _PandasExecutionResult( + dataframe=final, + failed_records=[public_failure], + terminal_outcomes=verifier.take_terminal_outcomes(), + result_row_tokens=verifier.take_result_order(), + failed_row_evidence=(_FailedRowEvidence(first, public_failure),), + ) + + execution = _AccountingGraphRuntime(_PartialBackend()).run( + _compiled(_graph("a", "b")), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + ) + + assert isinstance(execution.accounting.tasks[0], _TaskFailed) + assert isinstance(execution.accounting.tasks[1], _TaskSucceeded) + released = tuple( + datum_id.value + for group in execution.accounting.groups + if isinstance(group, _GroupReleased) + for datum_id, _candidate in group.outputs + ) + assert released == ("b",) + assert execution.failed_records == (public_failure,) + + +def test_accounting_runtime_rejects_unbound_failed_record_evidence( + stub_slim_model_selection: ModelSelection, +) -> None: + public_failure = FailedRecord("public-a", "replace", "dropped") + unrelated_failure = FailedRecord("public-b", "replace", "dropped") + + class _MismatchedBackend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + first = dataframe[PRIVATE_CORRELATION_COLUMN].iloc[0] + surviving = dataframe.iloc[[1]].assign(**{COL_FINAL_ENTITIES: [{"entities": []}]}) + verifier.freeze_accepted_detections(surviving) + final = verifier.finish(surviving) + return _PandasExecutionResult( + dataframe=final, + failed_records=[public_failure], + terminal_outcomes=verifier.take_terminal_outcomes(), + result_row_tokens=verifier.take_result_order(), + failed_row_evidence=(_FailedRowEvidence(first, unrelated_failure),), + ) + + execution = _AccountingGraphRuntime(_MismatchedBackend()).run( + _compiled(_graph("a", "b")), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + ) + + assert isinstance(execution.accounting.invocation, _InvocationInconsistent) + assert all(isinstance(group, _GroupWithheld) for group in execution.accounting.groups) + + +def test_accounting_runtime_blocks_dependent_before_effect_when_prerequisite_fails_release( + stub_slim_model_selection: ModelSelection, +) -> None: + frames: list[tuple[str, ...]] = [] + + class _Backend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + frames.append(tuple(dataframe[COL_TEXT])) + detected = dataframe.assign(**{COL_FINAL_ENTITIES: [{"entities": []}] * len(dataframe)}) + verifier.freeze_accepted_detections(detected) + final = verifier.finish(detected) + return _PandasExecutionResult( + dataframe=final, + failed_records=[], + terminal_outcomes=verifier.take_terminal_outcomes(), + result_row_tokens=verifier.take_result_order(), + ) + + graph = _graph("a", "b") + graph = replace(graph, dependencies=(_dependency(graph, "a", "b"),)) + plan = _compile_accounting_plan(graph, limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + runtime = _AccountingGraphRuntime(_Backend()) + + execution = runtime.run( + plan, + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + datum_release_predicate=lambda datum_id, _candidate: datum_id.value != "a", + ) + + assert frames == [("synthetic a",)] + assert any(isinstance(outcome, _TaskBlocked) for outcome in execution.accounting.tasks) + + +def test_accounting_admission_rejects_cycle_before_backend_effect( + stub_slim_model_selection: ModelSelection, +) -> None: + calls = 0 + + class _Backend: + def run(self, *_args: object, **_kwargs: object) -> _AccountingGraphExecution[_RedactCandidate]: + nonlocal calls + calls += 1 + raise AssertionError + + graph = _graph("a", "b") + graph = replace( + graph, + dependencies=(_dependency(graph, "a", "b"), _dependency(graph, "b", "a")), + ) + + result = _RedactProtectionService(_Backend()).admit(graph, limits=_LIMITS) + + assert isinstance(result, _AccountingRejected) + assert calls == 0 + + +def test_accounting_runtime_classifies_worker_process_death_as_lost( + tmp_path: Path, + stub_slim_model_selection: ModelSelection, +) -> None: + marker = tmp_path / "backend-worker-started" + calls = 0 + + class _CrashableBackend: + def run(self, *_args: object, **_kwargs: object) -> _PandasExecutionResult: + nonlocal calls + calls += 1 + with ProcessPoolExecutor(max_workers=1) as executor: + executor.submit(_crash_backend_worker, str(marker)).result(timeout=10) + raise AssertionError("crashable worker must not report success") + + execution = _AccountingGraphRuntime(_CrashableBackend()).run( + _compiled(_graph("a")), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + ) + reference = reduce_reference( + _SINGLETON, + {"datum-a": ReferenceTaskOutcome.LOST}, + ) + + assert isinstance(execution.accounting.invocation, _InvocationLost) + assert not any(isinstance(group, _GroupReleased) for group in execution.accounting.groups) + assert reference.released_groups == frozenset() + assert marker.read_text() == "started" + assert calls == 1 + + +def test_accounting_runtime_rejects_malformed_terminal_evidence_without_raising( + stub_slim_model_selection: ModelSelection, +) -> None: + class _MalformedBackend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records, verifier + token = dataframe[PRIVATE_CORRELATION_COLUMN].iloc[0] + return _PandasExecutionResult( + dataframe=pd.DataFrame(), + failed_records=[], + terminal_outcomes=((token, cast(_TerminalOutcome, "success")),), + result_row_tokens=(), + ) + + execution = _AccountingGraphRuntime(_MalformedBackend()).run( + _compiled(_graph("a")), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=lambda datum, row: (datum.id.value, row[COL_TEXT]), + ) + + assert isinstance(execution.accounting.invocation, _InvocationInconsistent) + assert isinstance(execution.accounting.groups[0], _GroupWithheld) + + +def test_accounting_runtime_localizes_hydration_failure_and_withholds_output( + stub_slim_model_selection: ModelSelection, +) -> None: + class _Backend: + def run( + self, + dataframe: pd.DataFrame, + *, + invocation: _CompiledInvocation, + data_summary: str | None, + preview_num_records: int | None, + verifier: _InvocationRowVerifier, + ) -> _PandasExecutionResult: + del invocation, data_summary, preview_num_records + detected = dataframe.assign(**{COL_FINAL_ENTITIES: [{"entities": []}]}) + verifier.freeze_accepted_detections(detected) + final = verifier.finish(detected) + return _PandasExecutionResult( + dataframe=final, + failed_records=[], + terminal_outcomes=verifier.take_terminal_outcomes(), + result_row_tokens=verifier.take_result_order(), + ) + + def _malformed_hydration(_datum: _TextDatum, _row: pd.Series) -> str: + raise TypeError("private-hydration-canary") + + execution = _AccountingGraphRuntime(_Backend()).run( + _compiled(_graph("a")), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + hydrate=_malformed_hydration, + ) + + assert isinstance(execution.accounting.invocation, _InvocationCompleted) + assert isinstance(execution.accounting.tasks[0], _TaskFailed) + assert isinstance(execution.accounting.groups[0], _GroupWithheld) + assert "private-hydration-canary" not in repr(execution) + + +@pytest.mark.parametrize("raises", [False, True]) +def test_group_release_predicate_failure_never_exposes_group_output(raises: bool) -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + ledger.accept_success(dispatch, "protected") + + def _group_predicate(_outputs: tuple[tuple[_DatumId, str], ...]) -> bool: + if raises: + raise RuntimeError("private-group-canary") + return False + + result = ledger.finish(group_release_predicate=_group_predicate) + + if raises: + assert isinstance(result.invocation, _InvocationFailed) + else: + assert isinstance(result.invocation, _InvocationCompleted) + assert isinstance(result.groups[0], _GroupWithheld) + + +def test_group_release_predicate_requires_an_exact_boolean() -> None: + ledger = _ledger(_compiled(_graph("a"))) + ledger.open() + dispatch = ledger.dispatch(ledger.ready_tasks()[0]) + ledger.accept_success(dispatch, "protected") + + result = ledger.finish(group_release_predicate=lambda _outputs: cast(bool, "truthy-not-bool")) + + assert isinstance(result.invocation, _InvocationFailed) + assert isinstance(result.groups[0], _GroupWithheld) + assert "private-group-canary" not in repr(result) + + +def test_group_predicate_failure_propagates_through_explicit_dependencies() -> None: + graph = _graph("a", "b") + graph = replace(graph, dependencies=(_dependency(graph, "a", "b"),)) + plan = _compiled(graph) + ledger = _ledger(plan) + ledger.open() + + first = ledger.ready_tasks() + assert tuple(_datum_id(task).value for task in first) == ("a",) + ledger.accept_success(ledger.dispatch(first[0]), "protected-a") + second = ledger.ready_tasks() + assert tuple(_datum_id(task).value for task in second) == ("b",) + ledger.accept_success(ledger.dispatch(second[0]), "protected-b") + + result = ledger.finish( + group_release_predicate=lambda outputs: all(datum_id.value != "a" for datum_id, _output in outputs) + ) + + assert tuple(type(group) for group in result.groups) == (_GroupWithheld, _GroupWithheld) + + +def test_accounting_admission_compiles_a_detached_singleton_plan() -> None: + source = _TextDatum(_DatumId("datum-a"), "synthetic input") + + result = _compile_accounting_plan(_trivial_graph((source,)), limits=_LIMITS) + + assert isinstance(result, _AccountingPlan) + assert result.datums == (source,) + assert result.datums[0] is not source + assert tuple(_datum_id(task) for task in result.tasks) == (source.id,) + object.__setattr__(source, "text", "mutated after compilation") + assert result.datums[0].text == "synthetic input" + + +def test_compiled_plan_is_frozen(stub_slim_model_selection: ModelSelection) -> None: + plan = _compile_accounting_plan(_graph("a"), limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + + with pytest.raises(FrozenInstanceError): + setattr(plan, "datums", ()) + + +def test_accounting_admission_accepts_exact_byte_limits_and_rejects_limit_plus_one() -> None: + exact = _trivial_graph((_TextDatum(_DatumId("iiii"), "1234"),)) + limits = _AccountingLimits(max_datums=1, max_datum_bytes=4, max_graph_bytes=4, max_id_bytes=4) + + assert isinstance(_compile_accounting_plan(exact, limits=limits), _AccountingPlan) + assert _compile_accounting_plan( + _trivial_graph((_TextDatum(_DatumId("iiiii"), "1234"),)), + limits=limits, + ) == _AccountingRejected(_AccountingAdmissionCode.MALFORMED_GRAPH) + assert _compile_accounting_plan( + _trivial_graph((_TextDatum(_DatumId("iiii"), "12345"),)), + limits=limits, + ) == _AccountingRejected(_AccountingAdmissionCode.DATUM_TOO_LARGE) + + +@pytest.mark.parametrize( + "datum", + [ + _TextDatum(_DatumId("invalid-\ud800-id"), "valid text"), + _TextDatum(_DatumId("valid-id"), "invalid-\ud800-text"), + ], +) +def test_accounting_admission_rejects_non_utf8_datum_values(datum: _TextDatum) -> None: + assert _compile_accounting_plan(_trivial_graph((datum,)), limits=_LIMITS) == _AccountingRejected( + _AccountingAdmissionCode.MALFORMED_GRAPH + ) + + +def test_accounting_admission_rejects_empty_and_count_limit_plus_one() -> None: + assert _compile_accounting_plan(_trivial_graph(()), limits=_LIMITS) == _AccountingRejected( + _AccountingAdmissionCode.MALFORMED_GRAPH + ) + assert _compile_accounting_plan(_graph("a", "b"), limits=replace(_LIMITS, max_datums=1)) == _AccountingRejected( + _AccountingAdmissionCode.TOO_MANY_DATUMS + ) + + +def test_accounting_admission_accepts_exact_count_limits_and_rejects_plus_one() -> None: + four = _graph("a", "b", "c", "d") + assert isinstance(_compile_accounting_plan(four, limits=_LIMITS), _AccountingPlan) + assert _compile_accounting_plan(_graph("a", "b", "c", "d", "e"), limits=_LIMITS) == _AccountingRejected( + _AccountingAdmissionCode.TOO_MANY_DATUMS + ) + + one_edge = replace(four, dependencies=(_dependency(four, "a", "b"),)) + assert isinstance(_compile_accounting_plan(one_edge, limits=replace(_LIMITS, max_dependencies=1)), _AccountingPlan) + two_edges = replace( + four, + dependencies=(_dependency(four, "a", "b"), _dependency(four, "a", "c")), + ) + assert _compile_accounting_plan(two_edges, limits=replace(_LIMITS, max_dependencies=1)) == _AccountingRejected( + _AccountingAdmissionCode.TOO_MANY_DEPENDENCIES + ) + + assert isinstance(_compile_accounting_plan(four, limits=replace(_LIMITS, max_atomic_groups=4)), _AccountingPlan) + assert _compile_accounting_plan(four, limits=replace(_LIMITS, max_atomic_groups=3)) == _AccountingRejected( + _AccountingAdmissionCode.TOO_MANY_ATOMIC_GROUPS + ) + + +def test_accounting_admission_enforces_aggregate_graph_bytes_and_exact_stage_limit() -> None: + graph = _trivial_graph((_TextDatum(_DatumId("a"), "12"), _TextDatum(_DatumId("b"), "34"))) + exact = _AccountingLimits(max_datums=2, max_datum_bytes=2, max_graph_bytes=4, max_stages=3) + + assert isinstance(_compile_accounting_plan(graph, limits=exact, stages=("a", "b", "c")), _AccountingPlan) + assert _compile_accounting_plan(graph, limits=replace(exact, max_graph_bytes=3)) == _AccountingRejected( + _AccountingAdmissionCode.GRAPH_TOO_LARGE + ) + + +@pytest.mark.parametrize( + ("mutation", "code"), + [ + ( + lambda graph: replace( + graph, + links=(_DatumLink(graph.datums[0].id, graph.datums[1].id, _RelationKind.RELATED),), + ), + _AccountingAdmissionCode.UNSUPPORTED_RELATIONSHIPS, + ), + ( + lambda graph: replace( + graph, + context_scopes=(_ContextScope(graph.datums[0].id, (graph.datums[1].id,)),), + ), + _AccountingAdmissionCode.UNSUPPORTED_CONTEXT, + ), + ( + lambda graph: replace( + graph, + coherence_scopes=(_CoherenceScope(tuple(datum.id for datum in graph.datums)),), + ), + _AccountingAdmissionCode.UNSUPPORTED_COHERENCE, + ), + ], +) +def test_accounting_admission_rejects_unsupported_phase4_semantics( + mutation: Callable[[_ProtectionGraph], _ProtectionGraph], + code: _AccountingAdmissionCode, +) -> None: + assert _compile_accounting_plan(mutation(_graph("a", "b")), limits=_LIMITS) == _AccountingRejected(code) + + +def test_accounting_admission_rejects_duplicate_empty_context_declarations() -> None: + graph = _graph("a", "b") + duplicated = replace(graph, context_scopes=(*graph.context_scopes, graph.context_scopes[0])) + + assert _compile_accounting_plan(duplicated, limits=_LIMITS) == _AccountingRejected( + _AccountingAdmissionCode.UNSUPPORTED_CONTEXT + ) + + +@pytest.mark.parametrize("stages", [(), ("",), ("same", "same"), ("a", "b", "c", "d")]) +def test_accounting_admission_rejects_unsupported_stage_cardinality(stages: tuple[str, ...]) -> None: + assert _compile_accounting_plan(_graph("a"), limits=_LIMITS, stages=stages) == _AccountingRejected( + _AccountingAdmissionCode.UNSUPPORTED_TASK_CARDINALITY + ) + + +def test_streaming_conformance_corpus_matches_ledger_and_frozen_digest() -> None: + digest = sha256() + graph_count = 0 + trace_count = 0 + for case in streaming_conformance_cases(): + digest.update(_canonical_case(case.declaration, case.observations)) + trace_count += 1 + if not case.declaration.datum_ids: + graph_count += 1 + assert _compile_accounting_plan(_graph(), limits=_LIMITS) == _AccountingRejected( + _AccountingAdmissionCode.MALFORMED_GRAPH + ) + continue + if case.graph_witness and case.declaration.stages == ("stage-0",): + graph_count += 1 + expected = reduce_observations(case.declaration, case.observations) + actual = _run_ledger_case(case.declaration, case.observations) + assert _ledger_shape(actual) == _reference_shape(expected) + + assert _CONFORMANCE_MANIFEST["generator_version"] == "phase4-stream-v4" + assert graph_count == _CONFORMANCE_MANIFEST["graph_count"] + assert trace_count == _CONFORMANCE_MANIFEST["canonical_trace_count"] + assert digest.hexdigest() == _CONFORMANCE_MANIFEST["sha256"] + + +def test_barrier_race_matrix_preserves_one_shot_terminal_conservation() -> None: + """Barrier gates force each legal terminal ordering without timing assumptions.""" + + def ordered(first: Callable[[], object], second: Callable[[], object]) -> tuple[object, object]: + started = Barrier(3) + first_finished = Barrier(2) + + def run_first() -> object: + started.wait() + outcome = first() + first_finished.wait() + return outcome + + def run_second() -> object: + started.wait() + first_finished.wait() + return second() + + with ThreadPoolExecutor(max_workers=2) as executor: + first_future = executor.submit(run_first) + second_future = executor.submit(run_second) + started.wait() + return first_future.result(), second_future.result() + + def opened() -> tuple[_AccountingPlan, _AccountingLedger[str], _Dispatch]: + plan = _compiled(_graph("a")) + ledger = _ledger(plan) + ledger.open() + return plan, ledger, ledger.dispatch(ledger.ready_tasks()[0]) + + def assert_conserved( + plan: _AccountingPlan, + result: _AccountingResult[str], + *, + released: tuple[str, ...], + ) -> None: + assert len(result.tasks) == len(plan.tasks) == 1 + assert len({outcome.task for outcome in result.tasks}) == 1 + assert len(result.datums) == len(plan.datums) == 1 + assert len(result.dependencies) == len(plan.dependencies) == 0 + assert len(result.stages) == len(plan.stages) == 1 + assert len(result.groups) == len(plan.atomic_groups) == 1 + outputs = tuple( + datum.value for group in result.groups if isinstance(group, _GroupReleased) for datum, _ in group.outputs + ) + assert outputs == released + assert sum(isinstance(group, _GroupReleased) for group in result.groups) == (1 if released else 0) + + # Success publishes before a later cancellation: exactly one successful publication wins. + plan, ledger, dispatch = opened() + first, second = ordered( + lambda: ledger.accept_success(dispatch, "protected"), + lambda: ledger.finish(), + ) + assert first is _EvidenceAcceptance.ACCEPTED + assert isinstance(second, _AccountingResult) + assert isinstance(second.invocation, _InvocationCompleted) + assert_conserved(plan, second, released=("a",)) + with pytest.raises(_LedgerClosedError): + ledger.finish() + assert ledger.request_cancellation() is None + + # Cancellation is recorded before a late success; publication withholds output. + plan, ledger, dispatch = opened() + first, second = ordered(ledger.request_cancellation, lambda: ledger.accept_success(dispatch, "late")) + assert first is None + assert second is _EvidenceAcceptance.ACCEPTED + result = ledger.finish() + assert isinstance(result.invocation, _InvocationCancelled) + assert_conserved(plan, result, released=()) + + # A trusted stop wins before a late success. + plan, ledger, dispatch = opened() + ledger.request_cancellation() + first, second = ordered(lambda: ledger.acknowledge_stop(dispatch), lambda: ledger.accept_success(dispatch, "late")) + assert first is _EvidenceAcceptance.ACCEPTED + assert second is _EvidenceAcceptance.REJECTED_STALE + result = ledger.finish() + assert isinstance(result.tasks[0], _TaskCancelled) + assert_conserved(plan, result, released=()) + + # Success wins before a late stop, but cancellation still withholds publication. + plan, ledger, dispatch = opened() + first, second = ordered(lambda: ledger.accept_success(dispatch, "protected"), ledger.request_cancellation) + assert first is _EvidenceAcceptance.ACCEPTED + assert second is None + assert ledger.acknowledge_stop(dispatch) is _EvidenceAcceptance.REJECTED_STALE + result = ledger.finish() + assert isinstance(result.tasks[0], _TaskSucceeded) + assert isinstance(result.invocation, _InvocationCancelled) + assert_conserved(plan, result, released=()) + + # Post-dispatch cancellation without a trusted stop closes as loss, never retrying. + plan, ledger, dispatch = opened() + first, second = ordered(ledger.request_cancellation, ledger.finish) + assert first is None + assert isinstance(second, _AccountingResult) + assert isinstance(second.tasks[0], _TaskLost) + assert isinstance(second.invocation, _InvocationLost) + assert_conserved(plan, second, released=()) + with pytest.raises(_LedgerStateError): + ledger.dispatch(dispatch.task) + with pytest.raises(_LedgerClosedError): + ledger.finish() + + +def test_concurrent_frontier_terminals_preserve_hierarchical_fixed_point() -> None: + graph = _graph("a", "b", "c") + graph = replace( + graph, + dependencies=(_dependency(graph, "a", "b"),), + atomic_groups=(_group(graph, "a", "c"), _group(graph, "b")), + ) + ledger = _ledger(_compiled(graph)) + ledger.open() + first, peer = tuple(ledger.dispatch(task) for task in ledger.ready_tasks()) + gate = Barrier(3) + + def terminalize(dispatch: _Dispatch, *, succeeds: bool) -> _EvidenceAcceptance: + gate.wait() + return ( + ledger.accept_success(dispatch, f"protected-{_datum_id(dispatch.task).value}") + if succeeds + else ledger.accept_failure(dispatch) + ) + + with ThreadPoolExecutor(max_workers=2) as executor: + success = executor.submit(terminalize, first, succeeds=True) + failure = executor.submit(terminalize, peer, succeeds=False) + gate.wait() + assert success.result() is _EvidenceAcceptance.ACCEPTED + assert failure.result() is _EvidenceAcceptance.ACCEPTED + + dependent = ledger.dispatch(ledger.ready_tasks()[0]) + ledger.accept_success(dependent, "protected-b") + result = ledger.finish() + + assert isinstance(result.invocation, _InvocationCompleted) + assert all(isinstance(group, _GroupWithheld) for group in result.groups) + assert not any(isinstance(group, _GroupReleased) for group in result.groups) + + +def _canonical_case( + declaration: ReferenceDeclaration, + observations: tuple[ReferenceObservation, ...], +) -> bytes: + def event(observation: object) -> tuple[str, tuple[str, str] | None]: + match observation: + case ReferenceDispatch(task=task): + return ("dispatch", task) + case ReferenceSuccess(task=task): + return ("success", task) + case ReferenceFailure(task=task): + return ("failure", task) + case ReferenceCancellationRequest(): + return ("cancel", None) + case ReferenceStopAcknowledgement(task=task): + return ("stop", task) + case ReferenceTransportLoss(task=task): + return ("lost", task) + case ReferenceContradiction(): + return ("contradiction", None) + case ReferenceResultConstructionFailure(): + return ("result-construction-failure", None) + case ReferenceCorruptEvidence(kind=kind): + return (f"corruption:{kind.value}", None) + case _: + raise AssertionError("unknown reference observation") + + return dumps( + { + "atomic_groups": declaration.atomic_groups, + "datum_ids": declaration.datum_ids, + "dependencies": declaration.dependencies, + "observations": tuple(event(observation) for observation in observations), + "stages": declaration.stages, + }, + separators=(",", ":"), + sort_keys=True, + ).encode() + + +def _run_ledger_case( + declaration: ReferenceDeclaration, + observations: tuple[ReferenceObservation, ...], +) -> _AccountingResult[str]: + graph = _graph(*declaration.datum_ids) + graph = replace( + graph, + dependencies=tuple(_dependency(graph, *dependency) for dependency in declaration.dependencies), + atomic_groups=tuple(_group(graph, *group) for group in declaration.atomic_groups), + ) + plan = _compile_accounting_plan(graph, limits=_LIMITS, stages=declaration.stages) + assert isinstance(plan, _AccountingPlan) + ledger = _ledger(plan) + ledger.open() + result_construction_failure = False + tasks = {(task.stage.value, _datum_id(task).value): task for task in plan.tasks} + dispatches = {} + for observation in observations: + match observation: + case ReferenceDispatch(task=task): + dispatches[task] = ledger.dispatch(tasks[task]) + case ReferenceSuccess(task=task): + ledger.accept_success(dispatches[task], f"candidate-{task[0]}-{task[1]}") + case ReferenceFailure(task=task): + ledger.accept_failure(dispatches[task]) + case ReferenceCancellationRequest(): + ledger.request_cancellation() + case ReferenceStopAcknowledgement(task=task): + ledger.acknowledge_stop(dispatches[task]) + case ReferenceTransportLoss(task=task): + ledger.mark_transport_lost(dispatches[task]) + case ReferenceContradiction(): + ledger.mark_inconsistent(_CauseCode.CONTRADICTORY) + case ReferenceResultConstructionFailure(): + result_construction_failure = True + case ReferenceCorruptEvidence(kind=kind): + dispatch = dispatches[next(reversed(dispatches))] + match kind: + case ReferenceCorruption.MISSING: + ledger.reconcile((dispatch,), (), trusted_run_record=True) + case ReferenceCorruption.DUPLICATE: + ledger.reconcile( + (dispatch, dispatch), + (_SuccessRecord(dispatch, "duplicate"),), + trusted_run_record=True, + ) + case ReferenceCorruption.UNKNOWN: + unknown = replace( + dispatch, + attempt_id=_AttemptId("unknown-attempt"), + row_token=_RowToken("unknown-row"), + ) + ledger.reconcile((dispatch,), (_SuccessRecord(unknown, "unknown"),), trusted_run_record=True) + case ReferenceCorruption.FOREIGN: + foreign = replace(dispatch, row_token=_RowToken("foreign-token")) + ledger.reconcile((dispatch,), (_SuccessRecord(foreign, "foreign"),), trusted_run_record=True) + case ReferenceCorruption.STALE: + stale = replace(dispatch, attempt_id=_AttemptId("stale-attempt")) + ledger.reconcile((dispatch,), (_SuccessRecord(stale, "stale"),), trusted_run_record=True) + case ReferenceCorruption.SWAPPED: + swapped_task = next(task for task in plan.tasks if task != dispatch.task) + swapped = replace(dispatch, task=swapped_task) + ledger.reconcile((dispatch,), (_SuccessRecord(swapped, "swapped"),), trusted_run_record=True) + case ReferenceCorruption.PLAN_MISMATCH: + incompatible_plan = _compile_accounting_plan( + graph, + limits=_LIMITS, + stages=("incompatible-stage",), + ) + assert isinstance(incompatible_plan, _AccountingPlan) + incompatible_ledger = _ledger(incompatible_plan) + incompatible_ledger.open() + incompatible_dispatch = incompatible_ledger.dispatch(incompatible_ledger.ready_tasks()[0]) + ledger.reconcile( + (dispatch,), + (_SuccessRecord(incompatible_dispatch, "plan-mismatch"),), + trusted_run_record=True, + ) + case ReferenceCorruption.CONTRADICTORY: + ledger.reconcile((), (_SuccessRecord(dispatch, "contradiction"),), trusted_run_record=True) + case _: + raise AssertionError("unknown reference observation") + return ledger.finish( + group_release_predicate=( + (lambda _outputs: cast(bool, "invalid-result")) if result_construction_failure else (lambda _outputs: True) + ) + ) + + +def _ledger_shape(result: _AccountingResult[str]) -> tuple[object, ...]: + return ( + tuple(((task.task.stage.value, _datum_id(task.task).value), _outcome_name(task)) for task in result.tasks), + tuple( + ( + (task.task.stage.value, _datum_id(task.task).value), + tuple(cause.code.value for cause in getattr(task, "causes", ())), + ) + for task in result.tasks + ), + tuple((datum.datum_id.value, _outcome_name(datum)) for datum in result.datums), + tuple( + ( + (dependency.dependency.prerequisite.value, dependency.dependency.dependent.value), + isinstance(dependency, _DependencySatisfied), + ) + for dependency in result.dependencies + ), + tuple((stage.stage.value, _outcome_name(stage)) for stage in result.stages), + tuple( + tuple(datum_id.value for datum_id, _candidate in group.outputs) + for group in result.groups + if isinstance(group, _GroupReleased) + ), + _outcome_name(result.invocation), + ) + + +def _reference_shape(result: ReferenceHierarchyResult) -> tuple[object, ...]: + return ( + result.tasks, + result.task_causes, + result.datums, + result.dependencies, + result.stages, + result.released_group_order, + result.invocation.value, + ) + + +def _outcome_name(outcome: object) -> str: + name = type(outcome).__name__.removeprefix("_").removeprefix("Invocation") + return { + "TaskSucceeded": "succeeded", + "TaskFailed": "failed", + "TaskCancelled": "cancelled", + "TaskLost": "lost", + "TaskBlocked": "blocked", + "TaskInconsistent": "inconsistent", + "DatumQualified": "succeeded", + "DatumFailed": "failed", + "DatumCancelled": "cancelled", + "DatumLost": "lost", + "DatumBlocked": "blocked", + "DatumInconsistent": "inconsistent", + "StageSucceeded": "succeeded", + "StageFailed": "failed", + "StageCancelled": "cancelled", + "StageLost": "lost", + "StageBlocked": "blocked", + "StageInconsistent": "inconsistent", + "Completed": "completed", + "Failed": "failed", + "Cancelled": "cancelled", + "Lost": "lost", + "Inconsistent": "inconsistent", + }[name] + + +def _graph(*datum_names: str) -> _ProtectionGraph: + return _trivial_graph(tuple(_TextDatum(_DatumId(name), f"synthetic {name}") for name in datum_names)) + + +def _dependency(graph: _ProtectionGraph, prerequisite: str, dependent: str) -> _DatumDependency: + by_id = {datum.id.value: datum.id for datum in graph.datums} + return _DatumDependency(by_id[prerequisite], by_id[dependent]) + + +def _group(graph: _ProtectionGraph, *members: str) -> _AtomicGroup: + by_id = {datum.id.value: datum.id for datum in graph.datums} + return _AtomicGroup(tuple(by_id[member] for member in members)) + + +def test_accounting_admission_compiles_declaration_order_independent_dag_and_partition() -> None: + source = _graph("d", "c", "b", "a") + source = replace( + source, + dependencies=( + _dependency(source, "a", "b"), + _dependency(source, "a", "c"), + _dependency(source, "b", "d"), + _dependency(source, "c", "d"), + ), + atomic_groups=(_group(source, "a", "b"), _group(source, "c", "d")), + ) + + result = _compile_accounting_plan(source, limits=_LIMITS, stages=("detect", "protect")) + + assert isinstance(result, _AccountingPlan) + assert tuple(datum_id.value for datum_id in result.topological_datums) == ("a", "c", "b", "d") + assert tuple(_datum_id(task).value for task in result.tasks[:4]) == ("a", "c", "b", "d") + assert len(result.tasks) == 8 + assert {frozenset(member.value for member in group.members) for group in result.atomic_groups} == { + frozenset(("a", "b")), + frozenset(("c", "d")), + } + + +@pytest.mark.parametrize( + ("mutation", "code"), + [ + ( + lambda graph: replace(graph, dependencies=(_DatumDependency(graph.datums[0].id, graph.datums[0].id),)), + _AccountingAdmissionCode.SELF_DEPENDENCY, + ), + ( + lambda graph: replace( + graph, + dependencies=(_DatumDependency(graph.datums[0].id, _DatumId("missing")),), + ), + _AccountingAdmissionCode.DANGLING_DEPENDENCY, + ), + ( + lambda graph: replace( + graph, + dependencies=( + _DatumDependency(graph.datums[0].id, graph.datums[1].id), + _DatumDependency(graph.datums[0].id, graph.datums[1].id), + ), + ), + _AccountingAdmissionCode.DUPLICATE_DEPENDENCY, + ), + ( + lambda graph: replace( + graph, + dependencies=( + _DatumDependency(graph.datums[0].id, graph.datums[1].id), + _DatumDependency(graph.datums[1].id, graph.datums[0].id), + ), + ), + _AccountingAdmissionCode.DEPENDENCY_CYCLE, + ), + ( + lambda graph: replace(graph, atomic_groups=(_AtomicGroup(()), *graph.atomic_groups)), + _AccountingAdmissionCode.EMPTY_ATOMIC_GROUP, + ), + ( + lambda graph: replace( + graph, + atomic_groups=(_AtomicGroup((graph.datums[0].id, graph.datums[0].id)), *graph.atomic_groups[1:]), + ), + _AccountingAdmissionCode.DUPLICATE_ATOMIC_MEMBER, + ), + ( + lambda graph: replace( + graph, + atomic_groups=(_AtomicGroup((_DatumId("missing"),)), *graph.atomic_groups[1:]), + ), + _AccountingAdmissionCode.DANGLING_ATOMIC_MEMBER, + ), + ( + lambda graph: replace(graph, atomic_groups=(*graph.atomic_groups, graph.atomic_groups[0])), + _AccountingAdmissionCode.DUPLICATE_ATOMIC_GROUP, + ), + ( + lambda graph: replace(graph, atomic_groups=(_AtomicGroup((graph.datums[0].id,)),)), + _AccountingAdmissionCode.ATOMIC_COVERAGE_GAP, + ), + ( + lambda graph: replace( + graph, + atomic_groups=( + _AtomicGroup((graph.datums[0].id, graph.datums[1].id)), + _AtomicGroup((graph.datums[1].id, graph.datums[2].id)), + ), + ), + _AccountingAdmissionCode.ATOMIC_GROUP_OVERLAP, + ), + ( + lambda graph: replace( + graph, + atomic_groups=( + _AtomicGroup((graph.datums[0].id,)), + _AtomicGroup((graph.datums[0].id, graph.datums[1].id, graph.datums[2].id)), + ), + ), + _AccountingAdmissionCode.UNSUPPORTED_ATOMIC_NESTING, + ), + ], +) +def test_accounting_admission_rejects_invalid_topology( + mutation: Callable[[_ProtectionGraph], _ProtectionGraph], + code: _AccountingAdmissionCode, +) -> None: + graph = mutation(_graph("a", "b", "c")) + + result = _compile_accounting_plan(graph, limits=_LIMITS) + + assert result == _AccountingRejected(code) + + +def test_cycle_precedes_recognized_but_unsupported_atomic_nesting() -> None: + graph = _graph("a", "b") + graph = replace( + graph, + dependencies=(_dependency(graph, "a", "b"), _dependency(graph, "b", "a")), + atomic_groups=(_group(graph, "a"), _group(graph, "a", "b")), + ) + + result = _compile_accounting_plan(graph, limits=_LIMITS) + + assert result == _AccountingRejected(_AccountingAdmissionCode.DEPENDENCY_CYCLE) + + +def test_release_uses_dependency_group_fixed_point_and_matches_independent_model() -> None: + graph = _graph("a", "b", "c", "d", "e", "f") + graph = replace( + graph, + dependencies=(_dependency(graph, "a", "c"), _dependency(graph, "d", "e")), + atomic_groups=( + _group(graph, "a", "b"), + _group(graph, "c", "d"), + _group(graph, "e"), + _group(graph, "f"), + ), + ) + limits = replace(_LIMITS, max_datums=6, max_graph_bytes=256) + plan = _compile_accounting_plan(graph, limits=limits) + assert isinstance(plan, _AccountingPlan) + qualified = frozenset(datum.id for datum in plan.datums if datum.id.value != "b") + + actual = _qualify_release(plan, qualified) + reference = reduce_reference( + ReferenceDeclaration( + tuple(datum.id.value for datum in plan.datums), + tuple((edge.prerequisite.value, edge.dependent.value) for edge in plan.dependencies), + tuple(tuple(member.value for member in group.members) for group in plan.atomic_groups), + ), + { + datum.id.value: (ReferenceTaskOutcome.FAILED if datum.id.value == "b" else ReferenceTaskOutcome.SUCCEEDED) + for datum in plan.datums + }, + ) + + assert ( + frozenset(datum.value for datum in actual.release_eligible) == reference.release_eligible == frozenset(("f",)) + ) + released_members = frozenset( + frozenset(member.value for member in group.members) + for group in plan.atomic_groups + if group.key in actual.released_groups + ) + assert released_members == reference.released_groups == frozenset((frozenset(("f",)),)) + + +def test_release_matches_independent_model_for_all_graphs_through_three_datums() -> None: + for datum_count in range(1, 4): + names = tuple(chr(ord("a") + ordinal) for ordinal in range(datum_count)) + base = _graph(*names) + by_name = {datum.id.value: datum.id for datum in base.datums} + for dependencies in acyclic_dependencies(names): + for partition in flat_partitions(names): + graph = replace( + base, + dependencies=tuple( + _DatumDependency(by_name[prerequisite], by_name[dependent]) + for prerequisite, dependent in dependencies + ), + atomic_groups=tuple( + _AtomicGroup(tuple(by_name[member] for member in group)) for group in partition + ), + ) + plan = _compile_accounting_plan(graph, limits=_LIMITS) + assert isinstance(plan, _AccountingPlan) + declaration = ReferenceDeclaration(names, dependencies, partition) + for qualification_bits in product((False, True), repeat=datum_count): + qualified_names = frozenset( + name for name, qualified in zip(names, qualification_bits, strict=True) if qualified + ) + actual = _qualify_release( + plan, + frozenset(datum.id for datum in plan.datums if datum.id.value in qualified_names), + ) + reference = reduce_reference( + declaration, + { + name: ( + ReferenceTaskOutcome.SUCCEEDED + if name in qualified_names + else ReferenceTaskOutcome.FAILED + ) + for name in names + }, + ) + assert frozenset(datum.value for datum in actual.release_eligible) == reference.release_eligible + assert ( + frozenset( + frozenset(member.value for member in group.members) + for group in plan.atomic_groups + if group.key in actual.released_groups + ) + == reference.released_groups + ) + + +def test_seeded_opaque_id_renaming_and_declaration_permutations_preserve_release_semantics() -> None: + rng = random.Random(0xA11CE) + logical = tuple(range(4)) + for case in range(128): + presentation = list(logical) + rng.shuffle(presentation) + names = {ordinal: f"d{case:x}-{ordinal:x}" for ordinal in logical} + presented_names = tuple(names[ordinal] for ordinal in presentation) + graph = _graph(*presented_names) + by_name = {datum.id.value: datum.id for datum in graph.datums} + dependencies = tuple( + (left, right) for left in logical for right in logical if left < right and rng.choice((False, True)) + ) + bucket_by_member = {member: rng.randrange(4) for member in logical} + groups = tuple( + tuple(member for member in logical if bucket_by_member[member] == bucket) + for bucket in range(4) + if bucket in bucket_by_member.values() + ) + graph = replace( + graph, + dependencies=tuple( + _DatumDependency(by_name[names[left]], by_name[names[right]]) for left, right in dependencies + ), + atomic_groups=tuple(_AtomicGroup(tuple(by_name[names[member]] for member in group)) for group in groups), + ) + plan = _compile_accounting_plan(graph, limits=replace(_LIMITS, max_graph_bytes=512)) + assert isinstance(plan, _AccountingPlan) + qualified_members = frozenset(member for member in logical if rng.choice((False, True))) + actual = _qualify_release( + plan, + frozenset( + datum.id + for datum in plan.datums + if next(k for k, v in names.items() if v == datum.id.value) in qualified_members + ), + ) + reference = reduce_reference( + ReferenceDeclaration( + presented_names, + tuple((names[left], names[right]) for left, right in dependencies), + tuple(tuple(names[member] for member in group) for group in groups), + ), + { + names[member]: ( + ReferenceTaskOutcome.SUCCEEDED if member in qualified_members else ReferenceTaskOutcome.FAILED + ) + for member in logical + }, + ) + + assert frozenset(datum.value for datum in actual.release_eligible) == reference.release_eligible + + +def test_dispatch_batch_size_does_not_change_terminal_or_release_result() -> None: + graph = _graph("a", "b", "c", "d") + graph = replace( + graph, + dependencies=(_dependency(graph, "a", "c"),), + atomic_groups=(_group(graph, "a", "b"), _group(graph, "c", "d")), + ) + plan = _compiled(graph) + + def execute(batch_size: int) -> _AccountingResult[str]: + ledger = _ledger(plan) + ledger.open() + while ready := ledger.ready_tasks(): + for task in ready[:batch_size]: + dispatch = ledger.dispatch(task) + if _datum_id(task).value == "b": + ledger.accept_failure(dispatch) + else: + ledger.accept_success(dispatch, f"candidate-{_datum_id(task).value}") + return ledger.finish() + + assert _ledger_shape(execute(1)) == _ledger_shape(execute(4)) diff --git a/tests/engine/execution/test_pandas_runtime.py b/tests/engine/execution/test_pandas_runtime.py new file mode 100644 index 00000000..3a1f00c9 --- /dev/null +++ b/tests/engine/execution/test_pandas_runtime.py @@ -0,0 +1,152 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +from unittest.mock import Mock + +import pandas as pd + +from anonymizer.config.anonymizer_config import AnonymizerConfig, Rewrite +from anonymizer.config.models import ModelSelection +from anonymizer.config.replace_strategies import Redact +from anonymizer.engine.constants import ( + COL_CONTEXT_BINDING_ID, + COL_CONTEXT_ORDINAL, + COL_CONTEXT_OWNER_WORK_ID, + COL_CONTEXT_TEXT, + COL_FINAL_ENTITIES, + COL_REPLACED_TEXT, + COL_REWRITTEN_TEXT, + COL_TEXT, +) +from anonymizer.engine.detection.detection_workflow import EntityDetectionResult, EntityDetectionWorkflow +from anonymizer.engine.execution.context_workframes import ( + _BackendArtifactId, + _ContextPayload, + _ContextPayloadToken, +) +from anonymizer.engine.execution.invocation import _CompiledInvocation +from anonymizer.engine.execution.pandas_runtime import _PandasRuntime +from anonymizer.engine.private_row_verification import _InvocationRowVerifier +from anonymizer.engine.replace.replace_runner import ReplacementResult, ReplacementWorkflow +from anonymizer.engine.rewrite.rewrite_workflow import RewriteResult, RewriteWorkflow + + +def _detected(frame: pd.DataFrame) -> EntityDetectionResult: + return EntityDetectionResult(dataframe=frame.assign(**{COL_FINAL_ENTITIES: [{"entities": []}]}), failed_records=[]) + + +def test_runtime_delegates_replace_with_existing_workflow_arguments(stub_slim_model_selection: ModelSelection) -> None: + frame = pd.DataFrame({COL_TEXT: ["Alice"]}) + verifier = _InvocationRowVerifier(frame) + bound = verifier.bind(frame) + detection = Mock(spec=EntityDetectionWorkflow) + detection.run.side_effect = lambda dataframe, **_: _detected(dataframe) + replace = Mock(spec=ReplacementWorkflow) + replace.run.side_effect = lambda dataframe, **_: ReplacementResult( + dataframe=dataframe.assign(**{COL_REPLACED_TEXT: ["[REDACTED]"]}), failed_records=[] + ) + runtime = _PandasRuntime( + detection_workflow=detection, + replace_runner=replace, + rewrite_runner=Mock(spec=RewriteWorkflow), + combined_rewrite_runner=Mock(spec=RewriteWorkflow), + ) + + result = runtime.run( + bound, + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary="support tickets", + preview_num_records=3, + verifier=verifier, + ) + + assert "__anonymizer_private_row_correlation__" not in result.dataframe + assert detection.run.call_args.kwargs["tag_latent_entities"] is False + assert replace.run.call_args.kwargs["preview_num_records"] == 1 + + +def test_runtime_selects_combined_rewrite_graph(stub_slim_model_selection: ModelSelection) -> None: + frame = pd.DataFrame({COL_TEXT: ["Alice"]}) + verifier = _InvocationRowVerifier(frame) + bound = verifier.bind(frame) + detection = Mock(spec=EntityDetectionWorkflow) + detection.run.side_effect = lambda dataframe, **_: _detected(dataframe) + combined = Mock(spec=RewriteWorkflow) + combined.run.side_effect = lambda dataframe, **_: RewriteResult( + dataframe=dataframe.assign(**{COL_REWRITTEN_TEXT: ["Someone"]}), failed_records=[] + ) + rewrite = Mock(spec=RewriteWorkflow) + runtime = _PandasRuntime( + detection_workflow=detection, + replace_runner=Mock(spec=ReplacementWorkflow), + rewrite_runner=rewrite, + combined_rewrite_runner=combined, + ) + + runtime.run( + bound, + invocation=_CompiledInvocation.compile( + AnonymizerConfig(rewrite=Rewrite(use_combined_graph=True)), stub_slim_model_selection + ), + data_summary=None, + preview_num_records=None, + verifier=verifier, + ) + + rewrite.run.assert_not_called() + combined.run.assert_called_once() + + +def test_context_framing_does_not_change_phase5_workflow_or_prompt_inputs( + stub_slim_model_selection: ModelSelection, +) -> None: + context_canary = "CONTEXT-CANARY-bob@example.test" + frame = pd.DataFrame({COL_TEXT: ["target text"]}) + verifier = _InvocationRowVerifier(frame, correlations=("target-work",)) + bound = verifier.bind(frame) + context_frame = pd.DataFrame( + { + COL_CONTEXT_BINDING_ID: ["binding-work"], + COL_CONTEXT_OWNER_WORK_ID: ["target-work"], + COL_CONTEXT_ORDINAL: [0], + COL_CONTEXT_TEXT: [_ContextPayload(context_canary, _ContextPayloadToken("binding-work"))], + } + ) + detection = Mock(spec=EntityDetectionWorkflow) + + def detect(dataframe: pd.DataFrame, **_kwargs: object) -> EntityDetectionResult: + assert context_canary not in dataframe.to_string() + assert COL_CONTEXT_TEXT not in dataframe.columns + return _detected(dataframe) + + detection.run.side_effect = detect + replace = Mock(spec=ReplacementWorkflow) + replace.run.side_effect = lambda dataframe, **_: ReplacementResult( + dataframe=dataframe.assign(**{COL_REPLACED_TEXT: ["target text"]}), failed_records=[] + ) + runtime = _PandasRuntime( + detection_workflow=detection, + replace_runner=replace, + rewrite_runner=Mock(spec=RewriteWorkflow), + combined_rewrite_runner=Mock(spec=RewriteWorkflow), + ) + + result = runtime.run_context( + bound, + context_dataframe=context_frame, + artifact_id=_BackendArtifactId("artifact-work"), + invocation=_CompiledInvocation.compile(AnonymizerConfig(replace=Redact()), stub_slim_model_selection), + data_summary=None, + preview_num_records=None, + verifier=verifier, + ) + + assert len(result.context_binding_evidence) == 1 + assert result.context_binding_evidence[0].ordinal == 0 + assert result.closure_attestations[0].closed is True + detection.run.assert_called_once() + replace.run.assert_called_once() + assert context_canary not in repr(detection.run.call_args) + assert context_canary not in repr(replace.run.call_args) diff --git a/tests/engine/execution/test_phase5_reference_model.py b/tests/engine/execution/test_phase5_reference_model.py new file mode 100644 index 00000000..4cb9ceee --- /dev/null +++ b/tests/engine/execution/test_phase5_reference_model.py @@ -0,0 +1,148 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import ast +import json +from dataclasses import replace +from pathlib import Path + +import pytest + +from tests.engine.execution.phase5_reference_model import ( + ReferenceAdmission, + ReferenceEvent, + ReferenceEventKind, + ReferenceInvocation, + corpus_manifest, + evaluate, + reference_cases, +) + +_MANIFEST = json.loads(Path(__file__).with_name("phase5_reference_manifest.json").read_text()) + + +def test_phase5_reference_corpus_matches_frozen_manifest() -> None: + assert corpus_manifest() == _MANIFEST + + +def test_reference_model_conserves_bindings_and_respects_event_bound() -> None: + for case in reference_cases(): + result = evaluate(case) + assert result.event_count <= result.event_max + if result.admission is not ReferenceAdmission.ADMITTED: + assert result.invocation is ReferenceInvocation.NOT_OPENED + assert result.released == () + if result.cleanup != "verified": + assert result.released == () + + +def test_reference_model_distinguishes_local_and_global_binding_faults() -> None: + admitted = next( + case + for case in reference_cases() + if case.schedule_class == "exact:verified:success:accepted" + and sum(len(scope.context) for scope in case.scopes) + and len(case.targets) == 2 + ) + local = next(case for case in reference_cases() if case.case_id == f"{admitted.case_id}-binding-missing") + global_fault = next( + case for case in reference_cases() if case.case_id == f"{admitted.case_id}-binding-cross_target" + ) + + local_result = evaluate(local) + global_result = evaluate(global_fault) + + assert len(local_result.private_inconsistent) == 1 + assert local_result.invocation is ReferenceInvocation.COMPLETED + assert global_result.private_inconsistent == tuple(identifier for identifier, _text in global_fault.targets) + assert global_result.invocation is ReferenceInvocation.INCONSISTENT + assert global_result.released == () + + +def test_reference_generator_freezes_required_ceiling_payload_and_event_domains() -> None: + manifest = corpus_manifest() + + assert manifest["ceiling_domain"] == ["zero", "exact", "exact_plus_one"] + assert manifest["payload_domain"] == ["empty", "one_byte", "multibyte", "exact_limit", "one_over_limit"] + actual_events = manifest["actual_event_count"] + trace_count = manifest["canonical_trace_count"] + assert isinstance(actual_events, int) + assert isinstance(trace_count, int) + assert actual_events > trace_count + assert all(case.events for case in reference_cases()) + + +def test_reference_model_rejects_a_trace_over_its_computed_bound() -> None: + case = next(reference_cases()) + excessive = replace( + case, + events=case.events + + tuple(ReferenceEvent(ReferenceEventKind.CANCELLATION) for _index in range(case.limits.expanded_bytes + 100)), + ) + + try: + evaluate(excessive) + except AssertionError as error: + assert "bound" in str(error) + else: + raise AssertionError("over-bound trace was not rejected") + + +def test_reference_model_imports_no_production_or_dataframe_modules() -> None: + source = Path(__file__).with_name("phase5_reference_model.py").read_text() + imported = { + alias.name + for node in ast.walk(ast.parse(source)) + if isinstance(node, (ast.Import, ast.ImportFrom)) + for alias in node.names + } + + assert not any(name.startswith(("anonymizer", "pandas", "data_designer")) for name in imported) + + +def test_reference_cancellation_linearization_matches_release_embargo() -> None: + cases = tuple(reference_cases()) + late = next(case for case in cases if case.schedule_class == "exact:verified:terminal_then_cancel:accepted") + before_dispatch = next( + case for case in cases if case.schedule_class == "exact:verified:cancel_pre_dispatch:accepted" + ) + + late_result = evaluate(late) + pre_dispatch_result = evaluate(before_dispatch) + + assert late_result.invocation is ReferenceInvocation.CANCELLED + assert all(state == "succeeded" for _target, state, _reason in late_result.task_outcomes) + assert late_result.released == () + assert pre_dispatch_result.invocation is ReferenceInvocation.CANCELLED + assert all(state == "cancelled" for _target, state, _reason in pre_dispatch_result.task_outcomes) + assert pre_dispatch_result.cleanup == "not_entered" + + +def test_reference_model_rejects_publication_before_cleanup() -> None: + case = next( + case + for case in reference_cases() + if case.schedule_class == "exact:verified:success:accepted" and len(case.targets) == 2 + ) + publication = next(event for event in case.events if event.kind is ReferenceEventKind.PUBLICATION) + reordered = replace(case, events=(publication, *(event for event in case.events if event is not publication))) + + with pytest.raises(AssertionError, match="publication"): + evaluate(reordered) + + +def test_reference_corpus_covers_terminal_evidence_corruptions() -> None: + cases = tuple(reference_cases()) + for fault in ("missing", "duplicate", "foreign", "stale", "cross_target", "plan_mismatch", "contradictory"): + case = next( + case + for case in cases + if case.schedule_class == f"exact:verified:terminal_{fault}:accepted" and len(case.targets) == 2 + ) + result = evaluate(case) + expected_release = () if fault != "missing" else ("t1",) + assert result.released == expected_release + if fault != "missing": + assert result.invocation is ReferenceInvocation.INCONSISTENT diff --git a/tests/engine/execution/test_phase6_mentions.py b/tests/engine/execution/test_phase6_mentions.py new file mode 100644 index 00000000..060730d1 --- /dev/null +++ b/tests/engine/execution/test_phase6_mentions.py @@ -0,0 +1,260 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib +import importlib.util +import pickle +from dataclasses import replace + +import pytest + +from anonymizer.engine.execution import accounting_plan as accounting_plan_module +from anonymizer.engine.execution.accounting_admission import _compile_accounting_plan +from anonymizer.engine.execution.accounting_ledger import _AccountingLedger +from anonymizer.engine.execution.accounting_plan import _AccountingLimits, _AccountingPlan +from anonymizer.engine.execution.graph import _DatumId, _TextDatum, _trivial_graph +from anonymizer.engine.execution.mention_admission import ( + _CandidateToken, + _DetectedGraph, + _finalize_mentions, + _MentionLimits, + _MentionProvenance, + _MentionRejected, + _MentionRejectionCode, + _MentionTarget, + _MentionTargetToken, + _ProvisionalCandidate, + _ValidationDecision, + _ValidationDecisionKind, +) + +_MENTION_LIMITS = _MentionLimits( + max_candidates_per_target=8, + max_mentions_per_target=8, + max_label_bytes=64, + max_source_slice_bytes=64, +) + + +def test_phase6_test_infrastructure() -> None: + assert _AccountingPlan.__name__ == "_AccountingPlan" + + +def test_explicit_task_predecessor_delays_resolver_until_referenced_target_finalizes() -> None: + predecessor_type = getattr(accounting_plan_module, "_TaskPredecessor", None) + assert predecessor_type is not None, "Phase 6 requires typed cross-task predecessors" + + graph = _trivial_graph( + ( + _TextDatum(_DatumId("target-a"), "Alice"), + _TextDatum(_DatumId("target-b"), "A. Example"), + ) + ) + plan = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=2, max_datum_bytes=32, max_graph_bytes=64), + stages=("finalize", "resolve"), + ) + assert isinstance(plan, _AccountingPlan) + finalize_a, finalize_b, resolve_a, resolve_b = plan.tasks + extended = plan.with_task_predecessors((predecessor_type(finalize_b, resolve_a),)) + ledger: _AccountingLedger[str] = _AccountingLedger(extended) + ledger.open() + + assert ledger.ready_tasks() == (finalize_a, finalize_b) + ledger.accept_success(ledger.dispatch(finalize_a), "finalized-a") + assert ledger.ready_tasks() == (finalize_b,) + ledger.accept_success(ledger.dispatch(finalize_b), "finalized-b") + assert ledger.ready_tasks() == (resolve_a, resolve_b) + + +def test_phase6_mention_module_exposes_strict_finalization_boundary() -> None: + module_name = "anonymizer.engine.execution.mention_admission" + assert importlib.util.find_spec(module_name) is not None, "Phase 6 mention admission module is missing" + module = importlib.import_module(module_name) + + assert callable(getattr(module, "_finalize_mentions", None)) + + +def test_finalization_anchors_unicode_and_keeps_repeated_equal_text_distinct() -> None: + target_token = _MentionTargetToken() + target = _MentionTarget(target_token, _DatumId("target"), "A😀A") + first_token = _CandidateToken() + emoji_token = _CandidateToken() + last_token = _CandidateToken() + candidates = ( + _ProvisionalCandidate(first_token, target_token, 0, 1, "A", "name", _MentionProvenance.SPAN_DETECTOR), + _ProvisionalCandidate(emoji_token, target_token, 1, 2, "😀", "symbol", _MentionProvenance.EXACT_AUGMENTER), + _ProvisionalCandidate(last_token, target_token, 2, 3, "A", "name", _MentionProvenance.SPAN_DETECTOR), + ) + decisions = ( + _ValidationDecision(last_token, _ValidationDecisionKind.KEEP), + _ValidationDecision(first_token, _ValidationDecisionKind.KEEP), + _ValidationDecision(emoji_token, _ValidationDecisionKind.RECLASS, "emoji"), + ) + + result = _finalize_mentions((target,), candidates, decisions, limits=_MENTION_LIMITS) + + assert isinstance(result, _DetectedGraph) + assert tuple( + (mention.start, mention.end, mention.source_slice, mention.detector_label) for mention in result.mentions + ) == ((0, 1, "A", "name"), (1, 2, "😀", "emoji"), (2, 3, "A", "name")) + assert len({mention.id for mention in result.mentions}) == 3 + + +@pytest.mark.parametrize( + ("changes", "code"), + [ + ({"start": True}, _MentionRejectionCode.INVALID_OFFSET), + ({"start": -1}, _MentionRejectionCode.INVALID_OFFSET), + ({"start": 1, "end": 1}, _MentionRejectionCode.INVALID_OFFSET), + ({"end": 99}, _MentionRejectionCode.INVALID_OFFSET), + ({"source_slice": "Mallory"}, _MentionRejectionCode.SOURCE_SLICE_MISMATCH), + ({"detector_label": ""}, _MentionRejectionCode.CONTRADICTORY_CANDIDATE), + ({"provenance": "legacy"}, _MentionRejectionCode.UNSUPPORTED_PROVENANCE), + ], +) +def test_finalization_rejects_unverifiable_candidate_fields( + changes: dict[str, object], + code: _MentionRejectionCode, +) -> None: + target_token = _MentionTargetToken() + token = _CandidateToken() + candidate = _ProvisionalCandidate( + token, + target_token, + 0, + 5, + "Alice", + "name", + _MentionProvenance.SPAN_DETECTOR, + ) + + result = _finalize_mentions( + (_MentionTarget(target_token, _DatumId("target"), "Alice"),), + (replace(candidate, **changes),), + (_ValidationDecision(token, _ValidationDecisionKind.KEEP),), + limits=_MENTION_LIMITS, + ) + + assert result == _MentionRejected(code, target_token) + + +@pytest.mark.parametrize( + ("decisions", "code"), + [ + ((), _MentionRejectionCode.MISSING_DECISION), + ( + ( + _ValidationDecisionKind.KEEP, + _ValidationDecisionKind.KEEP, + ), + _MentionRejectionCode.DUPLICATE_DECISION, + ), + ], +) +def test_finalization_requires_exactly_one_terminal_decision( + decisions: tuple[_ValidationDecisionKind, ...], + code: _MentionRejectionCode, +) -> None: + target_token = _MentionTargetToken() + candidate_token = _CandidateToken() + candidate = _ProvisionalCandidate( + candidate_token, + target_token, + 0, + 5, + "Alice", + "name", + _MentionProvenance.SPAN_DETECTOR, + ) + + result = _finalize_mentions( + (_MentionTarget(target_token, _DatumId("target"), "Alice"),), + (candidate,), + tuple(_ValidationDecision(candidate_token, kind) for kind in decisions), + limits=_MENTION_LIMITS, + ) + + assert result == _MentionRejected(code, target_token) + + +def test_finalization_rejects_foreign_decision_and_overlapping_final_mentions() -> None: + target_token = _MentionTargetToken() + target = _MentionTarget(target_token, _DatumId("target"), "Alice") + first_token = _CandidateToken() + second_token = _CandidateToken() + first = _ProvisionalCandidate( + first_token, + target_token, + 0, + 5, + "Alice", + "name", + _MentionProvenance.SPAN_DETECTOR, + ) + second = _ProvisionalCandidate( + second_token, + target_token, + 1, + 4, + "lic", + "alias", + _MentionProvenance.EXACT_AUGMENTER, + ) + foreign = _ValidationDecision(_CandidateToken(), _ValidationDecisionKind.KEEP) + + assert _finalize_mentions((target,), (first,), (foreign,), limits=_MENTION_LIMITS) == _MentionRejected( + _MentionRejectionCode.FOREIGN_TOKEN + ) + assert _finalize_mentions( + (target,), + (first, second), + ( + _ValidationDecision(first_token, _ValidationDecisionKind.KEEP), + _ValidationDecision(second_token, _ValidationDecisionKind.KEEP), + ), + limits=_MENTION_LIMITS, + ) == _MentionRejected(_MentionRejectionCode.OVERLAP, target_token) + + +def test_finalization_collapses_exact_lineage_duplicates_and_honors_drop() -> None: + target_token = _MentionTargetToken() + first_token = _CandidateToken() + dropped_token = _CandidateToken() + first = _ProvisionalCandidate( + first_token, + target_token, + 0, + 5, + "Alice", + "name", + _MentionProvenance.SPAN_DETECTOR, + ) + dropped = _ProvisionalCandidate( + dropped_token, + target_token, + 6, + 9, + "Bob", + "name", + _MentionProvenance.SPAN_DETECTOR, + ) + + result = _finalize_mentions( + (_MentionTarget(target_token, _DatumId("target"), "Alice Bob"),), + (first, first, dropped), + ( + _ValidationDecision(first_token, _ValidationDecisionKind.KEEP), + _ValidationDecision(dropped_token, _ValidationDecisionKind.DROP), + ), + limits=_MENTION_LIMITS, + ) + + assert isinstance(result, _DetectedGraph) + assert tuple((mention.start, mention.end) for mention in result.mentions) == ((0, 5),) + assert "Alice" not in repr(result) + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(result) diff --git a/tests/engine/execution/test_phase6_ndd_backend.py b/tests/engine/execution/test_phase6_ndd_backend.py new file mode 100644 index 00000000..e8a010e2 --- /dev/null +++ b/tests/engine/execution/test_phase6_ndd_backend.py @@ -0,0 +1,228 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from typing import Any, cast + +import pandas as pd +import pytest + +from anonymizer.config.anonymizer_config import AnonymizerConfig +from anonymizer.config.replace_strategies import Redact +from anonymizer.engine.constants import ( + COL_PHASE6_AUGMENTED, + COL_PHASE6_CONTEXT, + COL_PHASE6_VALIDATION, + COL_RAW_DETECTED, + COL_TEXT, + _jinja, +) +from anonymizer.engine.execution.graph import _DatumId +from anonymizer.engine.execution.mention_admission import ( + _CandidateToken, + _MentionProvenance, + _MentionTarget, + _MentionTargetToken, + _ProvisionalCandidate, +) +from anonymizer.engine.execution.phase6_ndd_backend import ( + _Phase6NddBackend, + _Phase6NddStageError, +) +from anonymizer.engine.execution.phase6_runtime import ( + _Phase6AugmentationWork, + _Phase6CandidateWork, + _Phase6ValidationWork, +) +from anonymizer.engine.ndd.adapter import NddAdapter, WorkflowRunResult +from tests.streaming.structured_trace_prototype import build_synthetic_anonymizer + + +@dataclass +class _ScriptedAdapter: + detector_payload: object + calls: list[tuple[str, tuple[str, ...]]] + prompts: list[str] = field(default_factory=list) + augmentation_payload: object = field(default_factory=lambda: {"entities": []}) + validation_payload: object = field( + default_factory=lambda: {"decisions": [{"ordinal": 0, "decision": "keep", "proposed_label": None}]} + ) + + def run_workflow( + self, + dataframe: pd.DataFrame, + *, + workflow_name: str, + columns: list[Any], + **_: Any, + ) -> WorkflowRunResult: + self.calls.append((workflow_name, tuple(dataframe.columns))) + self.prompts.extend(str(column.prompt) for column in columns) + output = dataframe.copy() + if workflow_name == "phase6-detect": + output[COL_RAW_DETECTED] = [self.detector_payload] + elif workflow_name == "phase6-augment": + output[COL_PHASE6_AUGMENTED] = [self.augmentation_payload] + elif workflow_name == "phase6-validate": + output[COL_PHASE6_VALIDATION] = [self.validation_payload] + else: # pragma: no cover - challenge guard + raise AssertionError(workflow_name) + return WorkflowRunResult(output, []) + + +def _backend(adapter: _ScriptedAdapter) -> _Phase6NddBackend: + anonymizer = build_synthetic_anonymizer({}) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + return _Phase6NddBackend(cast(NddAdapter, adapter), plan.invocation) + + +def test_phase6_ndd_detector_uses_only_target_text_and_preserves_exact_spans() -> None: + payload = json.dumps({"entities": [{"text": "Alice", "label": "name", "start": 0, "end": 5, "score": 0.9}]}) + adapter = _ScriptedAdapter(payload, []) + backend = _backend(adapter) + target = _MentionTarget(_MentionTargetToken(), _DatumId("target"), "Alice met Bob") + + proposals = backend.detect(_Phase6CandidateWork(target)) + + assert tuple((item.start, item.end, item.source_slice, item.detector_label) for item in proposals) == ( + (0, 5, "Alice", "name"), + ) + assert adapter.calls == [("phase6-detect", (COL_TEXT,))] + + +@pytest.mark.parametrize( + "payload", + [ + {"entities": [{"text": "Alice", "label": "name", "start": 0}]}, + {"entities": "not-a-list"}, + {"entities": [], "unexpected": True}, + ], +) +def test_phase6_ndd_detector_rejects_malformed_payload_without_partial_candidates(payload: object) -> None: + backend = _backend(_ScriptedAdapter(payload, [])) + target = _MentionTarget(_MentionTargetToken(), _DatumId("target"), "Alice") + + with pytest.raises(_Phase6NddStageError): + backend.detect(_Phase6CandidateWork(target)) + + +def test_phase6_ndd_augmentation_uses_valid_jinja_for_target_and_context() -> None: + adapter = _ScriptedAdapter({"entities": []}, []) + backend = _backend(adapter) + target = _MentionTarget(_MentionTargetToken(), _DatumId("target"), "Alice") + + assert backend.augment(_Phase6AugmentationWork(target, ())) == () + + prompt = adapter.prompts[-1] + assert "{{ {{" not in prompt + assert _jinja(COL_TEXT) in prompt + assert _jinja(COL_PHASE6_CONTEXT) in prompt + + +@pytest.mark.parametrize( + "field_value", + [ + pytest.param("0", id="string"), + pytest.param(0.0, id="float"), + pytest.param(False, id="boolean"), + ], +) +@pytest.mark.parametrize("field_name", ["start", "end"]) +def test_phase6_ndd_augmentation_rejects_non_integer_offsets(field_name: str, field_value: object) -> None: + entity: dict[str, object] = { + "start": 0, + "end": 5, + "source_slice": "Alice", + "detector_label": "name", + } + entity[field_name] = field_value + payload = {"entities": [entity]} + backend = _backend(_ScriptedAdapter({"entities": []}, [], augmentation_payload=payload)) + target = _MentionTarget(_MentionTargetToken(), _DatumId("target"), "Alice") + + with pytest.raises(_Phase6NddStageError): + backend.augment(_Phase6AugmentationWork(target, ())) + + +@pytest.mark.parametrize("field_name", ["source_slice", "detector_label"]) +def test_phase6_ndd_augmentation_rejects_coerced_string_fields(field_name: str) -> None: + entity: dict[str, object] = { + "start": 0, + "end": 5, + "source_slice": "Alice", + "detector_label": "name", + } + entity[field_name] = b"coerced" + payload = {"entities": [entity]} + backend = _backend(_ScriptedAdapter({"entities": []}, [], augmentation_payload=payload)) + target = _MentionTarget(_MentionTargetToken(), _DatumId("target"), "Alice") + + with pytest.raises(_Phase6NddStageError): + backend.augment(_Phase6AugmentationWork(target, ())) + + +def test_phase6_ndd_validator_rejects_a_missing_candidate_decision() -> None: + backend = _backend(_ScriptedAdapter({"entities": []}, [])) + target = _MentionTarget(_MentionTargetToken(), _DatumId("target"), "Alice Bob") + candidates = tuple( + _ProvisionalCandidate( + _CandidateToken(), + target.token, + start, + end, + target.text[start:end], + "name", + _MentionProvenance.SPAN_DETECTOR, + ) + for start, end in ((0, 5), (6, 9)) + ) + + with pytest.raises(_Phase6NddStageError): + backend.validate(_Phase6ValidationWork(target, candidates)) + + +@pytest.mark.parametrize( + "ordinal", + [ + pytest.param("0", id="string"), + pytest.param(0.0, id="float"), + pytest.param(False, id="boolean"), + ], +) +def test_phase6_ndd_validator_rejects_non_integer_ordinals(ordinal: object) -> None: + payload = {"decisions": [{"ordinal": ordinal, "decision": "drop", "proposed_label": None}]} + backend = _backend(_ScriptedAdapter({"entities": []}, [], validation_payload=payload)) + target = _MentionTarget(_MentionTargetToken(), _DatumId("target"), "Alice") + candidate = _ProvisionalCandidate( + _CandidateToken(), + target.token, + 0, + 5, + "Alice", + "name", + _MentionProvenance.SPAN_DETECTOR, + ) + + with pytest.raises(_Phase6NddStageError): + backend.validate(_Phase6ValidationWork(target, (candidate,))) + + +def test_phase6_ndd_validator_rejects_a_coerced_proposed_label() -> None: + payload = {"decisions": [{"ordinal": 0, "decision": "reclass", "proposed_label": b"name"}]} + backend = _backend(_ScriptedAdapter({"entities": []}, [], validation_payload=payload)) + target = _MentionTarget(_MentionTargetToken(), _DatumId("target"), "Alice") + candidate = _ProvisionalCandidate( + _CandidateToken(), + target.token, + 0, + 5, + "Alice", + "name", + _MentionProvenance.SPAN_DETECTOR, + ) + + with pytest.raises(_Phase6NddStageError): + backend.validate(_Phase6ValidationWork(target, (candidate,))) diff --git a/tests/engine/execution/test_phase6_redact.py b/tests/engine/execution/test_phase6_redact.py new file mode 100644 index 00000000..0086f244 --- /dev/null +++ b/tests/engine/execution/test_phase6_redact.py @@ -0,0 +1,196 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib +import importlib.util +import pickle +from dataclasses import replace + +import pytest + +from anonymizer.engine.execution.graph import _DatumId +from anonymizer.engine.execution.mention_admission import ( + _AnchoredMention, + _DetectedGraph, + _MentionId, + _MentionProvenance, + _MentionTarget, + _MentionTargetToken, +) +from anonymizer.engine.execution.mention_resolution import _ClusteredGraph, _ClusterId, _EntityCluster +from anonymizer.engine.execution.redact_patches import ( + _apply_redact_patches, + _bind_patch_manifest, + _BoundPatchManifest, + _build_patch_manifest, + _materialize_redact_patches, + _PatchManifest, + _PatchRejected, + _PatchRejectionCode, + _PatchToken, + _RedactPatch, + _ReturnedRedact, + _VerifiedGraph, + _verify_redact_patches, +) +from anonymizer.engine.execution.role_policy import ( + _classify_roles, + _compile_role_policy, + _ResolvedGraph, + _RolePolicy, + _RolePolicyVersion, +) + + +def test_phase6_redact_test_infrastructure() -> None: + assert _ResolvedGraph.__name__ == "_ResolvedGraph" + + +def test_phase6_redact_module_exposes_mention_keyed_verification_boundary() -> None: + module_name = "anonymizer.engine.execution.redact_patches" + assert importlib.util.find_spec(module_name) is not None, "Phase 6 Redact patch module is missing" + module = importlib.import_module(module_name) + + assert callable(getattr(module, "_verify_redact_patches", None)) + + +def _resolved_graph( + text: str, + spans: tuple[tuple[int, int, str], ...], +) -> tuple[_ResolvedGraph, _MentionTargetToken]: + target_token = _MentionTargetToken() + target = _MentionTarget(target_token, _DatumId("target"), text) + mentions = tuple( + _AnchoredMention( + _MentionId(), + target.datum_id, + start, + end, + text[start:end], + label, + _MentionProvenance.SPAN_DETECTOR, + ) + for start, end, label in spans + ) + clustered = _ClusteredGraph( + _DetectedGraph((target,), mentions), + tuple(_EntityCluster(_ClusterId(), (mention.id,), ()) for mention in mentions), + (), + ) + policy = _compile_role_policy(_RolePolicyVersion.V1, ()) + assert isinstance(policy, _RolePolicy) + resolved = _classify_roles(clustered, policy) + assert isinstance(resolved, _ResolvedGraph) + return resolved, target_token + + +def _bound_and_patches(resolved: _ResolvedGraph) -> tuple[_BoundPatchManifest, tuple[_RedactPatch, ...]]: + manifest = _build_patch_manifest(resolved) + assert isinstance(manifest, _PatchManifest) + bound = _bind_patch_manifest(manifest) + assert isinstance(bound, _BoundPatchManifest) + patches = _materialize_redact_patches(bound) + assert isinstance(patches, tuple) + return bound, patches + + +def test_exact_reconstruction_protects_only_anchored_repeated_occurrence() -> None: + resolved, target_token = _resolved_graph("Alice and Alice", ((0, 5, "name"),)) + bound, patches = _bound_and_patches(resolved) + + result = _verify_redact_patches( + bound, + patches, + (_ReturnedRedact(target_token, "[REDACTED] and Alice"),), + ) + + assert isinstance(result, _VerifiedGraph) + assert result.datums[0].output == "[REDACTED] and Alice" + assert result.datums[0].applied is True + assert "Alice" not in repr(result) + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(result) + + +def test_patch_application_is_anchored_and_does_not_reconsider_inserted_text() -> None: + resolved, target_token = _resolved_graph("Alice and Alice", ((0, 5, "name"),)) + _bound, patches = _bound_and_patches(resolved) + + returned = _apply_redact_patches(resolved, patches) + + assert returned == (_ReturnedRedact(target_token, "[REDACTED] and Alice"),) + + +def test_verified_no_work_path_requires_exact_unchanged_output_and_zero_patches() -> None: + resolved, target_token = _resolved_graph("plain text", ()) + bound, patches = _bound_and_patches(resolved) + + accepted = _verify_redact_patches( + bound, + patches, + (_ReturnedRedact(target_token, "plain text"),), + ) + rejected = _verify_redact_patches( + bound, + patches, + (_ReturnedRedact(target_token, "changed"),), + ) + + assert isinstance(accepted, _VerifiedGraph) + assert accepted.datums[0].applied is False + assert rejected == _PatchRejected(_PatchRejectionCode.RELEASE_PREDICATE_FAILED, target_token) + + +@pytest.mark.parametrize("fault", ["missing", "duplicate", "foreign", "offset", "replacement", "target"]) +def test_patch_verification_rejects_every_non_bijective_or_inexact_patch(fault: str) -> None: + resolved, target_token = _resolved_graph("Alice", ((0, 5, "name"),)) + bound, patches = _bound_and_patches(resolved) + patch = patches[0] + match fault: + case "missing": + corrupted = () + case "duplicate": + corrupted = (patch, patch) + case "foreign": + corrupted = (replace(patch, token=_PatchToken()),) + case "offset": + corrupted = (replace(patch, start=1),) + case "replacement": + corrupted = (replace(patch, replacement="Alice"),) + case "target": + corrupted = (replace(patch, target=_MentionTargetToken()),) + case unreachable: + raise AssertionError(unreachable) + + result = _verify_redact_patches( + bound, + corrupted, + (_ReturnedRedact(target_token, "[REDACTED]"),), + ) + + assert isinstance(result, _PatchRejected) + assert result.code in {_PatchRejectionCode.INVALID_PATCH, _PatchRejectionCode.FOREIGN_TOKEN} + + +@pytest.mark.parametrize("fault", ["missing", "duplicate", "foreign", "mismatch"]) +def test_output_verification_requires_one_exact_result_per_target(fault: str) -> None: + resolved, target_token = _resolved_graph("Alice", ((0, 5, "name"),)) + bound, patches = _bound_and_patches(resolved) + valid = _ReturnedRedact(target_token, "[REDACTED]") + match fault: + case "missing": + returned = () + case "duplicate": + returned = (valid, valid) + case "foreign": + returned = (_ReturnedRedact(_MentionTargetToken(), "[REDACTED]"),) + case "mismatch": + returned = (_ReturnedRedact(target_token, "Alice"),) + case unreachable: + raise AssertionError(unreachable) + + result = _verify_redact_patches(bound, patches, returned) + + assert isinstance(result, _PatchRejected) diff --git a/tests/engine/execution/test_phase6_reference_conformance.py b/tests/engine/execution/test_phase6_reference_conformance.py new file mode 100644 index 00000000..d7ecc15f --- /dev/null +++ b/tests/engine/execution/test_phase6_reference_conformance.py @@ -0,0 +1,451 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import pytest + +import anonymizer.engine.execution.phase6_runtime as phase6_runtime +from anonymizer.engine.execution.accounting_admission import _compile_accounting_plan +from anonymizer.engine.execution.accounting_evidence import _Dispatch, _SuccessRecord +from anonymizer.engine.execution.accounting_ledger import _AccountingLedger +from anonymizer.engine.execution.accounting_outcomes import _GroupReleased +from anonymizer.engine.execution.accounting_plan import ( + _AccountingLimits, + _AccountingPlan, + _DatumTaskSubject, + _TaskKey, +) +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextBackendCapability, + _ContextExecutionContract, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.graph import ( + _AtomicGroup, + _CoherenceScope, + _ContextScope, + _DatumDependency, + _DatumId, + _DatumPurpose, + _ProtectionGraph, + _TextDatum, + _trivial_graph, +) +from anonymizer.engine.execution.mention_admission import ( + _AnchoredMention, + _MentionLimits, + _ValidationDecision, + _ValidationDecisionKind, +) +from anonymizer.engine.execution.mention_resolution import ( + _ClusteredGraph, + _DistinctSubjectEvidence, + _EvidenceVersion, + _SameSubjectEvidence, + _SubjectEvidence, +) +from anonymizer.engine.execution.phase6_plan import _compile_phase6_plan, _Phase6Plan +from anonymizer.engine.execution.phase6_runtime import ( + _CandidateProposal, + _Phase6AugmentationWork, + _Phase6CandidateWork, + _Phase6ResolverWork, + _Phase6Runtime, + _Phase6ValidationWork, +) +from anonymizer.engine.execution.role_policy import _RolePolicy +from tests.engine.execution.phase6_reference_model import ( + ReferenceCandidate, + ReferenceCase, + ReferenceEventKind, + finite_reference_cases, + reduce_reference, + schedule_reference_cases, +) + + +def _datum_id(task: _TaskKey) -> _DatumId: + assert isinstance(task.subject, _DatumTaskSubject) + return task.subject.datum_id + + +@pytest.mark.parametrize("case", finite_reference_cases(), ids=lambda case: case.name) +def test_phase6_runtime_matches_every_frozen_reference_schedule( + case: ReferenceCase, + monkeypatch: pytest.MonkeyPatch, +) -> None: + expected = reduce_reference(case) + observed_clusters: list[_ClusteredGraph] = [] + classify_roles = phase6_runtime._classify_roles + + def _observe_clusters(clustered: _ClusteredGraph, policy: _RolePolicy) -> object: + observed_clusters.append(clustered) + return classify_roles(clustered, policy) + + monkeypatch.setattr(phase6_runtime, "_classify_roles", _observe_clusters) + backend = _ReferenceBackend(case) + execution = _Phase6Runtime(backend).run(_plan(case)) + + released = tuple( + group_index + for group_index, group in enumerate(execution.accounting.groups) + if isinstance(group, _GroupReleased) + ) + expected_released_outputs = tuple( + (target_index, expected.outputs[target_index]) + for group_index in expected.released_groups + for target_index in _groups(case)[group_index] + ) + actual_released_outputs = tuple( + (int(datum.datum_id.value.removeprefix("target-")), datum.output) for datum in execution.released + ) + + assert released == expected.released_groups + assert actual_released_outputs == expected_released_outputs + assert ( + type(execution.accounting.invocation).__name__ + == { + "completed": "_InvocationCompleted", + "cancelled": "_InvocationCancelled", + "failed": "_InvocationFailed", + "lost": "_InvocationLost", + "inconsistent": "_InvocationInconsistent", + }[expected.schedule.invocation] + ) + assert backend.closed is (expected.schedule.cleanup == "accepted") + assert _normalize_mentions(backend.observed_mentions) == tuple( + (mention.target, mention.start, mention.end, mention.source_slice, mention.label) + for mention in expected.mentions + ) + assert _normalize_clusters(observed_clusters) == expected.clusters + + +@pytest.mark.parametrize("case", schedule_reference_cases(), ids=lambda case: case.name) +def test_production_accounting_matches_every_frozen_lifecycle_schedule(case: ReferenceCase) -> None: + expected = reduce_reference(case) + actual = _execute_accounting_schedule(case) + + assert actual == ( + expected.schedule.invocation, + expected.schedule.task_outcomes, + expected.schedule.cancellation, + expected.schedule.released_subjects, + ) + + +def _execute_accounting_schedule( + case: ReferenceCase, +) -> tuple[str, tuple[tuple[str, str], ...], str, tuple[str, ...]]: + plan = _accounting_plan(case) + ledger: _AccountingLedger[str] = _AccountingLedger(plan) + ledger.open() + dispatches: dict[str, _Dispatch] = {} + terminal_subjects: set[str] = set() + finalized = False + verified = False + cleanup = "unconfirmed" + teardown = "unconfirmed" + immutable_result = False + cancellation = "none" + result = None + + for event in case.events: + subject = event.subject if event.subject != "invocation" else "target-0" + match event.kind: + case ReferenceEventKind.DISPATCH: + ready = {_datum_id(task).value: task for task in ledger.ready_tasks()} + if subject in ready: + dispatches[subject] = ledger.dispatch(ready[subject]) + case ReferenceEventKind.TERMINAL: + dispatch = dispatches.get(subject) + if dispatch is None: + continue + if event.outcome == "contradictory": + records = ( + _SuccessRecord(dispatch, "first"), + _SuccessRecord(dispatch, "second"), + ) + ledger.reconcile((dispatch,), records, trusted_run_record=True) + elif event.outcome == "failed": + ledger.accept_failure(dispatch) + else: + ledger.accept_success(dispatch, f"verified-{subject}") + terminal_subjects.add(subject) + case ReferenceEventKind.CANCEL: + cancellation = ( + "before_dispatch" if not dispatches else "after_terminal" if terminal_subjects else "after_dispatch" + ) + ledger.request_cancellation() + for dispatched_subject, dispatch in dispatches.items(): + if dispatched_subject not in terminal_subjects: + ledger.acknowledge_stop(dispatch) + case ReferenceEventKind.LOSS: + dispatch = dispatches.get(subject) + if dispatch is not None: + ledger.mark_transport_lost(dispatch) + case ReferenceEventKind.CANDIDATE_DECISION: + dispatch = dispatches.get(subject) + if dispatch is not None: + ledger.accept_success(dispatch, f"candidate-{subject}") + case ReferenceEventKind.EVIDENCE: + dispatch = dispatches.get(subject) + if dispatch is not None: + if event.outcome == "duplicate": + record = _SuccessRecord(dispatch, f"evidence-{subject}") + ledger.reconcile((dispatch,), (record, record), trusted_run_record=True) + else: + ledger.accept_success(dispatch, f"evidence-{subject}") + case ReferenceEventKind.FINALIZE: + finalized = True + case ReferenceEventKind.VERIFY: + verified = True + case ReferenceEventKind.CLEANUP: + cleanup = event.outcome + if cleanup == "unconfirmed": + ledger.mark_cleanup_unconfirmed() + elif cleanup == "failed": + ledger.mark_cleanup_failed() + case ReferenceEventKind.IMMUTABLE_ACCEPT: + immutable_result = event.outcome == "accepted" + result = ledger.finish( + group_release_predicate=lambda _outputs: ( + finalized and verified and cleanup == "accepted" and teardown != "failed" and immutable_result + ) + ) + case ReferenceEventKind.TEARDOWN: + teardown = event.outcome + if teardown == "failed" and result is None: + ledger.mark_cleanup_failed() + case ReferenceEventKind.RELEASE: + if result is None: + result = ledger.finish(group_release_predicate=lambda _outputs: False) + break + case _: + pass + + if result is None: + if cleanup == "unconfirmed": + ledger.mark_cleanup_unconfirmed() + result = ledger.finish(group_release_predicate=lambda _outputs: False) + invocation = type(result.invocation).__name__.removeprefix("_Invocation").lower() + tasks = tuple( + (_datum_id(outcome.task).value, type(outcome).__name__.removeprefix("_Task").lower()) + for outcome in result.tasks + ) + released = tuple( + datum_id.value + for group in result.groups + if isinstance(group, _GroupReleased) + for datum_id, _candidate in group.outputs + ) + return invocation, tasks, cancellation, released + + +def _accounting_plan(case: ReferenceCase) -> _AccountingPlan: + graph = _trivial_graph( + tuple( + _TextDatum(_DatumId(f"target-{index}"), text, _DatumPurpose.TARGET) for index, text in enumerate(case.texts) + ) + ) + compiled = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=4, max_datum_bytes=128, max_graph_bytes=512), + ) + assert isinstance(compiled, _AccountingPlan) + return compiled + + +class _ReferenceBackend: + """Translate declarations, never reduced expected results, into runtime effects.""" + + def __init__(self, case: ReferenceCase) -> None: + self._case = case + self.observed_mentions: list[_AnchoredMention] = [] + self.closed = False + + def context_capability(self) -> _ContextBackendCapability: + return _contract_and_capability()[1] + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + target = _target_index(work.target.datum_id) + if not _group_passes(self._case)[_group_for_target(self._case, target)]: + raise RuntimeError("declared local target failure") + return tuple( + _CandidateProposal(candidate.start, candidate.end, candidate.source_slice, candidate.label) + for candidate in self._case.candidates + if candidate.target == target + ) + + def augment(self, work: _Phase6AugmentationWork) -> tuple[_CandidateProposal, ...]: + del work + return () + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + target = _target_index(work.target.datum_id) + declarations = tuple(candidate for candidate in self._case.candidates if candidate.target == target) + decisions: list[_ValidationDecision] = [] + for candidate, declaration in zip(work.candidates, declarations, strict=True): + if declaration.decision == "missing": + continue + kind = { + "keep": _ValidationDecisionKind.KEEP, + "drop": _ValidationDecisionKind.DROP, + "reclass": _ValidationDecisionKind.RECLASS, + }[declaration.decision] + decisions.append(_ValidationDecision(candidate.token, kind, declaration.reclassified_label)) + return tuple(decisions) + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SubjectEvidence, ...]: + self.observed_mentions.extend(work.eligible_mentions) + owner = _target_index(work.owner.datum_id) + mentions = { + (candidate.target, candidate.start, candidate.end, candidate.source_slice): mention + for candidate in self._case.candidates + for mention in work.eligible_mentions + if _mention_matches(candidate, mention) + } + evidence: list[_SubjectEvidence] = [] + for declaration in self._case.evidence: + left = self._case.candidates[declaration.left_candidate] + right = self._case.candidates[declaration.right_candidate] + if owner != left.target: + continue + evidence_type = _SameSubjectEvidence if declaration.kind == "same_subject" else _DistinctSubjectEvidence + evidence.append( + evidence_type( + work.owner.token, + mentions[(left.target, left.start, left.end, left.source_slice)].id, + mentions[(right.target, right.start, right.end, right.source_slice)].id, + _EvidenceVersion.V1, + ) + ) + return tuple(evidence) + + def close_phase6(self) -> bool: + self.closed = True + return True + + +def _plan(case: ReferenceCase) -> _Phase6Plan: + datums = tuple( + _TextDatum(_DatumId(f"target-{index}"), text, _DatumPurpose.TARGET) for index, text in enumerate(case.texts) + ) + ids = tuple(datum.id for datum in datums) + context_scopes = tuple( + _ContextScope(datum_id, tuple(candidate for candidate in ids if candidate != datum_id)) for datum_id in ids + ) + groups = _groups(case) + graph = _ProtectionGraph( + datums, + (), + context_scopes, + tuple(_CoherenceScope(tuple(ids[index] for index in group)) for group in groups), + tuple(_AtomicGroup(tuple(ids[index] for index in group)) for group in groups), + tuple(_DatumDependency(ids[left], ids[right]) for left, right in case.dependencies), + ) + contract, capability = _contract_and_capability() + compiled = _compile_phase6_plan( + graph, + accounting_limits=_AccountingLimits( + max_datums=8, + max_datum_bytes=128, + max_graph_bytes=512, + max_stages=8, + ), + context_contract=contract, + capability=capability, + mention_limits=_MentionLimits(16, 16, 64, 128), + ) + assert isinstance(compiled, _Phase6Plan) + return compiled + + +def _contract_and_capability() -> tuple[_ContextExecutionContract, _ContextBackendCapability]: + limits = _ContextLimits(8, 128, 16, 1024) + contract = _ContextExecutionContract( + _ContextProfile.TARGET_CONTEXT_V1, + _ContextSchemaVersion.V1, + limits, + True, + _ContextOrdering.DECLARED, + (_BackendArtifactClass.CONTEXT_REQUEST,), + ) + capability = _ContextBackendCapability( + contract.profile, + contract.schema_version, + limits, + True, + contract.ordering, + contract.required_artifacts, + _RetentionPosture.DISABLED, + ) + return contract, capability + + +def _groups(case: ReferenceCase) -> tuple[tuple[int, ...], ...]: + return case.groups or tuple((index,) for index in range(len(case.texts))) + + +def _group_passes(case: ReferenceCase) -> tuple[bool, ...]: + return case.group_passes or tuple(True for _group in _groups(case)) + + +def _group_for_target(case: ReferenceCase, target: int) -> int: + return next(index for index, group in enumerate(_groups(case)) if target in group) + + +def _target_index(datum_id: _DatumId) -> int: + return int(datum_id.value.removeprefix("target-")) + + +def _mention_matches(candidate: ReferenceCandidate, mention: _AnchoredMention) -> bool: + return ( + _target_index(mention.target_datum_id) == candidate.target + and mention.start == candidate.start + and mention.end == candidate.end + and mention.source_slice == candidate.source_slice + ) + + +def _ordered_mentions(graphs: list[_ClusteredGraph]) -> tuple[_AnchoredMention, ...]: + mentions = {mention.id: mention for graph in graphs for mention in graph.detected.mentions} + return tuple( + sorted( + mentions.values(), + key=lambda mention: (_target_index(mention.target_datum_id), mention.start, mention.end), + ) + ) + + +def _normalize_mentions(mentions: list[_AnchoredMention]) -> tuple[tuple[int, int, int, str, str], ...]: + unique = {mention.id: mention for mention in mentions} + return tuple( + ( + _target_index(mention.target_datum_id), + mention.start, + mention.end, + mention.source_slice, + mention.detector_label, + ) + for mention in sorted( + unique.values(), + key=lambda mention: (_target_index(mention.target_datum_id), mention.start, mention.end), + ) + ) + + +def _normalize_clusters(graphs: list[_ClusteredGraph]) -> tuple[tuple[int, ...], ...]: + mentions = _ordered_mentions(graphs) + position = {mention.id: index for index, mention in enumerate(mentions)} + clusters = { + tuple(sorted(position[mention_id] for mention_id in cluster.ordered_mention_ids)) + for graph in graphs + for cluster in graph.clusters + } + return tuple(sorted(clusters)) diff --git a/tests/engine/execution/test_phase6_reference_model.py b/tests/engine/execution/test_phase6_reference_model.py new file mode 100644 index 00000000..32e223c4 --- /dev/null +++ b/tests/engine/execution/test_phase6_reference_model.py @@ -0,0 +1,223 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import ast +import json +from dataclasses import replace +from pathlib import Path + +import pytest + +from tests.engine.execution.phase6_reference_model import ( + INDEPENDENCE_RELATION, + ReferenceCandidate, + ReferenceCase, + ReferenceEvent, + ReferenceEventKind, + ReferenceEvidence, + canonical_schedule, + default_schedule, + finite_reference_cases, + lifecycle_reference_cases, + ordered_race_schedules, + reduce_reference, + reference_manifest, +) + + +def test_phase6_reference_model_is_independent_and_manifest_is_frozen() -> None: + source_path = Path(__file__).with_name("phase6_reference_model.py") + tree = ast.parse(source_path.read_text(encoding="utf-8")) + imported_roots = { + alias.name.split(".", maxsplit=1)[0] + for node in ast.walk(tree) + if isinstance(node, (ast.Import, ast.ImportFrom)) + for alias in node.names + } + + assert imported_roots.isdisjoint({"anonymizer", "pandas", "pytest"}) + manifest = json.loads(Path(__file__).with_name("phase6_reference_manifest.json").read_text(encoding="utf-8")) + assert reference_manifest() == manifest + assert manifest["case_count"] == len(finite_reference_cases()) + assert manifest["actual_event_count"] > manifest["canonical_trace_count"] + assert manifest["max_event_count"] > 0 + + +def test_phase6_reference_cases_contain_executable_bounded_schedules() -> None: + cases = finite_reference_cases() + + assert all(case.events for case in cases) + assert all(reduce_reference(case).event_count <= reduce_reference(case).max_event_count for case in cases) + schedule_class_counts = reference_manifest()["schedule_class_counts"] + assert isinstance(schedule_class_counts, dict) + assert schedule_class_counts == { + "cancel-after-verification": 1, + "cancel-dispatch": 2, + "dispatch-terminal": 2, + "cancel-terminal": 2, + "cleanup-failure": 1, + "contradictory-record-patch": 2, + "duplicate-resolver-completion": 1, + "finalize-release": 2, + "late-candidate-after-cancel": 1, + "late-candidate-after-loss": 1, + "late-evidence-after-cancel": 1, + "late-evidence-after-loss": 1, + "local-failure-independent-success": 1, + "success": 21, + "teardown-failure-after-acceptance": 1, + "teardown-acceptance": 2, + "verify-release": 2, + } + + assert {case.name for case in lifecycle_reference_cases()} == { + "cancel-before-dispatch", + "cancel-after-dispatch", + "late-candidate-after-cancel", + "late-evidence-after-cancel", + "late-candidate-after-loss", + "late-evidence-after-loss", + "duplicate-resolver-completion", + "patch-before-contradictory-record", + "contradictory-record-before-patch", + "local-failure-independent-success", + "cancel-after-verification", + "cleanup-failure", + "teardown-failure-after-acceptance", + } + assert all(case.events for case in lifecycle_reference_cases()) + assert all( + reduce_reference(case).event_count <= reduce_reference(case).max_event_count + for case in lifecycle_reference_cases() + ) + + +def test_phase6_schedule_canonicalization_collapses_only_commuting_swaps() -> None: + for left, right in INDEPENDENCE_RELATION: + first = ( + ReferenceEvent(ReferenceEventKind(left), "z"), + ReferenceEvent(ReferenceEventKind(right), "a"), + ) + second = tuple(reversed(first)) + assert canonical_schedule(first) == canonical_schedule(second) + + for _name, first, second in ordered_race_schedules(): + assert canonical_schedule(first) != canonical_schedule(second) + + +@pytest.mark.parametrize(("name", "first", "second"), ordered_race_schedules()) +def test_phase6_ordered_races_have_distinct_observable_outcomes( + name: str, + first: tuple[ReferenceEvent, ...], + second: tuple[ReferenceEvent, ...], +) -> None: + case = ReferenceCase("race", ("A",), (), events=first) + first_result = reduce_reference(case) + second_result = reduce_reference(replace(case, events=second)) + + assert first_result.schedule != second_result.schedule, name + if name == "dispatch-terminal": + assert first_result.schedule.task_terminal != second_result.schedule.task_terminal + elif name == "cancel-terminal": + assert first_result.schedule.cancellation != second_result.schedule.cancellation + else: + assert first_result.schedule.release != second_result.schedule.release + + +def test_phase6_lifecycle_schedules_are_fail_closed_without_rewriting_accepted_results() -> None: + results = {case.name: reduce_reference(case) for case in lifecycle_reference_cases()} + + assert results["late-candidate-after-cancel"].schedule.task_outcomes == (("target-0", "cancelled"),) + assert results["late-evidence-after-cancel"].schedule.task_outcomes == (("target-0", "cancelled"),) + assert results["late-candidate-after-loss"].schedule.task_outcomes == (("target-0", "lost"),) + assert results["late-evidence-after-loss"].schedule.task_outcomes == (("target-0", "lost"),) + assert results["duplicate-resolver-completion"].schedule.invocation == "inconsistent" + assert results["patch-before-contradictory-record"].schedule.invocation == "inconsistent" + assert results["local-failure-independent-success"].released_groups == (1,) + assert results["cancel-after-verification"].released_groups == () + assert results["cleanup-failure"].schedule.invocation == "inconsistent" + accepted = results["teardown-failure-after-acceptance"] + assert accepted.schedule.immutable_result + assert accepted.schedule.teardown == "failed" + assert accepted.schedule.invocation == "completed" + assert accepted.released_groups == (0,) + + +def test_phase6_reference_reducer_rejects_schedule_over_bound() -> None: + case = ReferenceCase("over-bound", ("A",), (), events=default_schedule()) + result = reduce_reference(case) + excessive = replace( + case, + events=case.events + + tuple(ReferenceEvent(ReferenceEventKind.CANCEL) for _index in range(result.max_event_count + 1)), + ) + + with pytest.raises(AssertionError, match="bound"): + reduce_reference(excessive) + + +def test_reference_oracle_keeps_repeated_occurrences_anchored_and_reconstructs_exactly() -> None: + result = reduce_reference( + ReferenceCase( + "repeated", + ("Alice and Alice",), + (ReferenceCandidate(0, 0, 5, "Alice", "name"),), + ) + ) + + assert result.rejection is None + assert result.outputs == ("[REDACTED] and Alice",) + assert result.clusters == ((0,),) + assert result.released_groups == (0,) + + +def test_reference_oracle_clusters_only_explicit_evidence_and_rejects_transitive_contradiction() -> None: + candidates = ( + ReferenceCandidate(0, 0, 1, "A", "name"), + ReferenceCandidate(0, 2, 3, "B", "name"), + ReferenceCandidate(0, 4, 5, "C", "name"), + ) + separate = reduce_reference(ReferenceCase("separate", ("A B C",), candidates)) + same = reduce_reference( + ReferenceCase( + "same", + ("A B C",), + candidates, + (ReferenceEvidence("same_subject", 0, 1),), + ) + ) + contradictory = reduce_reference( + ReferenceCase( + "contradictory", + ("A B C",), + candidates, + ( + ReferenceEvidence("same_subject", 0, 1), + ReferenceEvidence("same_subject", 1, 2), + ReferenceEvidence("distinct_subject", 0, 2), + ), + ) + ) + + assert separate.clusters == ((0,), (1,), (2,)) + assert same.clusters == ((0, 1), (2,)) + assert contradictory.rejection == "evidence_contradiction" + assert contradictory.released_groups == () + + +def test_reference_group_predicate_failure_is_monotone_through_dependencies() -> None: + result = reduce_reference( + ReferenceCase( + "propagation", + ("A", "B"), + (), + dependencies=((0, 1),), + groups=((0,), (1,)), + group_passes=(False, True), + ) + ) + + assert result.rejection is None + assert result.released_groups == () diff --git a/tests/engine/execution/test_phase6_resolution.py b/tests/engine/execution/test_phase6_resolution.py new file mode 100644 index 00000000..f562a8ca --- /dev/null +++ b/tests/engine/execution/test_phase6_resolution.py @@ -0,0 +1,140 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib +import importlib.util +from typing import cast + +import pytest + +from anonymizer.engine.execution.graph import _DatumId +from anonymizer.engine.execution.mention_admission import ( + _AnchoredMention, + _DetectedGraph, + _MentionId, + _MentionProvenance, + _MentionTarget, + _MentionTargetToken, +) +from anonymizer.engine.execution.mention_resolution import ( + _ClusteredGraph, + _DistinctSubjectEvidence, + _EvidenceVersion, + _ResolutionRejected, + _ResolutionRejectionCode, + _resolve_mentions, + _ResolverScope, + _SameSubjectEvidence, +) + + +def test_phase6_resolution_test_infrastructure() -> None: + assert _DetectedGraph.__name__ == "_DetectedGraph" + + +def test_phase6_resolution_module_exposes_explicit_evidence_boundary() -> None: + module_name = "anonymizer.engine.execution.mention_resolution" + assert importlib.util.find_spec(module_name) is not None, "Phase 6 mention resolution module is missing" + module = importlib.import_module(module_name) + + assert callable(getattr(module, "_resolve_mentions", None)) + + +def _detected_graph() -> tuple[_DetectedGraph, tuple[_MentionTargetToken, ...], tuple[_AnchoredMention, ...]]: + target_a = _MentionTargetToken() + target_b = _MentionTargetToken() + targets = ( + _MentionTarget(target_a, _DatumId("a"), "Alice met Bob"), + _MentionTarget(target_b, _DatumId("b"), "A. Example"), + ) + mentions = ( + _AnchoredMention(_MentionId(), targets[0].datum_id, 0, 5, "Alice", "name", _MentionProvenance.SPAN_DETECTOR), + _AnchoredMention(_MentionId(), targets[0].datum_id, 10, 13, "Bob", "name", _MentionProvenance.SPAN_DETECTOR), + _AnchoredMention( + _MentionId(), targets[1].datum_id, 0, 10, "A. Example", "name", _MentionProvenance.EXACT_AUGMENTER + ), + ) + return _DetectedGraph(targets, mentions), (target_a, target_b), mentions + + +def _members(result: _ClusteredGraph) -> frozenset[frozenset[_MentionId]]: + return frozenset(frozenset(cluster.ordered_mention_ids) for cluster in result.clusters) + + +def test_resolution_starts_singleton_and_never_merges_by_label_or_content() -> None: + detected, (target_a, target_b), mentions = _detected_graph() + + result = _resolve_mentions( + detected, + (_ResolverScope(target_a, (target_a, target_b)), _ResolverScope(target_b, (target_b,))), + (), + ) + + assert isinstance(result, _ClusteredGraph) + assert _members(result) == frozenset(frozenset((mention.id,)) for mention in mentions) + + +def test_resolution_merges_only_explicit_same_subject_components_independent_of_order() -> None: + detected, (target_a, target_b), mentions = _detected_graph() + scopes = (_ResolverScope(target_a, (target_a, target_b)), _ResolverScope(target_b, (target_a, target_b))) + first = _SameSubjectEvidence(target_a, mentions[0].id, mentions[2].id, _EvidenceVersion.V1) + duplicate_other_owner = _SameSubjectEvidence(target_b, mentions[2].id, mentions[0].id, _EvidenceVersion.V1) + + forward = _resolve_mentions(detected, scopes, (first, duplicate_other_owner)) + reverse = _resolve_mentions(detected, tuple(reversed(scopes)), (duplicate_other_owner, first)) + + assert isinstance(forward, _ClusteredGraph) + assert isinstance(reverse, _ClusteredGraph) + expected = frozenset((frozenset((mentions[0].id, mentions[2].id)), frozenset((mentions[1].id,)))) + assert _members(forward) == expected + assert _members(reverse) == expected + assert len(forward.accepted_evidence) == len(reverse.accepted_evidence) == 1 + + +def test_resolution_rejects_transitive_distinct_subject_contradiction() -> None: + detected, (target_a, target_b), mentions = _detected_graph() + scopes = (_ResolverScope(target_a, (target_a, target_b)), _ResolverScope(target_b, (target_a, target_b))) + evidence = ( + _SameSubjectEvidence(target_a, mentions[0].id, mentions[1].id, _EvidenceVersion.V1), + _SameSubjectEvidence(target_b, mentions[1].id, mentions[2].id, _EvidenceVersion.V1), + _DistinctSubjectEvidence(target_a, mentions[0].id, mentions[2].id, _EvidenceVersion.V1), + ) + + assert _resolve_mentions(detected, scopes, evidence) == _ResolutionRejected( + _ResolutionRejectionCode.EVIDENCE_CONTRADICTION + ) + + +@pytest.mark.parametrize("fault", ["self", "duplicate", "foreign", "ownerless"]) +def test_resolution_rejects_unattributable_or_malformed_evidence(fault: str) -> None: + detected, (target_a, target_b), mentions = _detected_graph() + scopes = (_ResolverScope(target_a, (target_a, target_b)), _ResolverScope(target_b, (target_b,))) + valid = _SameSubjectEvidence(target_a, mentions[0].id, mentions[2].id, _EvidenceVersion.V1) + match fault: + case "self": + evidence = (_SameSubjectEvidence(target_a, mentions[0].id, mentions[0].id, _EvidenceVersion.V1),) + case "duplicate": + evidence = (valid, _SameSubjectEvidence(target_a, mentions[2].id, mentions[0].id, _EvidenceVersion.V1)) + case "foreign": + evidence = (_SameSubjectEvidence(target_a, mentions[0].id, _MentionId(), _EvidenceVersion.V1),) + case "ownerless": + evidence = (_SameSubjectEvidence(target_a, mentions[2].id, mentions[2].id, _EvidenceVersion.V1),) + case unreachable: + raise AssertionError(unreachable) + + result = _resolve_mentions(detected, scopes, evidence) + + assert isinstance(result, _ResolutionRejected) + assert result.code in {_ResolutionRejectionCode.INVALID_EVIDENCE, _ResolutionRejectionCode.FOREIGN_TOKEN} + + +def test_resolver_rejects_a_malformed_detected_graph_without_inspecting_foreign_values() -> None: + detected, (target_a, target_b), _mentions = _detected_graph() + malformed = _DetectedGraph(detected.targets, cast(tuple[_AnchoredMention, ...], (object(),))) + scopes = (_ResolverScope(target_a, (target_a, target_b)), _ResolverScope(target_b, (target_b,))) + + result = _resolve_mentions(malformed, scopes, ()) + + assert result == _ResolutionRejected(_ResolutionRejectionCode.STALE_TOKEN) diff --git a/tests/engine/execution/test_phase6_roles.py b/tests/engine/execution/test_phase6_roles.py new file mode 100644 index 00000000..d16569f6 --- /dev/null +++ b/tests/engine/execution/test_phase6_roles.py @@ -0,0 +1,218 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib +import importlib.util +import json +from pathlib import Path +from typing import cast + +import pytest + +from anonymizer.engine.execution import role_policy as role_policy_module +from anonymizer.engine.execution.graph import _DatumId +from anonymizer.engine.execution.mention_admission import ( + _AnchoredMention, + _DetectedGraph, + _MentionId, + _MentionProvenance, + _MentionTarget, + _MentionTargetToken, +) +from anonymizer.engine.execution.mention_resolution import _ClusteredGraph, _ClusterId, _EntityCluster +from anonymizer.engine.execution.role_policy import ( + _ClassifiedRole, + _classify_roles, + _compile_role_policy, + _load_redact_role_policy, + _ResolvedGraph, + _RolePolicy, + _RolePolicyRejected, + _RolePolicyVersion, + _UnsupportedRole, +) + + +def test_phase6_role_test_infrastructure() -> None: + assert _ClusteredGraph.__name__ == "_ClusteredGraph" + + +def test_phase6_role_policy_module_exposes_closed_classification_boundary() -> None: + module_name = "anonymizer.engine.execution.role_policy" + assert importlib.util.find_spec(module_name) is not None, "Phase 6 role policy module is missing" + module = importlib.import_module(module_name) + + assert callable(getattr(module, "_classify_roles", None)) + + +def _clustered_graph(*labels: str) -> _ClusteredGraph: + target_token = _MentionTargetToken() + target = _MentionTarget(target_token, _DatumId("target"), "Alice Bob") + mentions = tuple( + _AnchoredMention( + _MentionId(), + target.datum_id, + index * 6, + index * 6 + 5 if index == 0 else index * 6 + 3, + "Alice" if index == 0 else "Bob", + label, + _MentionProvenance.SPAN_DETECTOR, + ) + for index, label in enumerate(labels) + ) + clusters = tuple(_EntityCluster(_ClusterId(), (mention.id,), ()) for mention in mentions) + return _ClusteredGraph(_DetectedGraph((target,), mentions), clusters, ()) + + +def test_role_policy_classifies_only_frozen_mappings_and_marks_unknown_labels_unsupported() -> None: + policy = _compile_role_policy(_RolePolicyVersion.V1, (("name", "person_name"),)) + assert isinstance(policy, _RolePolicy) + + result = _classify_roles(_clustered_graph("name", "custom_secret"), policy) + + assert isinstance(result, _ResolvedGraph) + assert isinstance(result.mentions[0].role_result, _ClassifiedRole) + assert result.mentions[0].role_result.role.value == "person_name" + assert isinstance(result.mentions[1].role_result, _UnsupportedRole) + assert result.policy_version is _RolePolicyVersion.V1 + assert len(result.policy_digest) == 64 + + +def test_role_policy_digest_and_results_are_declaration_order_invariant() -> None: + forward = _compile_role_policy(_RolePolicyVersion.V1, (("name", "person_name"), ("email", "contact"))) + reverse = _compile_role_policy(_RolePolicyVersion.V1, (("email", "contact"), ("name", "person_name"))) + + assert isinstance(forward, _RolePolicy) + assert isinstance(reverse, _RolePolicy) + assert forward.digest == reverse.digest + assert tuple(label for label, _role in forward.mappings) == ("email", "name") + assert tuple(label for label, _role in reverse.mappings) == ("email", "name") + + +def test_empty_redact_policy_is_fail_closed_without_blocking_structural_resolution() -> None: + policy = _compile_role_policy(_RolePolicyVersion.V1, ()) + assert isinstance(policy, _RolePolicy) + + result = _classify_roles(_clustered_graph("name"), policy) + + assert isinstance(result, _ResolvedGraph) + assert isinstance(result.mentions[0].role_result, _UnsupportedRole) + + +def test_role_policy_rejects_unknown_version_duplicate_labels_and_unsealed_policy() -> None: + assert isinstance(_compile_role_policy(cast(_RolePolicyVersion, "unknown"), ()), _RolePolicyRejected) + assert isinstance( + _compile_role_policy(_RolePolicyVersion.V1, (("name", "one"), ("name", "two"))), + _RolePolicyRejected, + ) + direct = _RolePolicy(_RolePolicyVersion.V1, (), "0" * 64) + + assert isinstance(_classify_roles(_clustered_graph("name"), direct), _RolePolicyRejected) + + +def test_redact_role_policy_manifest_freezes_fail_closed_structural_version() -> None: + manifest_path = ( + Path(__file__).parents[3] / "src" / "anonymizer" / "engine" / "execution" / "phase6_redact_role_policy.json" + ) + assert manifest_path.is_file() + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + policy = _compile_role_policy(_RolePolicyVersion(manifest["version"]), tuple(map(tuple, manifest["mappings"]))) + + assert isinstance(policy, _RolePolicy) + assert manifest == { + "digest": policy.digest, + "mappings": [], + "version": _RolePolicyVersion.V1.value, + } + + +@pytest.mark.parametrize( + "manifest_text", + [ + pytest.param("{", id="invalid-json"), + pytest.param( + json.dumps( + { + "digest": "e11a29db1af26c9572e1b4dec9e0a91e80966c6de1d813a378b64210a3bdfc40", + "mappings": [], + "unexpected": True, + "version": _RolePolicyVersion.V1.value, + } + ), + id="extra-key", + ), + pytest.param( + json.dumps( + { + "digest": "e11a29db1af26c9572e1b4dec9e0a91e80966c6de1d813a378b64210a3bdfc40", + "mappings": {}, + "version": _RolePolicyVersion.V1.value, + } + ), + id="non-list-mappings", + ), + pytest.param( + json.dumps( + { + "digest": "e11a29db1af26c9572e1b4dec9e0a91e80966c6de1d813a378b64210a3bdfc40", + "mappings": [["name"]], + "version": _RolePolicyVersion.V1.value, + } + ), + id="malformed-mapping", + ), + pytest.param( + json.dumps( + { + "digest": "e11a29db1af26c9572e1b4dec9e0a91e80966c6de1d813a378b64210a3bdfc40", + "mappings": [], + "version": 1, + } + ), + id="non-string-version", + ), + ], +) +def test_redact_role_policy_manifest_loader_rejects_noncanonical_content( + monkeypatch: pytest.MonkeyPatch, + manifest_text: str, +) -> None: + class _ManifestResource: + def joinpath(self, _name: str) -> _ManifestResource: + return self + + def read_text(self, *, encoding: str) -> str: + assert encoding == "utf-8" + return manifest_text + + monkeypatch.setattr(role_policy_module, "files", lambda _package: _ManifestResource()) + + assert isinstance(_load_redact_role_policy(), _RolePolicyRejected) + + +def test_redact_role_policy_manifest_loader_rejects_a_self_consistent_nonempty_policy( + monkeypatch: pytest.MonkeyPatch, +) -> None: + nonempty = _compile_role_policy(_RolePolicyVersion.V1, (("name", "person_name"),)) + assert isinstance(nonempty, _RolePolicy) + manifest_text = json.dumps( + { + "digest": nonempty.digest, + "mappings": [["name", "person_name"]], + "version": _RolePolicyVersion.V1.value, + } + ) + + class _ManifestResource: + def joinpath(self, _name: str) -> _ManifestResource: + return self + + def read_text(self, *, encoding: str) -> str: + assert encoding == "utf-8" + return manifest_text + + monkeypatch.setattr(role_policy_module, "files", lambda _package: _ManifestResource()) + + assert isinstance(_load_redact_role_policy(), _RolePolicyRejected) diff --git a/tests/engine/execution/test_phase6_runtime.py b/tests/engine/execution/test_phase6_runtime.py new file mode 100644 index 00000000..a9f5b6f8 --- /dev/null +++ b/tests/engine/execution/test_phase6_runtime.py @@ -0,0 +1,542 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib.util +import json +import pickle +from dataclasses import replace + +import pytest + +from anonymizer.engine.execution import role_policy as role_policy_module +from anonymizer.engine.execution.accounting_ledger import _AccountingLedger +from anonymizer.engine.execution.accounting_outcomes import _InvocationFailed, _InvocationInconsistent +from anonymizer.engine.execution.accounting_plan import _AccountingLimits, _DatumTaskSubject, _TaskKey +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextBackendCapability, + _ContextExecutionContract, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.graph import ( + _AtomicGroup, + _CoherenceScope, + _ContextScope, + _DatumId, + _DatumPurpose, + _ProtectionGraph, + _TextDatum, +) +from anonymizer.engine.execution.mention_admission import ( + _MentionId, + _MentionLimits, + _ValidationDecision, + _ValidationDecisionKind, +) +from anonymizer.engine.execution.mention_resolution import ( + _EvidenceVersion, + _SameSubjectEvidence, +) +from anonymizer.engine.execution.phase6_plan import ( + _compile_phase6_plan, + _is_admitted_phase6_plan, + _Phase6Plan, + _Phase6ProfileVersion, + _Phase6Rejected, +) +from anonymizer.engine.execution.phase6_runtime import ( + _CandidateProposal, + _Phase6AugmentationWork, + _Phase6CandidateWork, + _Phase6Execution, + _Phase6ResolverWork, + _Phase6Runtime, + _Phase6RuntimeAdmissionError, + _Phase6ValidationWork, +) + + +def _datum_id(task: _TaskKey) -> _DatumId: + assert isinstance(task.subject, _DatumTaskSubject) + return task.subject.datum_id + + +def test_phase6_runtime_test_infrastructure() -> None: + assert importlib.util.find_spec("anonymizer.engine.execution.phase6_plan") is not None + + +def test_phase6_plan_freezes_stages_target_only_resolver_scopes_and_predecessors() -> None: + contract, capability = _contract_and_capability() + + result = _compile_phase6_plan( + _context_graph(), + accounting_limits=_ACCOUNTING_LIMITS, + context_contract=contract, + capability=capability, + mention_limits=_MENTION_LIMITS, + ) + + assert isinstance(result, _Phase6Plan) + assert _is_admitted_phase6_plan(result) + assert tuple(stage.value for stage in result.accounting.stages) == ( + "detect", + "augment", + "validate", + "finalize", + "resolve", + "classify", + "transform", + "verify", + ) + target_by_token = {target.token: target.datum_id.value for target in result.targets} + assert tuple( + ( + target_by_token[scope.owner], + tuple(target_by_token[token] for token in scope.eligible_targets), + ) + for scope in result.resolver_scopes + ) == ( + ("target-a", ("target-a", "target-b")), + ("target-b", ("target-b", "target-a")), + ) + assert all( + "context-c" not in members + for _owner, members in ( + ( + target_by_token[scope.owner], + tuple(target_by_token[token] for token in scope.eligible_targets), + ) + for scope in result.resolver_scopes + ) + ) + + tasks = {(task.stage.value, _datum_id(task).value): task for task in result.accounting.tasks} + ledger: _AccountingLedger[str] = _AccountingLedger(result.accounting) + ledger.open() + for stage in ("detect", "augment", "validate"): + ready = ledger.ready_tasks() + assert {task.stage.value for task in ready} == {stage} + for task in ready: + ledger.accept_success(ledger.dispatch(task), stage) + finalize = ledger.ready_tasks() + by_datum = {_datum_id(task).value: task for task in finalize} + ledger.accept_success(ledger.dispatch(by_datum["target-a"]), "finalized-a") + assert ledger.ready_tasks() == (by_datum["target-b"],) + ledger.accept_success(ledger.dispatch(by_datum["target-b"]), "finalized-b") + assert set(ledger.ready_tasks()) == { + tasks[("resolve", "target-a")], + tasks[("resolve", "target-b")], + } + + with_pickle_error = result + try: + pickle.dumps(with_pickle_error) + except TypeError as error: + assert "not serializable" in str(error) + else: + raise AssertionError("Phase 6 plans must remain private") + + +def test_phase6_plan_rejects_a_role_policy_manifest_digest_mismatch(monkeypatch: pytest.MonkeyPatch) -> None: + class _ManifestResource: + def joinpath(self, _name: str) -> _ManifestResource: + return self + + def read_text(self, *, encoding: str) -> str: + assert encoding == "utf-8" + return json.dumps( + { + "digest": "0" * 64, + "mappings": [], + "version": "phase6-role-result/v1", + } + ) + + monkeypatch.setattr(role_policy_module, "files", lambda _package: _ManifestResource(), raising=False) + contract, capability = _contract_and_capability() + + result = _compile_phase6_plan( + _context_graph(), + accounting_limits=_ACCOUNTING_LIMITS, + context_contract=contract, + capability=capability, + mention_limits=_MENTION_LIMITS, + ) + + assert isinstance(result, _Phase6Rejected) + + +def test_default_phase6_profile_freezes_redact_policy_transform_and_verification() -> None: + plan = _plan(_independent_graph()) + + class _Backend(_NoMentionBackend): + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + self.effect_count += 1 + return (_CandidateProposal(0, len(work.target.text), work.target.text, "first_name"),) + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + self.effect_count += 1 + return tuple( + _ValidationDecision(candidate.token, _ValidationDecisionKind.KEEP) for candidate in work.candidates + ) + + result = _Phase6Runtime(_Backend()).run(plan) + + assert plan.profile_version is _Phase6ProfileVersion.REDACT_V1 + assert plan.role_policy.version.value == "phase6-role-result/v1" + assert plan.role_policy.mappings == () + assert plan.role_policy.digest == "e11a29db1af26c9572e1b4dec9e0a91e80966c6de1d813a378b64210a3bdfc40" + assert tuple(stage.value for stage in plan.accounting.stages) == ( + "detect", + "augment", + "validate", + "finalize", + "resolve", + "classify", + "transform", + "verify", + ) + assert tuple((datum.datum_id.value, datum.output) for datum in result.released) == ( + ("target-a", "[REDACTED]"), + ("target-b", "[REDACTED]"), + ) + assert { + (_datum_id(outcome.task).value, type(outcome).__name__) + for outcome in result.accounting.tasks + if outcome.task.stage.value == "verify" + } == {("target-a", "_TaskSucceeded"), ("target-b", "_TaskSucceeded")} + + +def test_phase6_runtime_accounts_effects_and_releases_exact_local_redact_outputs() -> None: + plan = _plan(_context_graph()) + + class _Backend: + def __init__(self) -> None: + self.calls: list[tuple[str, str]] = [] + + def context_capability(self) -> _ContextBackendCapability: + return _contract_and_capability()[1] + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + self.calls.append(("detect", work.target.datum_id.value)) + return (_CandidateProposal(0, len(work.target.text), work.target.text, "name"),) + + def augment(self, work: _Phase6AugmentationWork) -> tuple[_CandidateProposal, ...]: + self.calls.append(("augment", work.target.datum_id.value)) + return () + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + self.calls.append(("validate", work.target.datum_id.value)) + return tuple( + _ValidationDecision(candidate.token, _ValidationDecisionKind.KEEP) for candidate in work.candidates + ) + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SameSubjectEvidence, ...]: + self.calls.append(("resolve", work.owner.datum_id.value)) + if work.owner.datum_id.value != "target-a": + return () + owned = tuple( + mention for mention in work.eligible_mentions if mention.target_datum_id == work.owner.datum_id + ) + other = tuple( + mention for mention in work.eligible_mentions if mention.target_datum_id != work.owner.datum_id + ) + return ( + _SameSubjectEvidence( + work.owner.token, + owned[0].id, + other[0].id, + _EvidenceVersion.V1, + ), + ) + + def close_phase6(self) -> bool: + return True + + backend = _Backend() + result = _Phase6Runtime(backend).run(plan) + + assert isinstance(result, _Phase6Execution) + assert tuple((datum.datum_id.value, datum.output) for datum in result.released) == ( + ("target-a", "[REDACTED]"), + ("target-b", "[REDACTED]"), + ) + assert len(backend.calls) == 8 + assert {type(task).__name__ for task in result.accounting.tasks} == {"_TaskSucceeded"} + assert "Alice" not in repr(result) + assert "[REDACTED]" not in repr(result) + try: + pickle.dumps(result) + except TypeError as error: + assert "not serializable" in str(error) + else: + raise AssertionError("Phase 6 executions must remain private") + + +def test_phase6_detector_work_is_target_only_and_augmentation_receives_context() -> None: + plan = _plan(_context_graph()) + context_by_target: dict[str, tuple[str, ...]] = {} + + class _Backend(_NoMentionBackend): + def detect(self, work: object) -> tuple[_CandidateProposal, ...]: + assert not hasattr(work, "context") + target = getattr(work, "target") + assert isinstance(target.text, str) + return () + + def augment(self, work: object) -> tuple[_CandidateProposal, ...]: + target = getattr(work, "target") + context = getattr(work, "context") + context_by_target[target.datum_id.value] = tuple(datum.text for datum in context) + return () + + result = _Phase6Runtime(_Backend()).run(plan) + + assert tuple(datum.datum_id.value for datum in result.released) == ("target-a", "target-b") + assert context_by_target == { + "target-a": ("private context", "A. Example"), + "target-b": ("Alice",), + } + + +def test_phase6_runtime_localizes_known_target_failure_without_raw_fallback() -> None: + plan = _plan(_independent_graph()) + + class _Backend: + def context_capability(self) -> _ContextBackendCapability: + return _contract_and_capability()[1] + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + if work.target.datum_id.value == "target-a": + raise RuntimeError("known detector failure") + return () + + def augment(self, work: _Phase6AugmentationWork) -> tuple[_CandidateProposal, ...]: + return () + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + return () + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SameSubjectEvidence, ...]: + return () + + def close_phase6(self) -> bool: + return True + + result = _Phase6Runtime(_Backend()).run(plan) + + assert tuple((datum.datum_id.value, datum.output) for datum in result.released) == (("target-b", "public"),) + assert all(datum.datum_id.value != "target-a" for datum in result.released) + + +def test_phase6_attributable_resolver_fault_does_not_withhold_context_peer() -> None: + plan = _plan(_context_graph()) + + class _Backend(_NoMentionBackend): + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + return (_CandidateProposal(0, len(work.target.text), work.target.text, "name"),) + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + return tuple( + _ValidationDecision(candidate.token, _ValidationDecisionKind.KEEP) for candidate in work.candidates + ) + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SameSubjectEvidence, ...]: + if work.owner.datum_id.value != "target-a": + return () + owned = next( + mention for mention in work.eligible_mentions if mention.target_datum_id == work.owner.datum_id + ) + return ( + _SameSubjectEvidence( + work.owner.token, + owned.id, + owned.id, + _EvidenceVersion.V1, + ), + ) + + result = _Phase6Runtime(_Backend()).run(plan) + + assert tuple((datum.datum_id.value, datum.output) for datum in result.released) == (("target-b", "[REDACTED]"),) + task_outcomes = { + (outcome.task.stage.value, _datum_id(outcome.task).value): type(outcome).__name__ + for outcome in result.accounting.tasks + } + assert task_outcomes[("resolve", "target-a")] == "_TaskFailed" + assert task_outcomes[("resolve", "target-b")] == "_TaskSucceeded" + assert task_outcomes[("verify", "target-b")] == "_TaskSucceeded" + + +def test_phase6_runtime_rechecks_capability_before_opening_effects() -> None: + plan = _plan(_independent_graph()) + + class _Backend(_NoMentionBackend): + def context_capability(self) -> _ContextBackendCapability: + capability = _contract_and_capability()[1] + return replace(capability, retention=_RetentionPosture.ENABLED) + + backend = _Backend() + + try: + _Phase6Runtime(backend).run(plan) + except _Phase6RuntimeAdmissionError: + pass + else: + raise AssertionError("runtime capability drift must reject before effects") + assert backend.effect_count == 0 + assert backend.close_count == 0 + + +def test_phase6_runtime_cleanup_failure_embargoes_verified_outputs() -> None: + plan = _plan(_independent_graph()) + backend = _NoMentionBackend(clean=False) + + result = _Phase6Runtime(backend).run(plan) + + assert isinstance(result.accounting.invocation, _InvocationFailed) + assert result.released == () + assert backend.close_count == 1 + + +def test_phase6_runtime_foreign_resolver_endpoint_causes_global_embargo() -> None: + plan = _plan(_context_graph()) + + class _Backend(_NoMentionBackend): + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + self.effect_count += 1 + return (_CandidateProposal(0, len(work.target.text), work.target.text, "name"),) + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + self.effect_count += 1 + return tuple( + _ValidationDecision(candidate.token, _ValidationDecisionKind.KEEP) for candidate in work.candidates + ) + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SameSubjectEvidence, ...]: + self.effect_count += 1 + return ( + _SameSubjectEvidence( + work.owner.token, + _MentionId(), + work.eligible_mentions[0].id, + _EvidenceVersion.V1, + ), + ) + + result = _Phase6Runtime(_Backend()).run(plan) + + assert isinstance(result.accounting.invocation, _InvocationInconsistent) + assert result.released == () + + +def _plan(graph: _ProtectionGraph) -> _Phase6Plan: + contract, capability = _contract_and_capability() + result = _compile_phase6_plan( + graph, + accounting_limits=_ACCOUNTING_LIMITS, + context_contract=contract, + capability=capability, + mention_limits=_MENTION_LIMITS, + ) + assert isinstance(result, _Phase6Plan) + return result + + +def _contract_and_capability() -> tuple[_ContextExecutionContract, _ContextBackendCapability]: + limits = _ContextLimits(4, 128, 8, 512) + contract = _ContextExecutionContract( + _ContextProfile.TARGET_CONTEXT_V1, + _ContextSchemaVersion.V1, + limits, + True, + _ContextOrdering.DECLARED, + (_BackendArtifactClass.CONTEXT_REQUEST,), + ) + capability = _ContextBackendCapability( + contract.profile, + contract.schema_version, + limits, + True, + contract.ordering, + contract.required_artifacts, + _RetentionPosture.DISABLED, + ) + return contract, capability + + +def _context_graph() -> _ProtectionGraph: + target_a = _TextDatum(_DatumId("target-a"), "Alice", _DatumPurpose.TARGET) + target_b = _TextDatum(_DatumId("target-b"), "A. Example", _DatumPurpose.TARGET) + context = _TextDatum(_DatumId("context-c"), "private context", _DatumPurpose.CONTEXT_ONLY) + return _ProtectionGraph( + datums=(target_a, target_b, context), + links=(), + context_scopes=( + _ContextScope(target_a.id, (context.id, target_b.id)), + _ContextScope(target_b.id, (target_a.id,)), + ), + coherence_scopes=(_CoherenceScope((target_a.id,)), _CoherenceScope((target_b.id,))), + atomic_groups=(_AtomicGroup((target_a.id,)), _AtomicGroup((target_b.id,))), + ) + + +def _independent_graph() -> _ProtectionGraph: + target_a = _TextDatum(_DatumId("target-a"), "private", _DatumPurpose.TARGET) + target_b = _TextDatum(_DatumId("target-b"), "public", _DatumPurpose.TARGET) + return _ProtectionGraph( + datums=(target_a, target_b), + links=(), + context_scopes=(_ContextScope(target_a.id), _ContextScope(target_b.id)), + coherence_scopes=(_CoherenceScope((target_a.id,)), _CoherenceScope((target_b.id,))), + atomic_groups=(_AtomicGroup((target_a.id,)), _AtomicGroup((target_b.id,))), + ) + + +class _NoMentionBackend: + def __init__(self, *, clean: bool = True) -> None: + self.clean = clean + self.effect_count = 0 + self.close_count = 0 + + def context_capability(self) -> _ContextBackendCapability: + return _contract_and_capability()[1] + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + del work + self.effect_count += 1 + return () + + def augment(self, work: _Phase6AugmentationWork) -> tuple[_CandidateProposal, ...]: + del work + self.effect_count += 1 + return () + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + del work + self.effect_count += 1 + return () + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SameSubjectEvidence, ...]: + del work + self.effect_count += 1 + return () + + def close_phase6(self) -> bool: + self.close_count += 1 + return self.clean + + +_ACCOUNTING_LIMITS = _AccountingLimits( + max_datums=8, + max_datum_bytes=128, + max_graph_bytes=512, + max_stages=8, +) +_MENTION_LIMITS = _MentionLimits(8, 8, 64, 128) diff --git a/tests/engine/execution/test_phase6_substitute_handoff.py b/tests/engine/execution/test_phase6_substitute_handoff.py new file mode 100644 index 00000000..1f812296 --- /dev/null +++ b/tests/engine/execution/test_phase6_substitute_handoff.py @@ -0,0 +1,437 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import json +import pickle +from dataclasses import replace +from pathlib import Path +from typing import cast + +import pytest + +from anonymizer.engine.execution import phase6_plan as phase6_plan_module +from anonymizer.engine.execution import phase6_runtime as phase6_runtime_module +from anonymizer.engine.execution import role_policy as role_policy_module +from anonymizer.engine.execution.accounting_outcomes import _AccountingResult, _TaskFailed +from anonymizer.engine.execution.accounting_plan import _AccountingLimits +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextBackendCapability, + _ContextExecutionContract, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.graph import ( + _AtomicGroup, + _CoherenceScope, + _ContextScope, + _DatumId, + _DatumPurpose, + _ProtectionGraph, + _TextDatum, +) +from anonymizer.engine.execution.mention_admission import ( + _AnchoredMention, + _DetectedGraph, + _MentionId, + _MentionLimits, + _MentionProvenance, + _MentionTarget, + _MentionTargetToken, + _ValidationDecision, + _ValidationDecisionKind, +) +from anonymizer.engine.execution.mention_resolution import ( + _ClusteredGraph, + _ClusterId, + _EntityCluster, + _EvidenceVersion, + _SameSubjectEvidence, +) +from anonymizer.engine.execution.phase6_plan import _Phase6Plan, _Phase6ProfileVersion, _Phase6Rejected +from anonymizer.engine.execution.phase6_runtime import ( + _CandidateProposal, + _Phase6AugmentationWork, + _Phase6CandidateWork, + _Phase6Execution, + _Phase6ResolverWork, + _Phase6Runtime, + _Phase6ValidationWork, +) +from anonymizer.engine.execution.role_policy import ( + _ClassifiedRole, + _ReplacementRole, + _ResolvedGraph, + _RolePolicy, + _RolePolicyRejected, + _RolePolicyVersion, +) + +_SUBSTITUTE_DIGEST = "c27580bd2cc4051bdd11b63a91391f8995bdef1ed2052534623cdd3160318ef8" +_SUBSTITUTE_POLICY_VERSION = "phase6-substitute-role-policy/v1" +_POLICY_PATH = ( + Path(__file__).parents[3] / "src" / "anonymizer" / "engine" / "execution" / "phase6_substitute_role_policy.json" +) +_ACCOUNTING_LIMITS = _AccountingLimits(8, 128, 512, max_stages=8) +_MENTION_LIMITS = _MentionLimits(8, 8, 64, 128) + + +class _Resource: + def __init__(self, payloads: dict[str, str], name: str | None = None) -> None: + self._payloads = payloads + self._name = name + + def joinpath(self, name: str) -> _Resource: + return _Resource(self._payloads, name) + + def read_text(self, *, encoding: str) -> str: + assert encoding == "utf-8" + assert self._name is not None + return self._payloads[self._name] + + +class _Backend: + def __init__(self, label: str, *, join_targets: bool = False) -> None: + self.label = label + self.join_targets = join_targets + self.calls: list[str] = [] + self.planner_effect_count = 0 + + def context_capability(self) -> _ContextBackendCapability: + return _contract_and_capability()[1] + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + self.calls.append("detect") + return (_CandidateProposal(0, len(work.target.text), work.target.text, self.label),) + + def augment(self, work: _Phase6AugmentationWork) -> tuple[_CandidateProposal, ...]: + self.calls.append("augment") + return () + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + self.calls.append("validate") + return tuple(_ValidationDecision(item.token, _ValidationDecisionKind.KEEP) for item in work.candidates) + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SameSubjectEvidence, ...]: + self.calls.append("resolve") + if not self.join_targets or work.owner.datum_id.value != "target-a": + return () + owned = next(item for item in work.eligible_mentions if item.target_datum_id == work.owner.datum_id) + foreign = next(item for item in work.eligible_mentions if item.target_datum_id != work.owner.datum_id) + return (_SameSubjectEvidence(work.owner.token, owned.id, foreign.id, _EvidenceVersion.V1),) + + def plan(self, _work: object) -> None: + self.planner_effect_count += 1 + + def close_phase6(self) -> bool: + return True + + +def _contract_and_capability() -> tuple[_ContextExecutionContract, _ContextBackendCapability]: + limits = _ContextLimits(4, 128, 8, 512) + contract = _ContextExecutionContract( + _ContextProfile.TARGET_CONTEXT_V1, + _ContextSchemaVersion.V1, + limits, + True, + _ContextOrdering.DECLARED, + (_BackendArtifactClass.CONTEXT_REQUEST,), + ) + capability = _ContextBackendCapability( + contract.profile, + contract.schema_version, + limits, + True, + contract.ordering, + contract.required_artifacts, + _RetentionPosture.DISABLED, + ) + return contract, capability + + +def _graph(*, two_targets: bool = False) -> _ProtectionGraph: + target_a = _TextDatum(_DatumId("target-a"), "Alice", _DatumPurpose.TARGET) + targets = (target_a,) + if two_targets: + targets = (*targets, _TextDatum(_DatumId("target-b"), "Alicia", _DatumPurpose.TARGET)) + ids = tuple(target.id for target in targets) + return _ProtectionGraph( + datums=targets, + links=(), + context_scopes=tuple( + _ContextScope(target.id, tuple(candidate.id for candidate in targets if candidate.id != target.id)) + for target in targets + ), + coherence_scopes=tuple(_CoherenceScope((datum_id,)) for datum_id in ids), + atomic_groups=(_AtomicGroup(ids),), + ) + + +def _substitute_profile() -> _Phase6ProfileVersion: + profile = getattr(_Phase6ProfileVersion, "SUBSTITUTE_V1", None) + assert isinstance(profile, _Phase6ProfileVersion), "the private Phase 6 Substitute profile is missing" + return profile + + +def _compile_substitute(graph: _ProtectionGraph) -> _Phase6Plan: + contract, capability = _contract_and_capability() + result = phase6_plan_module._compile_phase6_plan( + graph, + accounting_limits=_ACCOUNTING_LIMITS, + context_contract=contract, + capability=capability, + mention_limits=_MENTION_LIMITS, + profile_version=_substitute_profile(), + ) + assert isinstance(result, _Phase6Plan) + return result + + +def _valid_execution(*, two_targets: bool = False) -> tuple[_Phase6Plan, _Backend, _Phase6Execution]: + plan = _compile_substitute(_graph(two_targets=two_targets)) + backend = _Backend("first_name") + return plan, backend, _Phase6Runtime(backend).run(plan) + + +def _foreign_resolved_graph(policy: _RolePolicy) -> _ResolvedGraph: + token = _MentionTargetToken() + target = _MentionTarget(token, _DatumId("foreign"), "Mallory") + mention = _AnchoredMention( + _MentionId(), + target.datum_id, + 0, + len(target.text), + target.text, + "first_name", + _MentionProvenance.SPAN_DETECTOR, + ) + clustered = _ClusteredGraph( + _DetectedGraph((target,), (mention,)), + (_EntityCluster(_ClusterId(), (mention.id,), ()),), + (), + ) + result = role_policy_module._classify_roles(clustered, policy) + assert isinstance(result, _ResolvedGraph) + return result + + +def test_substitute_policy_loader_requires_the_exact_p0_result_version_policy_version_and_digest() -> None: + loader = getattr(role_policy_module, "_load_substitute_role_policy", None) + assert callable(loader), "the exact private Substitute role-policy loader is missing" + + result = loader() + + assert isinstance(result, _RolePolicy) + assert result.result_version is _RolePolicyVersion.V1 + assert result.policy_version == _SUBSTITUTE_POLICY_VERSION + assert result.digest == _SUBSTITUTE_DIGEST + assert tuple((label, role.value) for label, role in result.mappings) == ( + ("email", "email_address"), + ("fax_number", "fax_number"), + ("first_name", "person_given_name"), + ("last_name", "person_family_name"), + ("phone_number", "voice_phone_number"), + ("user_name", "user_name"), + ) + + +def test_substitute_policy_loader_rejects_a_wrong_policy_version(monkeypatch: pytest.MonkeyPatch) -> None: + loader = getattr(role_policy_module, "_load_substitute_role_policy", None) + assert callable(loader), "the exact private Substitute role-policy loader is missing" + payload = json.loads(_POLICY_PATH.read_text(encoding="utf-8")) + payload["version"] = "phase6-substitute-role-policy/v2" + monkeypatch.setattr( + role_policy_module, + "files", + lambda _package: _Resource({"phase6_substitute_role_policy.json": json.dumps(payload)}), + ) + + assert isinstance(loader(), _RolePolicyRejected) + + +def test_substitute_plan_rejects_a_wrong_policy_digest(monkeypatch: pytest.MonkeyPatch) -> None: + loader = getattr(role_policy_module, "_load_substitute_role_policy", None) + assert callable(loader), "the exact private Substitute role-policy loader is missing" + policy = loader() + assert isinstance(policy, _RolePolicy) + monkeypatch.setattr( + phase6_plan_module, + "_load_substitute_role_policy", + lambda: replace(policy, digest="0" * 64), + raising=False, + ) + contract, capability = _contract_and_capability() + + result = phase6_plan_module._compile_phase6_plan( + _graph(), + accounting_limits=_ACCOUNTING_LIMITS, + context_contract=contract, + capability=capability, + mention_limits=_MENTION_LIMITS, + profile_version=_substitute_profile(), + ) + + assert isinstance(result, _Phase6Rejected) + + +def test_future_substitute_policy_releases_exact_classified_roles_with_zero_planner_effects() -> None: + plan, backend, execution = _valid_execution() + + assert plan.profile_version is _substitute_profile() + assert tuple(stage.value for stage in plan.accounting.stages) == ( + "detect", + "augment", + "validate", + "finalize", + "resolve", + "classify", + ) + assert plan.role_policy.result_version is _RolePolicyVersion.V1 + assert plan.role_policy.policy_version == _SUBSTITUTE_POLICY_VERSION + assert plan.role_policy.digest == _SUBSTITUTE_DIGEST + assert execution.released == () + assert len(execution.handoffs) == 1 + handoff = execution.handoffs[0] + assert handoff.result_version == _RolePolicyVersion.V1.value + assert handoff.policy_version == _SUBSTITUTE_POLICY_VERSION + assert handoff.policy_digest == _SUBSTITUTE_DIGEST + role_results = tuple(item.role_result for item in handoff.resolved.mentions) + assert all(isinstance(item, _ClassifiedRole) for item in role_results) + assert tuple(item.role.value for item in role_results if isinstance(item, _ClassifiedRole)) == ( + "person_given_name", + ) + assert tuple(item.mention for item in handoff.resolved.mentions) == handoff.resolved.clustered.detected.mentions + assert tuple(item.cluster_id for item in handoff.resolved.mentions) == tuple( + cluster.id for cluster in handoff.resolved.clustered.clusters + ) + assert tuple(datum_id.value for datum_id in handoff.terminal_evidence.datum_ids) == ("target-a",) + assert tuple(task.stage.value for task in handoff.terminal_evidence.tasks) == ( + "detect", + "augment", + "validate", + "finalize", + "resolve", + "classify", + ) + assert backend.planner_effect_count == 0 + assert backend.calls == ["detect", "augment", "validate", "resolve"] + assert phase6_runtime_module._is_admitted_substitute_handoff(handoff, plan) + with pytest.raises((AttributeError, TypeError)): + setattr(handoff, "policy_digest", "changed") + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(handoff) + + +def test_substitute_handoff_rejects_an_unsupported_role() -> None: + plan = _compile_substitute(_graph()) + + execution = _Phase6Runtime(_Backend("custom_secret")).run(plan) + + assert execution.handoffs == () + assert any( + isinstance(outcome, _TaskFailed) and outcome.task.stage.value == "classify" + for outcome in execution.accounting.tasks + ) + + +@pytest.mark.parametrize( + "mutation", + [ + pytest.param("wrong-result-version", id="wrong-result-version"), + pytest.param("wrong-policy-version", id="wrong-policy-version"), + pytest.param("wrong-policy-digest", id="wrong-policy-digest"), + pytest.param("mismatched-role", id="mismatched-role"), + pytest.param("missing-role-result", id="missing-role-result"), + pytest.param("duplicate-role-result", id="duplicate-role-result"), + pytest.param("stale-graph", id="stale-graph"), + pytest.param("foreign-graph", id="foreign-graph"), + pytest.param("unresolved-graph", id="unresolved-graph"), + ], +) +def test_substitute_runtime_rejects_inexact_classified_graphs( + monkeypatch: pytest.MonkeyPatch, + mutation: str, +) -> None: + plan = _compile_substitute(_graph()) + classify = role_policy_module._classify_roles + + def _mutated_classification(clustered: _ClusteredGraph, policy: _RolePolicy) -> object: + resolved = classify(clustered, policy) + assert isinstance(resolved, _ResolvedGraph) + if mutation == "wrong-result-version": + return replace(resolved, policy_version=cast(_RolePolicyVersion, "phase6-role-result/v2")) + if mutation == "wrong-policy-version": + return replace(resolved, source_policy_version="phase6-substitute-role-policy/v2") + if mutation == "wrong-policy-digest": + return replace(resolved, policy_digest="0" * 64) + if mutation == "mismatched-role": + role_result = resolved.mentions[0].role_result + assert isinstance(role_result, _ClassifiedRole) + wrong = replace(role_result, role=_ReplacementRole("email_address")) + return replace(resolved, mentions=(replace(resolved.mentions[0], role_result=wrong),)) + if mutation == "missing-role-result": + return replace(resolved, mentions=()) + if mutation == "duplicate-role-result": + return replace(resolved, mentions=(resolved.mentions[0], resolved.mentions[0])) + if mutation == "stale-graph": + stale = replace(resolved.clustered, detected=replace(resolved.clustered.detected, mentions=())) + return replace(resolved, clustered=stale) + if mutation == "foreign-graph": + return _foreign_resolved_graph(policy) + return role_policy_module._RolePolicyRejected(role_policy_module._RolePolicyRejectionCode.UNSUPPORTED_ROLE) + + monkeypatch.setattr(phase6_runtime_module, "_classify_roles", _mutated_classification) + + execution = _Phase6Runtime(_Backend("first_name")).run(plan) + + assert execution.handoffs == () + assert any( + isinstance(outcome, _TaskFailed) and outcome.task.stage.value == "classify" + for outcome in execution.accounting.tasks + ) + + +@pytest.mark.parametrize( + "mutation", + ["missing-task", "duplicate-task", "missing-datum", "duplicate-datum", "missing-group", "duplicate-group"], +) +def test_substitute_handoff_rejects_incomplete_or_duplicate_terminal_evidence(mutation: str) -> None: + plan, _backend, execution = _valid_execution() + assert len(execution.handoffs) == 1 + builder = getattr(phase6_runtime_module, "_build_substitute_handoffs", None) + assert callable(builder), "the terminal-evidence handoff builder is missing" + accounting = cast(_AccountingResult[object], execution.accounting) + corrupted = accounting + if mutation == "missing-task": + corrupted = replace(accounting, tasks=accounting.tasks[1:]) + elif mutation == "duplicate-task": + corrupted = replace(accounting, tasks=(*accounting.tasks, accounting.tasks[0])) + elif mutation == "missing-datum": + corrupted = replace(accounting, datums=()) + elif mutation == "duplicate-datum": + corrupted = replace(accounting, datums=(*accounting.datums, *accounting.datums)) + elif mutation == "missing-group": + corrupted = replace(accounting, groups=()) + else: + corrupted = replace(accounting, groups=(*accounting.groups, *accounting.groups)) + + result = builder(plan, corrupted) + + assert type(result).__name__ == "_Phase6HandoffRejected" + + +def test_substitute_handoff_rejects_cross_scope_cluster_evidence() -> None: + plan = _compile_substitute(_graph(two_targets=True)) + + execution = _Phase6Runtime(_Backend("first_name", join_targets=True)).run(plan) + + assert execution.handoffs == () + assert any( + isinstance(outcome, _TaskFailed) and outcome.task.stage.value == "classify" + for outcome in execution.accounting.tasks + ) diff --git a/tests/engine/execution/test_phase7_admission.py b/tests/engine/execution/test_phase7_admission.py new file mode 100644 index 00000000..031d2b57 --- /dev/null +++ b/tests/engine/execution/test_phase7_admission.py @@ -0,0 +1,959 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib +import importlib.util +import pickle +from collections.abc import Callable +from dataclasses import FrozenInstanceError, dataclass, replace +from types import ModuleType +from typing import Any + +import pytest + +from anonymizer.engine.execution.accounting_plan import ( + _AccountingLimits, + _DatumTaskSubject, + _TaskKey, + _TaskPredecessor, +) +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextBackendCapability, + _ContextExecutionContract, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.graph import ( + _AtomicGroup, + _CoherenceScope, + _ContextScope, + _DatumId, + _DatumPurpose, + _ProtectionGraph, + _TextDatum, +) +from anonymizer.engine.execution.mention_admission import ( + _MentionId, + _MentionLimits, + _ValidationDecision, + _ValidationDecisionKind, +) +from anonymizer.engine.execution.mention_resolution import ( + _EvidenceVersion, + _SameSubjectEvidence, +) +from anonymizer.engine.execution.phase6_plan import ( + _compile_phase6_plan, + _Phase6Plan, + _Phase6ProfileVersion, +) +from anonymizer.engine.execution.phase6_runtime import ( + _CandidateProposal, + _Phase6AugmentationWork, + _Phase6CandidateWork, + _Phase6Execution, + _Phase6ResolverWork, + _Phase6Runtime, + _Phase6ValidationWork, +) +from anonymizer.engine.execution.phase7_contract import _load_phase7_contract +from anonymizer.engine.execution.role_policy import _ClassifiedRole + +_ACCOUNTING_LIMITS = _AccountingLimits(16, 16_384, 65_536, max_stages=8) +_MENTION_LIMITS = _MentionLimits(16, 16, 128, 512) + + +@dataclass(frozen=True) +class _Proposal: + source: str + label: str + cluster: str + + +class _Backend: + def __init__(self, proposals: dict[str, tuple[_Proposal, ...]]) -> None: + self._proposals = proposals + self.calls: list[str] = [] + self.planner_effect_count = 0 + + def context_capability(self) -> _ContextBackendCapability: + return _contract_and_capability()[1] + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + self.calls.append("detect") + proposals = self._proposals.get(work.target.datum_id.value, ()) + return tuple( + _CandidateProposal( + work.target.text.index(proposal.source), + work.target.text.index(proposal.source) + len(proposal.source), + proposal.source, + proposal.label, + ) + for proposal in proposals + ) + + def augment(self, work: _Phase6AugmentationWork) -> tuple[_CandidateProposal, ...]: + del work + self.calls.append("augment") + return () + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + self.calls.append("validate") + return tuple(_ValidationDecision(item.token, _ValidationDecisionKind.KEEP) for item in work.candidates) + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SameSubjectEvidence, ...]: + self.calls.append("resolve") + proposal_by_source = { + proposal.source: proposal for proposals in self._proposals.values() for proposal in proposals + } + grouped: dict[str, list[_MentionId]] = {} + for mention in work.eligible_mentions: + proposal = proposal_by_source[mention.source_slice] + grouped.setdefault(proposal.cluster, []).append(mention.id) + evidence: list[_SameSubjectEvidence] = [] + for mention_ids in grouped.values(): + for left, right in zip(mention_ids, mention_ids[1:], strict=False): + evidence.append(_SameSubjectEvidence(work.owner.token, left, right, _EvidenceVersion.V1)) + return tuple(evidence) + + def close_phase6(self) -> bool: + return True + + def plan(self, _work: object) -> None: + self.planner_effect_count += 1 + + +def _contract_and_capability() -> tuple[_ContextExecutionContract, _ContextBackendCapability]: + limits = _ContextLimits(16, 16_384, 16, 65_536) + contract = _ContextExecutionContract( + _ContextProfile.TARGET_CONTEXT_V1, + _ContextSchemaVersion.V1, + limits, + True, + _ContextOrdering.DECLARED, + (_BackendArtifactClass.CONTEXT_REQUEST,), + ) + capability = _ContextBackendCapability( + contract.profile, + contract.schema_version, + limits, + True, + contract.ordering, + contract.required_artifacts, + _RetentionPosture.DISABLED, + ) + return contract, capability + + +def _graph( + texts: tuple[str, ...], + scopes: tuple[tuple[str, ...], ...], + *, + connect_context: bool = False, + context_texts: tuple[str, ...] = (), +) -> _ProtectionGraph: + targets = tuple( + _TextDatum(_DatumId(f"target-{index}"), text, _DatumPurpose.TARGET) for index, text in enumerate(texts) + ) + context_datums = tuple( + _TextDatum(_DatumId(f"context-{index}"), text, _DatumPurpose.CONTEXT_ONLY) + for index, text in enumerate(context_texts) + ) + datums = (*targets, *context_datums) + by_value = {datum.id.value: datum.id for datum in targets} + return _ProtectionGraph( + datums=datums, + links=(), + context_scopes=tuple( + _ContextScope( + datum.id, + (tuple(candidate.id for candidate in targets if candidate.id != datum.id) if connect_context else ()) + + tuple(context.id for context in context_datums), + ) + for datum in targets + ), + coherence_scopes=tuple(_CoherenceScope(tuple(by_value[value] for value in members)) for members in scopes), + atomic_groups=tuple(_AtomicGroup((datum.id,)) for datum in targets), + ) + + +def _qualified_phase6( + texts: tuple[str, ...], + scopes: tuple[tuple[str, ...], ...], + proposals: dict[str, tuple[_Proposal, ...]], + *, + connect_context: bool = False, + context_texts: tuple[str, ...] = (), +) -> tuple[_Phase6Plan, _Backend, _Phase6Execution]: + contract, capability = _contract_and_capability() + plan = _compile_phase6_plan( + _graph(texts, scopes, connect_context=connect_context, context_texts=context_texts), + accounting_limits=_ACCOUNTING_LIMITS, + context_contract=contract, + capability=capability, + mention_limits=_MENTION_LIMITS, + profile_version=_Phase6ProfileVersion.SUBSTITUTE_V1, + ) + assert isinstance(plan, _Phase6Plan) + backend = _Backend(proposals) + execution = _Phase6Runtime(backend).run(plan) + assert len(execution.handoffs) == len(plan.components) + return plan, backend, execution + + +def _phase7_module() -> ModuleType: + module_name = "anonymizer.engine.execution.phase7_admission" + assert importlib.util.find_spec(module_name) is not None, "the private Phase 7 compiler module is missing" + return importlib.import_module(module_name) + + +def _compile_phase7( + plan: _Phase6Plan, + execution: _Phase6Execution, + scopes: tuple[_CoherenceScope, ...], + relations: tuple[object, ...] = (), +) -> object: + module = _phase7_module() + compiler = getattr(module, "_compile_phase7_plan", None) + declarations_type = getattr(module, "_Phase7Declarations", None) + assert callable(compiler), "the private Phase 7 compiler is missing" + assert callable(declarations_type), "the private Phase 7 declaration grammar is missing" + declarations = declarations_type(scopes, relations) + return compiler(plan, execution.handoffs, declarations, _load_phase7_contract()) + + +def _compile_raw( + plan: _Phase6Plan, + handoffs: tuple[object, ...], + scopes: tuple[_CoherenceScope, ...], + relations: tuple[object, ...] = (), +) -> object: + module = _phase7_module() + declarations = module._Phase7Declarations(scopes, relations) + return module._compile_phase7_plan(plan, handoffs, declarations, _load_phase7_contract()) + + +def _rejection_code(result: object) -> str: + code = getattr(result, "code", None) + assert code is not None, "Phase 7 malformed input did not return a typed rejection" + value = getattr(code, "value", None) + assert isinstance(value, str) + return value + + +def _ids(plan: _Phase6Plan) -> tuple[_DatumId, ...]: + return tuple(datum.id for datum in plan.accounting.datums) + + +def _required_type(module: ModuleType, name: str) -> Callable[..., Any]: + value = getattr(module, name, None) + assert callable(value), f"the private Phase 7 {name} type is missing" + return value + + +@pytest.mark.parametrize( + ("case", "expected"), + [ + pytest.param("empty", "empty_scope", id="empty-scope"), + pytest.param("duplicate-scope", "duplicate_scope", id="duplicate-semantic-scope"), + pytest.param("duplicate-member", "duplicate_scope_member", id="duplicate-member"), + pytest.param("gap", "scope_coverage_gap", id="coverage-gap"), + pytest.param("unknown", "unknown_scope_datum", id="unknown-datum"), + pytest.param("overlap", "scope_overlap", id="partial-overlap"), + pytest.param("nesting", "unsupported_scope_nesting", id="scope-nesting"), + ], +) +def test_phase7_scope_admission_rejects_every_non_exact_partition(case: str, expected: str) -> None: + plan, _backend, execution = _qualified_phase6( + ("one", "two", "three", "four"), + (("target-0",), ("target-1",), ("target-2",), ("target-3",)), + {}, + ) + first, second, third, fourth = _ids(plan) + scopes_by_case = { + "empty": (_CoherenceScope(()), _CoherenceScope((first, second, third, fourth))), + "duplicate-scope": ( + _CoherenceScope((first, second, third, fourth)), + _CoherenceScope((fourth, third, second, first)), + ), + "duplicate-member": ( + _CoherenceScope((first, first, second)), + _CoherenceScope((third, fourth)), + ), + "gap": (_CoherenceScope((first, second, third)),), + "unknown": ( + _CoherenceScope((first, second, third)), + _CoherenceScope((_DatumId("foreign"),)), + ), + "overlap": ( + _CoherenceScope((first, second)), + _CoherenceScope((second, third, fourth)), + ), + "nesting": ( + _CoherenceScope((first, second, third, fourth)), + _CoherenceScope((first, second)), + ), + } + + result = _compile_phase7(plan, execution, scopes_by_case[case]) + + assert _rejection_code(result) == expected + + +@pytest.mark.parametrize( + ("scopes", "expected"), + [ + pytest.param( + lambda ids: (_CoherenceScope(()), _CoherenceScope(())), + "empty_scope", + id="empty-before-duplicate", + ), + pytest.param( + lambda ids: (_CoherenceScope((ids[0], ids[1], ids[1])),) * 2, + "duplicate_scope", + id="duplicate-scope-before-duplicate-member", + ), + pytest.param( + lambda ids: ( + _CoherenceScope((ids[0], ids[0], _DatumId("foreign"))), + _CoherenceScope((ids[1], ids[2], ids[3])), + ), + "duplicate_scope_member", + id="duplicate-member-before-unknown", + ), + pytest.param( + lambda ids: ( + _CoherenceScope((ids[0], ids[1], _DatumId("foreign"))), + _CoherenceScope((ids[2],)), + ), + "unknown_scope_datum", + id="unknown-before-gap", + ), + pytest.param( + lambda ids: (_CoherenceScope((ids[0], ids[1])), _CoherenceScope((ids[1], ids[2]))), + "scope_coverage_gap", + id="gap-before-overlap", + ), + ], +) +def test_phase7_scope_rejection_precedence_is_fixed_under_declaration_permutation( + scopes: Callable[[tuple[_DatumId, ...]], tuple[_CoherenceScope, ...]], + expected: str, +) -> None: + plan, _backend, execution = _qualified_phase6( + ("one", "two", "three", "four"), + (("target-0",), ("target-1",), ("target-2",), ("target-3",)), + {}, + ) + declared = scopes(_ids(plan)) + assert isinstance(declared, tuple) + + forward = _compile_phase7(plan, execution, declared) + reverse = _compile_phase7(plan, execution, tuple(reversed(declared))) + + assert _rejection_code(forward) == expected + assert _rejection_code(reverse) == expected + + +def test_phase7_compiler_requires_exact_phase6_scope_cluster_role_and_terminal_handoffs() -> None: + plan, _backend, execution = _qualified_phase6( + ("Alice", "555-0100"), + (("target-0",), ("target-1",)), + { + "target-0": (_Proposal("Alice", "first_name", "person"),), + "target-1": (_Proposal("555-0100", "phone_number", "phone"),), + }, + ) + module = _phase7_module() + declared = (_CoherenceScope(tuple(reversed(_ids(plan)))),) + + result = _compile_phase7(plan, execution, declared) + + assert isinstance(result, module._Phase7Plan) + assert len(result.manifests) == 1 + assert tuple(member.value for member in result.manifests[0].members) == ("target-0", "target-1") + assert tuple(task.stage.value for task in result.scope_tasks) == ("phase7-plan",) + assert tuple(task.stage.value for task in result.application_tasks) == ("phase7-apply", "phase7-apply") + assert tuple( + task.subject.datum_id for task in result.application_tasks if isinstance(task.subject, _DatumTaskSubject) + ) == ( + _DatumId("target-0"), + _DatumId("target-1"), + ) + expected_phase7_predecessors = { + _TaskPredecessor(result.scope_tasks[0], result.application_tasks[0]), + _TaskPredecessor(result.scope_tasks[0], result.application_tasks[1]), + _TaskPredecessor( + _TaskKey(plan.accounting.stages[-1], _DatumTaskSubject(_DatumId("target-0"))), + result.application_tasks[0], + ), + _TaskPredecessor( + _TaskKey(plan.accounting.stages[-1], _DatumTaskSubject(_DatumId("target-1"))), + result.application_tasks[1], + ), + } + phase7_tasks = {*result.scope_tasks, *result.application_tasks} + assert set(result.application_predecessors) == expected_phase7_predecessors + assert { + predecessor + for predecessor in result.accounting.task_predecessors + if predecessor.prerequisite in phase7_tasks or predecessor.dependent in phase7_tasks + } == { + _TaskPredecessor(result.scope_tasks[0], result.application_tasks[0]), + _TaskPredecessor(result.scope_tasks[0], result.application_tasks[1]), + } + assert module._is_admitted_phase7_plan(result) + + extra = _TaskPredecessor(result.application_tasks[0], result.application_tasks[1]) + expanded_accounting = result.accounting.with_task_predecessors((*result.accounting.task_predecessors, extra)) + assert not module._has_exact_application_predecessors(replace(result, accounting=expanded_accounting)) + + +@pytest.mark.parametrize( + "mutation", + [ + pytest.param("missing", id="missing-handoff"), + pytest.param("duplicate", id="duplicate-handoff"), + pytest.param("foreign", id="foreign-handoff"), + pytest.param("terminal", id="terminal-evidence-tampering"), + pytest.param("role", id="role-result-tampering"), + ], +) +def test_phase7_compiler_rejects_inexact_phase6_handoff_equality(mutation: str) -> None: + plan, _backend, execution = _qualified_phase6( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ) + handoff = execution.handoffs[0] + handoffs: tuple[object, ...] + if mutation == "missing": + handoffs = () + elif mutation == "duplicate": + handoffs = (handoff, handoff) + elif mutation == "foreign": + _foreign_plan, _foreign_backend, foreign = _qualified_phase6( + ("Mallory",), + (("target-0",),), + {"target-0": (_Proposal("Mallory", "first_name", "person"),)}, + ) + handoffs = foreign.handoffs + elif mutation == "terminal": + handoffs = (replace(handoff, terminal_evidence=replace(handoff.terminal_evidence, datum_ids=())),) + else: + resolved_mention = handoff.resolved.mentions[0] + role_result = resolved_mention.role_result + assert isinstance(role_result, _ClassifiedRole) + changed_role = replace(role_result, role=replace(role_result.role, value="email_address")) + changed_resolved = replace(handoff.resolved, mentions=(replace(resolved_mention, role_result=changed_role),)) + handoffs = (replace(handoff, resolved=changed_resolved),) + + result = _compile_raw(plan, handoffs, plan.coherence_scopes) + + assert _rejection_code(result) == "phase6_handoff_mismatch" + + +def _person_relation_fixture() -> tuple[_Phase6Plan, _Backend, _Phase6Execution, object]: + plan, backend, execution = _qualified_phase6( + ("Alice Adams alice@example.com",), + (("target-0",),), + { + "target-0": ( + _Proposal("Alice", "first_name", "person"), + _Proposal("Adams", "last_name", "person"), + _Proposal("alice@example.com", "email", "person"), + ) + }, + ) + clusters = execution.handoffs[0].resolved.clustered.clusters + assert len(clusters) == 1 + return plan, backend, execution, clusters[0].id + + +def test_phase7_pre_slot_selectors_resolve_exact_cluster_role_targets_before_slot_identity() -> None: + plan, _backend, execution, cluster = _person_relation_fixture() + module = _phase7_module() + selector = _required_type(module, "_ClusterRoleSelector") + relation_type = _required_type(module, "_RelationDeclaration") + relation = relation_type( + "email_from_name/v1", + ( + selector("cluster_role/v1", cluster, "person_given_name"), + selector("cluster_role/v1", cluster, "person_family_name"), + ), + selector("cluster_role/v1", cluster, "email_address"), + ) + + result = _compile_phase7(plan, execution, plan.coherence_scopes, (relation,)) + + assert isinstance(result, module._Phase7Plan) + assert len(result.manifests[0].relations) == 1 + assert len(result.manifests[0].relations[0].upstream) == 2 + assert result.manifests[0].relations[0].downstream in {slot.id for slot in result.manifests[0].slots} + + +def test_phase7_selector_rejects_a_missing_cluster_role_target() -> None: + plan, _backend, execution, cluster = _person_relation_fixture() + module = _phase7_module() + selector = _required_type(module, "_ClusterRoleSelector") + relation_type = _required_type(module, "_RelationDeclaration") + relation = relation_type( + "email_from_name/v1", + (selector("cluster_role/v1", cluster, "user_name"),), + selector("cluster_role/v1", cluster, "email_address"), + ) + + result = _compile_phase7(plan, execution, plan.coherence_scopes, (relation,)) + + assert _rejection_code(result) == "selector_missing" + + +def test_phase7_selector_rejects_ambiguous_pre_slot_targets() -> None: + _plan, _backend, execution, cluster = _person_relation_fixture() + module = _phase7_module() + scope = module._Phase7ScopeId() + mention_id = execution.handoffs[0].resolved.mentions[0].mention.id + pre_slot = _required_type(module, "_PreSlot") + selector_type = _required_type(module, "_ClusterRoleSelector") + candidate = pre_slot(scope, cluster, "person_given_name", "format", "mask", (mention_id,)) + selector = selector_type("cluster_role/v1", cluster, "person_given_name") + + result = module._resolve_selector(selector, (candidate, candidate)) + + assert _rejection_code(result) == "selector_ambiguous" + + +def test_phase7_selector_never_resolves_by_an_opaque_slot_id() -> None: + plan, _backend, execution, cluster = _person_relation_fixture() + module = _phase7_module() + initial = _compile_phase7(plan, execution, plan.coherence_scopes) + assert isinstance(initial, module._Phase7Plan) + assert initial.manifests[0].slots, "qualified Phase 6 roles did not materialize private slots" + opaque_slot_id = initial.manifests[0].slots[0].id + selector_type = _required_type(module, "_ClusterRoleSelector") + relation_type = _required_type(module, "_RelationDeclaration") + selector = selector_type( + "cluster_role/v1", + opaque_slot_id, + "person_given_name", + ) + relation = relation_type( + "email_from_name/v1", + (selector,), + selector_type("cluster_role/v1", cluster, "email_address"), + ) + + result = _compile_phase7(plan, execution, plan.coherence_scopes, (relation,)) + + assert _rejection_code(result) == "selector_missing" + + +def test_phase7_slot_identity_is_one_opaque_capability_per_scope_cluster_role_key() -> None: + plan, _backend, execution = _qualified_phase6( + ("Alice Alicia",), + (("target-0",),), + { + "target-0": ( + _Proposal("Alice", "first_name", "person"), + _Proposal("Alicia", "first_name", "person"), + ) + }, + ) + module = _phase7_module() + + result = _compile_phase7(plan, execution, plan.coherence_scopes) + + assert isinstance(result, module._Phase7Plan) + assert len(result.manifests[0].slots) == 1 + slot = result.manifests[0].slots[0] + assert slot.role == "person_given_name" + assert len(slot.mention_ids) == 2 + assert type(slot.id).__name__ == "_ReplacementSlotId" + assert not hasattr(slot.id, "value") + + +def test_phase7_equal_text_and_labels_in_distinct_clusters_create_distinct_slots() -> None: + plan, _backend, execution = _qualified_phase6( + ("Alice", "Alice"), + (("target-0",), ("target-1",)), + { + "target-0": (_Proposal("Alice", "first_name", "first-person"),), + "target-1": (_Proposal("Alice", "first_name", "second-person"),), + }, + ) + module = _phase7_module() + declared = (_CoherenceScope(_ids(plan)),) + + result = _compile_phase7(plan, execution, declared) + + assert isinstance(result, module._Phase7Plan) + slots = result.manifests[0].slots + assert len(slots) == 2 + assert slots[0].id is not slots[1].id + assert slots[0].cluster_id is not slots[1].cluster_id + assert slots[0].role == slots[1].role == "person_given_name" + + +def test_phase7_materializes_every_required_distinct_pair_once_in_deterministic_order() -> None: + plan, _backend, execution = _qualified_phase6( + ("Alice Adams alice@example.com 555-0100",), + (("target-0",),), + { + "target-0": ( + _Proposal("Alice", "first_name", "person"), + _Proposal("Adams", "last_name", "person"), + _Proposal("alice@example.com", "email", "person"), + _Proposal("555-0100", "phone_number", "phone"), + ) + }, + ) + module = _phase7_module() + + forward = _compile_phase7(plan, execution, plan.coherence_scopes) + reverse = _compile_phase7(plan, execution, tuple(reversed(plan.coherence_scopes))) + + assert isinstance(forward, module._Phase7Plan) + assert isinstance(reverse, module._Phase7Plan) + manifest = forward.manifests[0] + assert len(manifest.slots) == 4 + assert len(manifest.required_pairs) == 6 + slot_position = {slot.id: index for index, slot in enumerate(manifest.slots)} + assert tuple((slot_position[pair.left], slot_position[pair.right]) for pair in manifest.required_pairs) == ( + (0, 1), + (0, 2), + (0, 3), + (1, 2), + (1, 3), + (2, 3), + ) + assert tuple(slot.role for slot in forward.manifests[0].slots) == tuple( + slot.role for slot in reverse.manifests[0].slots + ) + + +def test_phase7_count_limits_accept_exact_and_reject_one_over() -> None: + exact_plan, _backend, exact_execution = _qualified_phase6( + ("one", "two", "three", "four"), + (("target-0",), ("target-1",), ("target-2",), ("target-3",)), + {}, + ) + over_plan, _backend, over_execution = _qualified_phase6( + ("one", "two", "three", "four", "five"), + (("target-0",), ("target-1",), ("target-2",), ("target-3",), ("target-4",)), + {}, + ) + module = _phase7_module() + + exact = _compile_phase7(exact_plan, exact_execution, (_CoherenceScope(_ids(exact_plan)),)) + over = _compile_phase7(over_plan, over_execution, (_CoherenceScope(_ids(over_plan)),)) + + assert isinstance(exact, module._Phase7Plan) + assert _rejection_code(over) == "limit_exceeded" + + +def test_phase7_scope_count_limit_accepts_exact_and_precedes_partition_rejections_one_over() -> None: + plan, _backend, execution = _qualified_phase6( + ("one", "two", "three"), + (("target-0",), ("target-1",), ("target-2",)), + {}, + ) + first, second, third = _ids(plan) + module = _phase7_module() + + exact = _compile_phase7( + plan, + execution, + (_CoherenceScope((first, second)), _CoherenceScope((third,))), + ) + one_over_and_empty = _compile_phase7( + plan, + execution, + (_CoherenceScope((first,)), _CoherenceScope((second, third)), _CoherenceScope(())), + ) + + assert isinstance(exact, module._Phase7Plan) + assert _rejection_code(one_over_and_empty) == "limit_exceeded" + + +@pytest.mark.parametrize( + ("exact_count", "over_count", "cluster_mode"), + [ + pytest.param(3, 4, "distinct", id="clusters"), + pytest.param(6, 7, "shared", id="mentions"), + ], +) +def test_phase7_cluster_and_mention_limits_accept_exact_and_reject_one_over( + exact_count: int, + over_count: int, + cluster_mode: str, +) -> None: + def compile_count(count: int) -> object: + sources = tuple(chr(ord("A") + index) for index in range(count)) + proposals = tuple( + _Proposal(source, "first_name", source if cluster_mode == "distinct" else "shared") for source in sources + ) + plan, _backend, execution = _qualified_phase6( + (" ".join(sources),), + (("target-0",),), + {"target-0": proposals}, + ) + return _compile_phase7(plan, execution, plan.coherence_scopes) + + module = _phase7_module() + exact = compile_count(exact_count) + over = compile_count(over_count) + + assert isinstance(exact, module._Phase7Plan) + assert _rejection_code(over) == "limit_exceeded" + + +def test_phase7_slot_and_pair_limits_accept_exact_and_reject_one_over() -> None: + exact_plan, _backend, exact_execution = _qualified_phase6( + ("Alice Adams alice@example.com 555-0100",), + (("target-0",),), + { + "target-0": ( + _Proposal("Alice", "first_name", "person"), + _Proposal("Adams", "last_name", "person"), + _Proposal("alice@example.com", "email", "person"), + _Proposal("555-0100", "phone_number", "phone"), + ) + }, + ) + over_plan, _backend, over_execution = _qualified_phase6( + ("Alice Adams alice@example.com 555-0100 alice_user",), + (("target-0",),), + { + "target-0": ( + _Proposal("Alice", "first_name", "person"), + _Proposal("Adams", "last_name", "person"), + _Proposal("alice@example.com", "email", "person"), + _Proposal("555-0100", "phone_number", "phone"), + _Proposal("alice_user", "user_name", "account"), + ) + }, + ) + module = _phase7_module() + + exact = _compile_phase7(exact_plan, exact_execution, exact_plan.coherence_scopes) + over = _compile_phase7(over_plan, over_execution, over_plan.coherence_scopes) + + assert isinstance(exact, module._Phase7Plan) + assert len(exact.manifests[0].required_pairs) == 6 + assert _rejection_code(over) == "limit_exceeded" + + +def test_phase7_original_value_byte_limit_accepts_exact_and_rejects_one_over() -> None: + def compile_size(size: int) -> object: + source = "A" * size + plan, _backend, execution = _qualified_phase6( + (source,), + (("target-0",),), + {"target-0": (_Proposal(source, "first_name", "person"),)}, + ) + return _compile_phase7(plan, execution, plan.coherence_scopes) + + module = _phase7_module() + exact = compile_size(256) + over = compile_size(257) + + assert isinstance(exact, module._Phase7Plan) + assert _rejection_code(over) == "limit_exceeded" + + +@pytest.mark.parametrize( + ("exact_context", "over_context"), + [ + pytest.param(("a", "b", "c", "d"), ("a", "b", "c", "d", "e"), id="fragment-count"), + pytest.param(("A" * 4096,), ("A" * 4097,), id="fragment-bytes"), + pytest.param( + ("A" * 2731, "B" * 2731, "C" * 2730), + ("A" * 2731, "B" * 2731, "C" * 2731), + id="scope-context-bytes", + ), + ], +) +def test_phase7_context_limits_accept_exact_and_reject_one_over( + exact_context: tuple[str, ...], + over_context: tuple[str, ...], +) -> None: + def compile_context(context: tuple[str, ...]) -> object: + plan, _backend, execution = _qualified_phase6( + ("target",), + (("target-0",),), + {}, + context_texts=context, + ) + return _compile_phase7(plan, execution, plan.coherence_scopes) + + module = _phase7_module() + exact = compile_context(exact_context) + over = compile_context(over_context) + + assert isinstance(exact, module._Phase7Plan) + assert _rejection_code(over) == "limit_exceeded" + + +def test_phase7_relation_limit_accepts_exact_and_rejects_one_over() -> None: + plan, _backend, execution, cluster = _person_relation_fixture() + module = _phase7_module() + relation = module._RelationDeclaration( + "email_from_name/v1", + (module._ClusterRoleSelector("cluster_role/v1", cluster, "person_given_name"),), + module._ClusterRoleSelector("cluster_role/v1", cluster, "email_address"), + ) + + exact = _compile_phase7(plan, execution, plan.coherence_scopes, (relation,) * 4) + over = _compile_phase7(plan, execution, plan.coherence_scopes, (relation,) * 5) + + assert isinstance(exact, module._Phase7Plan) + assert len(exact.manifests[0].relations) == 4 + assert _rejection_code(over) == "limit_exceeded" + + +@pytest.mark.parametrize("failure", ["cross-scope", "role-mismatch", "unknown-relation", "unknown-selector"]) +def test_phase7_relation_declarations_fail_closed(failure: str) -> None: + plan, _backend, execution = _qualified_phase6( + ("Alice alice@example.com", "Bob bob@example.com"), + (("target-0",), ("target-1",)), + { + "target-0": ( + _Proposal("Alice", "first_name", "alice"), + _Proposal("alice@example.com", "email", "alice"), + ), + "target-1": ( + _Proposal("Bob", "first_name", "bob"), + _Proposal("bob@example.com", "email", "bob"), + ), + }, + ) + module = _phase7_module() + first_cluster = execution.handoffs[0].resolved.clustered.clusters[0].id + second_cluster = execution.handoffs[1].resolved.clustered.clusters[0].id + selector_version = "cluster_role/v2" if failure == "unknown-selector" else "cluster_role/v1" + upstream_role = "email_address" if failure == "role-mismatch" else "person_given_name" + downstream_cluster = second_cluster if failure == "cross-scope" else first_cluster + downstream_role = "person_given_name" if failure == "role-mismatch" else "email_address" + relation = module._RelationDeclaration( + "email_from_name/v2" if failure == "unknown-relation" else "email_from_name/v1", + (module._ClusterRoleSelector(selector_version, first_cluster, upstream_role),), + module._ClusterRoleSelector("cluster_role/v1", downstream_cluster, downstream_role), + ) + + result = _compile_phase7(plan, execution, plan.coherence_scopes, (relation,)) + + expected = { + "cross-scope": "cross_scope_relation", + "role-mismatch": "relation_role_mismatch", + "unknown-relation": "unsupported_relation", + "unknown-selector": "unsupported_selector", + } + assert _rejection_code(result) == expected[failure] + + +def test_phase7_plan_and_nested_manifests_reject_tampering_and_serialization() -> None: + plan, _backend, execution, _cluster = _person_relation_fixture() + module = _phase7_module() + result = _compile_phase7(plan, execution, plan.coherence_scopes) + assert isinstance(result, module._Phase7Plan) + manifest = result.manifests[0] + slot = manifest.slots[0] + changed_slot = replace(slot, role="email_address") + changed_manifest = replace(manifest, slots=(changed_slot, *manifest.slots[1:])) + + assert not module._is_admitted_phase7_plan(replace(result, manifests=(changed_manifest,))) + assert not module._is_admitted_phase7_plan(replace(result, manifests=(replace(manifest, members=()),))) + assert not module._is_admitted_phase7_plan(replace(result, application_tasks=())) + assert not module._is_admitted_phase7_plan(replace(result, application_predecessors=())) + with pytest.raises(FrozenInstanceError): + manifest.members = () + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(result) + + +def test_phase7_manifest_shape_is_invariant_to_scope_member_handoff_and_relation_permutations() -> None: + plan, _backend, execution = _qualified_phase6( + ("Alice alice@example.com", "Bob bob@example.com"), + (("target-0",), ("target-1",)), + { + "target-0": ( + _Proposal("Alice", "first_name", "alice"), + _Proposal("alice@example.com", "email", "alice"), + ), + "target-1": ( + _Proposal("Bob", "first_name", "bob"), + _Proposal("bob@example.com", "email", "bob"), + ), + }, + ) + module = _phase7_module() + selectors = [] + for handoff in execution.handoffs: + cluster = handoff.resolved.clustered.clusters[0].id + selectors.append( + module._RelationDeclaration( + "email_from_name/v1", + (module._ClusterRoleSelector("cluster_role/v1", cluster, "person_given_name"),), + module._ClusterRoleSelector("cluster_role/v1", cluster, "email_address"), + ) + ) + forward_scope = (_CoherenceScope(_ids(plan)),) + reverse_scope = (_CoherenceScope(tuple(reversed(_ids(plan)))),) + + forward = _compile_raw(plan, execution.handoffs, forward_scope, tuple(selectors)) + reverse = _compile_raw(plan, tuple(reversed(execution.handoffs)), reverse_scope, tuple(reversed(selectors))) + + assert isinstance(forward, module._Phase7Plan) + assert isinstance(reverse, module._Phase7Plan) + assert _normalized_manifest(forward.manifests[0]) == _normalized_manifest(reverse.manifests[0]) + + +def _normalized_manifest(manifest: object) -> tuple[object, ...]: + slots = getattr(manifest, "slots") + slot_position = {slot.id: index for index, slot in enumerate(slots)} + return ( + tuple(member.value for member in getattr(manifest, "members")), + tuple((slot.role, len(slot.mention_ids)) for slot in slots), + tuple((slot_position[pair.left], slot_position[pair.right]) for pair in getattr(manifest, "required_pairs")), + tuple( + ( + relation.version, + tuple(slot_position[slot_id] for slot_id in relation.upstream), + slot_position[relation.downstream], + ) + for relation in getattr(manifest, "relations") + ), + ) + + +def test_phase7_empty_manifest_and_compilation_have_zero_runtime_effects() -> None: + plan, backend, execution = _qualified_phase6( + ("no supported mention",), + (("target-0",),), + {}, + ) + module = _phase7_module() + calls_before = tuple(backend.calls) + + result = _compile_phase7(plan, execution, plan.coherence_scopes) + + assert isinstance(result, module._Phase7Plan) + assert result.manifests[0].slots == () + assert result.manifests[0].required_pairs == () + assert result.manifests[0].relations == () + assert tuple(backend.calls) == calls_before + assert backend.planner_effect_count == 0 + assert not { + "_AccountingLedger", + "_observe_context_boundary", + "_build_context_workframes", + "_apply_redact_patches", + "NddAdapter", + "DataDesigner", + } & set(vars(module)) diff --git a/tests/engine/execution/test_phase7_application.py b/tests/engine/execution/test_phase7_application.py new file mode 100644 index 00000000..70711042 --- /dev/null +++ b/tests/engine/execution/test_phase7_application.py @@ -0,0 +1,493 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib +import importlib.util +import itertools +import pickle +from dataclasses import FrozenInstanceError, dataclass, replace +from types import ModuleType +from typing import Any + +import pytest + +from anonymizer.engine.execution.graph import _CoherenceScope +from anonymizer.engine.execution.mention_admission import _MentionId, _ValidationDecision, _ValidationDecisionKind +from anonymizer.engine.execution.mention_resolution import _EvidenceVersion, _SameSubjectEvidence +from anonymizer.engine.execution.phase6_plan import _compile_phase6_plan, _Phase6Plan, _Phase6ProfileVersion +from anonymizer.engine.execution.phase6_runtime import ( + _CandidateProposal, + _Phase6CandidateWork, + _Phase6Execution, + _Phase6ResolverWork, + _Phase6Runtime, +) +from anonymizer.engine.execution.phase7_admission import _Phase7Plan +from anonymizer.engine.execution.phase7_contract import _load_phase7_contract +from anonymizer.engine.execution.phase7_validation import ( + _CandidateAssignment, + _validate_scope_bundle, + _ValidatedBundle, +) +from tests.engine.execution.test_phase7_admission import ( + _ACCOUNTING_LIMITS, + _MENTION_LIMITS, + _compile_phase7, + _contract_and_capability, + _graph, + _Proposal, + _qualified_phase6, +) + + +@dataclass(frozen=True) +class _SpanProposal: + start: int + end: int + source: str + label: str + cluster: str + + +class _SpanBackend: + def __init__(self, proposals: dict[str, tuple[_SpanProposal, ...]]) -> None: + self._proposals = proposals + self.calls: list[str] = [] + + def context_capability(self) -> object: + return _contract_and_capability()[1] + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + self.calls.append("detect") + return tuple( + _CandidateProposal(item.start, item.end, item.source, item.label) + for item in self._proposals.get(work.target.datum_id.value, ()) + ) + + def augment(self, work: object) -> tuple[()]: + del work + self.calls.append("augment") + return () + + def validate(self, work: object) -> tuple[_ValidationDecision, ...]: + self.calls.append("validate") + candidates = getattr(work, "candidates") + return tuple(_ValidationDecision(item.token, _ValidationDecisionKind.KEEP) for item in candidates) + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SameSubjectEvidence, ...]: + self.calls.append("resolve") + by_anchor = { + (datum_id, item.start, item.end, item.source, item.label): item + for datum_id, proposals in self._proposals.items() + for item in proposals + } + grouped: dict[str, list[_MentionId]] = {} + for mention in work.eligible_mentions: + proposal = by_anchor[ + ( + mention.target_datum_id.value, + mention.start, + mention.end, + mention.source_slice, + mention.detector_label, + ) + ] + grouped.setdefault(proposal.cluster, []).append(mention.id) + return tuple( + _SameSubjectEvidence(work.owner.token, left, right, _EvidenceVersion.V1) + for mention_ids in grouped.values() + for left, right in itertools.pairwise(mention_ids) + ) + + def close_phase6(self) -> bool: + return True + + +def _application_module() -> ModuleType: + module_name = "anonymizer.engine.execution.phase7_application" + assert importlib.util.find_spec(module_name) is not None, "the private Phase 7 application module is missing" + return importlib.import_module(module_name) + + +def _qualified_phase6_spans( + texts: tuple[str, ...], + scopes: tuple[tuple[str, ...], ...], + proposals: dict[str, tuple[_SpanProposal, ...]], +) -> tuple[_Phase6Plan, _SpanBackend, _Phase6Execution]: + contract, capability = _contract_and_capability() + plan = _compile_phase6_plan( + _graph(texts, scopes), + accounting_limits=_ACCOUNTING_LIMITS, + context_contract=contract, + capability=capability, + mention_limits=_MENTION_LIMITS, + profile_version=_Phase6ProfileVersion.SUBSTITUTE_V1, + ) + assert isinstance(plan, _Phase6Plan) + backend = _SpanBackend(proposals) + execution = _Phase6Runtime(backend).run(plan) + assert len(execution.handoffs) == len(plan.components) + return plan, backend, execution + + +def _validated( + texts: tuple[str, ...], + scopes: tuple[tuple[str, ...], ...], + proposals: dict[str, tuple[_Proposal, ...]], + values: tuple[str, ...], + *, + combined_scope: bool = False, +) -> tuple[_ValidatedBundle, object]: + plan, backend, execution = _qualified_phase6(texts, scopes, proposals) + return _validated_from_execution(plan, execution, values, backend, combined_scope=combined_scope) + + +def _validated_spans( + texts: tuple[str, ...], + scopes: tuple[tuple[str, ...], ...], + proposals: dict[str, tuple[_SpanProposal, ...]], + values: tuple[str, ...], + *, + combined_scope: bool = False, +) -> tuple[_ValidatedBundle, object]: + plan, backend, execution = _qualified_phase6_spans(texts, scopes, proposals) + return _validated_from_execution(plan, execution, values, backend, combined_scope=combined_scope) + + +def _validated_from_execution( + plan: _Phase6Plan, + execution: _Phase6Execution, + values: tuple[str, ...], + backend: object, + *, + combined_scope: bool, +) -> tuple[_ValidatedBundle, object]: + declared = ( + (_CoherenceScope(tuple(datum.id for datum in plan.accounting.datums)),) + if combined_scope + else plan.coherence_scopes + ) + compiled = _compile_phase7(plan, execution, declared) + assert isinstance(compiled, _Phase7Plan) + assert len(compiled.manifests) == 1 + manifest = compiled.manifests[0] + assert len(manifest.slots) == len(values) + assignments = tuple( + _CandidateAssignment(slot.id, value) for slot, value in zip(manifest.slots, values, strict=True) + ) + result = _validate_scope_bundle(manifest, execution.handoffs, assignments, _load_phase7_contract()) + assert isinstance(result, _ValidatedBundle) + return result, backend + + +def _materialize(bundle: object) -> object: + materialize = getattr(_application_module(), "_materialize_substitute_patches", None) + assert callable(materialize), "the private anchored-patch materializer is missing" + return materialize(bundle) + + +def _apply(bundle: object, patches: object) -> object: + apply_patches = getattr(_application_module(), "_apply_substitute_patches", None) + assert callable(apply_patches), "the private anchored reconstruction is missing" + return apply_patches(bundle, patches) + + +def _code(result: object) -> str: + code = getattr(result, "code", None) + assert code is not None, "invalid application did not return a typed rejection" + value = getattr(code, "value", None) + assert isinstance(value, str) + return value + + +def _outputs(result: object) -> dict[str, tuple[str, bool]]: + datums = getattr(result, "datums", None) + assert isinstance(datums, tuple) + return {item.datum_id.value: (item.output, item.applied) for item in datums} + + +def _patch_tuple(bundle: _ValidatedBundle) -> tuple[Any, ...]: + patches = _materialize(bundle) + assert isinstance(patches, tuple) + return patches + + +def test_phase7_application_requires_every_exact_opaque_mention_token_once() -> None: + bundle, _backend = _validated( + ("Alice Adams",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"), _Proposal("Adams", "last_name", "person"))}, + ("Nova", "Vale"), + ) + patches = _patch_tuple(bundle) + patch_type = getattr(_application_module(), "_SubstitutePatch") + + accepted = _apply(bundle, patches) + + assert _outputs(accepted) == {"target-0": ("Nova Vale", True)} + assert _code(_apply(bundle, patches[:-1])) == "invalid_application" + assert _code(_apply(bundle, (*patches, patches[0]))) == "invalid_application" + assert _code(_apply(bundle, (*patches, replace(patches[0], replacement="Mira")))) == "invalid_application" + + foreign_bundle, _foreign_backend = _validated( + ("Mallory",), + (("target-0",),), + {"target-0": (_Proposal("Mallory", "first_name", "other"),)}, + ("Tess",), + ) + foreign = _patch_tuple(foreign_bundle)[0] + assert isinstance(foreign, patch_type) + assert _code(_apply(bundle, (foreign, *patches[1:]))) == "invalid_application" + + +@pytest.mark.parametrize("malformed", [None, [], (), (object(),)], ids=["none", "list", "empty", "object"]) +def test_phase7_application_is_total_for_malformed_patch_bundles(malformed: object) -> None: + bundle, _backend = _validated( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ("Nova",), + ) + + assert _code(_apply(bundle, malformed)) == "invalid_application" + + +@pytest.mark.parametrize( + ("field_name", "replacement"), + [ + pytest.param("start", 1, id="start"), + pytest.param("end", 4, id="end"), + pytest.param("source_slice", "Alic", id="source-slice"), + pytest.param("replacement", "Vale", id="replacement"), + ], +) +def test_phase7_application_rejects_every_non_authoritative_patch_field( + field_name: str, + replacement: object, +) -> None: + bundle, _backend = _validated( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ("Nova",), + ) + patch = _patch_tuple(bundle)[0] + + assert _code(_apply(bundle, (replace(patch, **{field_name: replacement}),))) == "invalid_application" + + +def test_phase7_application_rejects_foreign_target_and_mention_tokens() -> None: + bundle, _backend = _validated( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ("Nova",), + ) + foreign_bundle, _foreign_backend = _validated( + ("Mallory",), + (("target-0",),), + {"target-0": (_Proposal("Mallory", "first_name", "other"),)}, + ("Tess",), + ) + patch = _patch_tuple(bundle)[0] + foreign = _patch_tuple(foreign_bundle)[0] + + assert _code(_apply(bundle, (replace(patch, mention_id=foreign.mention_id),))) == "invalid_application" + assert _code(_apply(bundle, (replace(patch, target=foreign.target),))) == "invalid_application" + + +def test_phase7_repeated_text_uses_only_each_authoritative_span_without_raw_fallback() -> None: + text = "Alice Alice Alice" + bundle, _backend = _validated_spans( + (text,), + (("target-0",),), + { + "target-0": ( + _SpanProposal(6, 11, "Alice", "first_name", "second"), + _SpanProposal(12, 17, "Alice", "first_name", "third"), + ) + }, + ("Nova", "Vale"), + ) + + result = _apply(bundle, _patch_tuple(bundle)) + + assert _outputs(result) == {"target-0": ("Alice Nova Vale", True)} + + +def test_phase7_shared_slot_consumes_each_repeated_mention_once() -> None: + text = "Alice/Alice" + bundle, _backend = _validated_spans( + (text,), + (("target-0",),), + { + "target-0": ( + _SpanProposal(0, 5, "Alice", "first_name", "person"), + _SpanProposal(6, 11, "Alice", "first_name", "person"), + ) + }, + ("Nova",), + ) + patches = _patch_tuple(bundle) + + assert len(patches) == 2 + assert len({patch.mention_id for patch in patches}) == 2 + assert _outputs(_apply(bundle, patches)) == {"target-0": ("Nova/Nova", True)} + + +def test_phase7_adjacent_mentions_are_reconstructed_in_source_order() -> None: + bundle, _backend = _validated_spans( + ("AliceAdams!",), + (("target-0",),), + { + "target-0": ( + _SpanProposal(0, 5, "Alice", "first_name", "person"), + _SpanProposal(5, 10, "Adams", "last_name", "person"), + ) + }, + ("Nova", "Vale"), + ) + + assert _outputs(_apply(bundle, _patch_tuple(bundle))) == {"target-0": ("NovaVale!", True)} + + +def test_phase7_combining_and_astral_unicode_offsets_are_python_source_intervals() -> None: + text = "😀Jose\u0301 met 𐐀lice" + first = "Jose\u0301" + second = "𐐀lice" + first_start = text.index(first) + second_start = text.index(second) + bundle, _backend = _validated_spans( + (text,), + (("target-0",),), + { + "target-0": ( + _SpanProposal(first_start, first_start + len(first), first, "first_name", "first"), + _SpanProposal(second_start, second_start + len(second), second, "first_name", "second"), + ) + }, + ("René", "Nova"), + ) + + assert _outputs(_apply(bundle, _patch_tuple(bundle))) == {"target-0": ("😀René met Nova", True)} + + +def test_phase7_reconstruction_preserves_every_unmentioned_source_interval() -> None: + text = "AA Alice :: Adams ZZ" + bundle, _backend = _validated( + (text,), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"), _Proposal("Adams", "last_name", "person"))}, + ("Alexandria", "Li"), + ) + + assert _outputs(_apply(bundle, _patch_tuple(bundle))) == {"target-0": ("AA Alexandria :: Li ZZ", True)} + + +def test_phase7_patch_input_permutations_have_one_canonical_reconstruction() -> None: + bundle, _backend = _validated( + ("Alice Adams",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"), _Proposal("Adams", "last_name", "person"))}, + ("Nova", "Vale"), + ) + patches = _patch_tuple(bundle) + + outputs = {_outputs(_apply(bundle, permutation))["target-0"] for permutation in itertools.permutations(patches)} + + assert outputs == {("Nova Vale", True)} + + +def test_phase7_application_never_searches_or_mutates_evolving_output() -> None: + bundle, _backend = _validated( + ("Alice met Nova",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "alice"), _Proposal("Nova", "first_name", "nova"))}, + ("Nova Blake", "Vale"), + ) + + result = _apply(bundle, _patch_tuple(bundle)) + + assert _outputs(result) == {"target-0": ("Nova Blake met Vale", True)} + + +def test_phase7_missing_application_never_releases_an_admitted_original() -> None: + bundle, _backend = _validated( + ("Alice and Bob",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "alice"), _Proposal("Bob", "first_name", "bob"))}, + ("Nova", "Vale"), + ) + patches = _patch_tuple(bundle) + + rejected = _apply(bundle, patches[:1]) + + assert _code(rejected) == "invalid_application" + assert not hasattr(rejected, "datums") + + +def test_phase7_empty_scope_and_unmentioned_members_pass_through_from_source() -> None: + empty, _empty_backend = _validated(("plain 😀 text",), (("target-0",),), {}, ()) + mixed, _mixed_backend = _validated( + ("Alice", "plain 😀 text"), + (("target-0",), ("target-1",)), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ("Nova",), + combined_scope=True, + ) + + assert _patch_tuple(empty) == () + assert _outputs(_apply(empty, ())) == {"target-0": ("plain 😀 text", False)} + assert _outputs(_apply(mixed, _patch_tuple(mixed))) == { + "target-0": ("Nova", True), + "target-1": ("plain 😀 text", False), + } + + +def test_phase7_application_rejects_a_stale_validated_bundle_before_materialization() -> None: + bundle, _backend = _validated( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ("Nova",), + ) + stale_manifest = replace(bundle.manifest, members=()) + stale_bundle = replace(bundle, manifest=stale_manifest) + + assert _code(_materialize(stale_bundle)) == "invalid_application" + assert _code(_apply(stale_bundle, ())) == "invalid_application" + + +def test_phase7_applied_result_is_private_immutable_nonserializable_and_content_free_in_repr() -> None: + bundle, _backend = _validated( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ("Nova",), + ) + result = _apply(bundle, _patch_tuple(bundle)) + + with pytest.raises(FrozenInstanceError): + setattr(result, "datums", ()) + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(result) + assert "Alice" not in repr(result) + assert "Nova" not in repr(result) + + +def test_phase7_application_has_no_backend_effects_or_public_substitute_dependency() -> None: + bundle, backend = _validated( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ("Nova",), + ) + calls_before = tuple(getattr(backend, "calls")) + + result = _apply(bundle, _patch_tuple(bundle)) + + assert _outputs(result) == {"target-0": ("Nova", True)} + assert tuple(getattr(backend, "calls")) == calls_before + assert not {"Substitute", "ReplacementWorkflow", "apply_replacements_to_spans"} & set(vars(_application_module())) diff --git a/tests/engine/execution/test_phase7_contract.py b/tests/engine/execution/test_phase7_contract.py new file mode 100644 index 00000000..6d16b36c --- /dev/null +++ b/tests/engine/execution/test_phase7_contract.py @@ -0,0 +1,286 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import copy +import hashlib +import importlib +import importlib.util +import json +import pickle +from collections.abc import Mapping +from dataclasses import FrozenInstanceError, replace +from importlib.resources import files +from types import ModuleType +from typing import cast + +import pytest + +from anonymizer.engine.execution import role_policy as role_policy_module +from anonymizer.engine.execution.role_policy import _load_redact_role_policy, _RolePolicyRejected + +_CONTRACT_DIGEST = "3755832ecc64fe6e9dbeccc136c40020e5158446e5c74d6dca2392efdbb006bb" +_POLICY_DIGEST = "c27580bd2cc4051bdd11b63a91391f8995bdef1ed2052534623cdd3160318ef8" +_CORPUS_DIGEST = "5ba1e69c7836a428b16e9d2ac7fc0cf3fbb445fd0ca72c1d0a8a049194115123" + + +def test_phase7_contract_test_infrastructure() -> None: + assert importlib.util.find_spec("anonymizer.engine.execution.role_policy") is not None + + +def _phase7_contract_module() -> ModuleType: + module_name = "anonymizer.engine.execution.phase7_contract" + assert importlib.util.find_spec(module_name) is not None, "Phase 7 contract module is missing" + return importlib.import_module(module_name) + + +def test_phase7_contract_loader_returns_the_exact_frozen_private_contract() -> None: + module = _phase7_contract_module() + + result = module._load_phase7_contract() + + assert type(result).__name__ == "_Phase7StableSubstituteContract" + assert result.version.value == "anonymizer-phase7-stable-substitute/v1" + assert result.digest == _CONTRACT_DIGEST + assert result.phase6_result_version == "phase6-role-result/v1" + assert result.phase6_policy_version == "phase6-substitute-role-policy/v1" + assert result.phase6_policy_digest == _POLICY_DIGEST + assert result.selectors == ("cluster_role/v1",) + assert result.relations == ("email_from_name/v1",) + assert result.formats == ( + "email_addr_spec_ascii/v1", + "telephone_ascii/v1", + "unicode_person_name/v1", + "username_ascii/v1", + ) + assert result.masks == ("digit_literal/v1", "none/v1") + assert result.corpus_version == "anonymizer-phase7-owner-contract-corpus/v1" + assert result.corpus_case_count == 30 + assert result.corpus_digest == _CORPUS_DIGEST + assert tuple(role.name for role in result.roles) == ( + "email_address", + "fax_number", + "person_family_name", + "person_given_name", + "user_name", + "voice_phone_number", + ) + assert dict(result.count_limits) == { + "max_clusters_per_invocation": 3, + "max_clusters_per_scope": 3, + "max_context_fragments_per_scope": 4, + "max_datums_per_invocation": 4, + "max_distinct_pairs_per_scope": 6, + "max_mentions_per_invocation": 6, + "max_mentions_per_scope": 6, + "max_relations_per_scope": 4, + "max_scope_members": 4, + "max_scopes_per_invocation": 2, + "max_slots_per_invocation": 4, + "max_slots_per_scope": 4, + } + assert module._is_admitted_phase7_contract(result) + assert repr(result).startswith(" tuple[dict[str, object], dict[str, object], dict[str, object]]: + package = files("anonymizer.engine.execution") + names = ( + "phase7_stable_substitute_contract.json", + "phase6_substitute_role_policy.json", + "phase7_owner_contract_corpus.json", + ) + payloads = tuple(json.loads(package.joinpath(name).read_text(encoding="utf-8")) for name in names) + return cast(tuple[dict[str, object], dict[str, object], dict[str, object]], payloads) + + +def _canonical_digest(value: object) -> str: + encoded = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def _set_nested(mapping: dict[str, object], path: tuple[str, ...], value: object) -> None: + current = mapping + for key in path[:-1]: + current = cast(dict[str, object], current[key]) + current[path[-1]] = value + + +@pytest.mark.parametrize( + ("path", "value"), + [ + pytest.param(("version",), "anonymizer-phase7-stable-substitute/v2", id="unknown-version"), + pytest.param(("selectors", "unknown_selector/v1"), {}, id="unknown-selector"), + pytest.param(("relations", "unknown_relation/v1"), {}, id="unknown-relation"), + pytest.param(("relations", "wildcard_constraints"), "wildcard/v1", id="unsupported-wildcard"), + pytest.param(("roles", "email_address", "format"), "unknown-format/v1", id="unknown-format"), + pytest.param(("roles", "email_address", "mask"), "unknown-mask/v1", id="unknown-mask"), + pytest.param( + ("relations", "email_from_name/v1", "downstream_role"), + "voice_phone_number", + id="unsupported-relation", + ), + pytest.param(("execution", "capability", "profile"), "unknown-profile/v1", id="invalid-capability"), + pytest.param(("limits", "counts", "max_slots_per_scope"), 5, id="count-one-over"), + pytest.param(("limits", "bytes", "max_candidate_value_bytes"), 257, id="bytes-one-over"), + ], +) +def test_phase7_contract_rejects_every_self_consistent_semantic_mutation( + path: tuple[str, ...], + value: object, +) -> None: + module = _phase7_contract_module() + envelope, policy, corpus = copy.deepcopy(_resource_payloads()) + contract = cast(dict[str, object], envelope["contract"]) + _set_nested(contract, path, value) + envelope["digest"] = _canonical_digest(contract) + + result = module._compile_phase7_contract(envelope, policy, corpus) + + assert type(result).__name__ == "_Phase7ContractRejected" + + +@pytest.mark.parametrize("mutation", ["missing-label", "unknown-role"]) +def test_phase7_contract_rejects_an_incomplete_or_unknown_phase6_disposition(mutation: str) -> None: + module = _phase7_contract_module() + envelope, policy, corpus = copy.deepcopy(_resource_payloads()) + contract = cast(dict[str, object], envelope["contract"]) + handoff = cast(dict[str, object], contract["phase6_handoff"]) + expected_policy = cast(dict[str, object], handoff["substitute_policy"]) + dispositions = cast(dict[str, object], policy["dispositions"]) + if mutation == "missing-label": + dispositions.pop("account_number") + expected_policy["detector_label_count"] = 64 + else: + dispositions["email"] = "unknown_role" + expected_policy["digest"] = _canonical_digest(policy) + envelope["digest"] = _canonical_digest(contract) + + result = module._compile_phase7_contract(envelope, policy, corpus) + + assert type(result).__name__ == "_Phase7ContractRejected" + + +def test_phase7_contract_rejects_noncanonical_digest_input() -> None: + module = _phase7_contract_module() + envelope, policy, corpus = copy.deepcopy(_resource_payloads()) + contract = cast(dict[str, object], envelope["contract"]) + envelope["digest"] = hashlib.sha256(json.dumps(contract, indent=2).encode("utf-8")).hexdigest() + + result = module._compile_phase7_contract(envelope, policy, corpus) + + assert type(result).__name__ == "_Phase7ContractRejected" + + +def test_phase7_contract_rejects_a_self_consistent_alternate_corpus() -> None: + module = _phase7_contract_module() + envelope, policy, corpus = copy.deepcopy(_resource_payloads()) + contract = cast(dict[str, object], envelope["contract"]) + expected_corpus = cast(dict[str, object], contract["oracle_contract_corpus"]) + cases = cast(list[object], corpus["cases"]) + cases.pop() + expected_corpus["case_count"] = 29 + expected_corpus["digest"] = _canonical_digest(corpus) + envelope["digest"] = _canonical_digest(contract) + + result = module._compile_phase7_contract(envelope, policy, corpus) + + assert type(result).__name__ == "_Phase7ContractRejected" + + +def test_phase7_contract_accepts_json_whitespace_and_object_key_order_variation( + monkeypatch: pytest.MonkeyPatch, +) -> None: + module = _phase7_contract_module() + envelope, policy, corpus = _resource_payloads() + texts = { + "phase7_stable_substitute_contract.json": json.dumps(envelope, indent=7), + "phase6_substitute_role_policy.json": json.dumps(policy, separators=(", ", ": ")), + "phase7_owner_contract_corpus.json": json.dumps(corpus, indent=1), + } + monkeypatch.setattr(module, "files", lambda _package: _ResourceDirectory(texts)) + + result = module._load_phase7_contract() + + assert type(result).__name__ == "_Phase7StableSubstituteContract" + assert result.digest == _CONTRACT_DIGEST + + +@pytest.mark.parametrize("failure", ["invalid-json", "duplicate-key", "read-error"]) +def test_phase7_contract_loader_fails_closed_on_malformed_or_unreadable_resources( + monkeypatch: pytest.MonkeyPatch, + failure: str, +) -> None: + module = _phase7_contract_module() + envelope, policy, corpus = _resource_payloads() + contract_text = json.dumps(envelope) + if failure == "invalid-json": + contract_text = "{" + elif failure == "duplicate-key": + contract_text = contract_text[:-1] + ',"schema_version":"anonymizer-phase7-owner-contract-envelope/v1"}' + texts: dict[str, str | OSError] = { + "phase7_stable_substitute_contract.json": OSError("unreadable") if failure == "read-error" else contract_text, + "phase6_substitute_role_policy.json": json.dumps(policy), + "phase7_owner_contract_corpus.json": json.dumps(corpus), + } + monkeypatch.setattr(module, "files", lambda _package: _ResourceDirectory(texts)) + + result = module._load_phase7_contract() + + assert type(result).__name__ == "_Phase7ContractRejected" + + +def test_phase7_contract_proof_rejects_forgery_and_nested_values_are_immutable() -> None: + module = _phase7_contract_module() + result = module._load_phase7_contract() + assert type(result).__name__ == "_Phase7StableSubstituteContract" + + forged = replace(result, digest="0" * 64) + + assert not module._is_admitted_phase7_contract(forged) + with pytest.raises(FrozenInstanceError): + result.roles[0].name = "changed" + assert _contains_only_immutable_json(result._source_snapshot) + + +def _contains_only_immutable_json(value: object) -> bool: + if value is None or type(value) in {bool, int, str}: + return True + return type(value) is tuple and all(_contains_only_immutable_json(item) for item in value) + + +def test_phase7_substitute_policy_cannot_load_through_the_redact_policy_path( + monkeypatch: pytest.MonkeyPatch, +) -> None: + package = files("anonymizer.engine.execution") + substitute_text = package.joinpath("phase6_substitute_role_policy.json").read_text(encoding="utf-8") + monkeypatch.setattr( + role_policy_module, + "files", + lambda _package: _ResourceDirectory({"phase6_redact_role_policy.json": substitute_text}), + ) + + assert isinstance(_load_redact_role_policy(), _RolePolicyRejected) + + +class _ResourceDirectory: + def __init__(self, texts: Mapping[str, str | OSError]) -> None: + self._texts = dict(texts) + self._name: str | None = None + + def joinpath(self, name: str) -> _ResourceDirectory: + resource = _ResourceDirectory(self._texts) + resource._name = name + return resource + + def read_text(self, *, encoding: str) -> str: + assert encoding == "utf-8" + assert self._name is not None + value = self._texts[self._name] + if isinstance(value, OSError): + raise value + return value diff --git a/tests/engine/execution/test_phase7_ndd_backend.py b/tests/engine/execution/test_phase7_ndd_backend.py new file mode 100644 index 00000000..b8c63a24 --- /dev/null +++ b/tests/engine/execution/test_phase7_ndd_backend.py @@ -0,0 +1,857 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib +import importlib.util +import inspect +import json +from collections.abc import Callable, Iterator +from contextlib import contextmanager +from dataclasses import dataclass, field, replace +from types import ModuleType +from typing import Any, cast + +import pandas as pd +import pytest + +from anonymizer.config.anonymizer_config import AnonymizerConfig +from anonymizer.config.replace_strategies import Redact +from anonymizer.engine import constants as execution_constants +from anonymizer.engine.execution.accounting_evidence import ( + _AttemptId, + _Dispatch, + _InvocationId, + _RowToken, +) +from anonymizer.engine.execution.accounting_plan import _ScopeTaskSubject, _StageId, _TaskKey +from anonymizer.engine.execution.graph import _CoherenceScope +from anonymizer.engine.execution.mention_resolution import _ClusterId +from anonymizer.engine.execution.phase7_admission import _Phase7Plan, _ScopeManifest +from anonymizer.engine.execution.phase7_contract import _load_phase7_contract, _Phase7StableSubstituteContract +from anonymizer.engine.execution.phase7_validation import _validate_scope_bundle, _ValidatedBundle +from anonymizer.engine.ndd.adapter import ( + RECORD_ID_COLUMN, + FailedRecord, + NddAdapter, + WorkflowRunResult, + _FailedRowEvidence, +) +from tests.engine.execution.test_phase7_admission import ( + _compile_phase7, + _ids, + _person_relation_fixture, + _Proposal, + _qualified_phase6, +) +from tests.engine.execution.test_phase7_validation import _compiled_scope +from tests.streaming.structured_trace_prototype import build_synthetic_anonymizer + +_Response = Callable[[pd.DataFrame, list[Any]], WorkflowRunResult] +_DISPATCH_TASK = _TaskKey(_StageId("phase7-plan"), _ScopeTaskSubject()) + + +@dataclass +class _ScriptedAdapter: + response: _Response + calls: list[pd.DataFrame] = field(default_factory=list) + columns: list[list[Any]] = field(default_factory=list) + workflow_names: list[str] = field(default_factory=list) + private_depth: int = 0 + + @contextmanager + def private_execution(self) -> Iterator[None]: + self.private_depth += 1 + try: + yield + finally: + self.private_depth -= 1 + + def run_workflow( + self, + dataframe: pd.DataFrame, + *, + columns: list[Any], + workflow_name: str, + **_: Any, + ) -> WorkflowRunResult: + assert self.private_depth == 1 + self.calls.append(dataframe.copy()) + self.columns.append(columns) + self.workflow_names.append(workflow_name) + return self.response(dataframe.copy(), columns) + + +def _backend_module() -> ModuleType: + module_name = "anonymizer.engine.execution.phase7_ndd_backend" + assert importlib.util.find_spec(module_name) is not None, "the private Phase 7 NDD backend module is missing" + return importlib.import_module(module_name) + + +def _column(name: str) -> str: + value = getattr(execution_constants, name, None) + assert isinstance(value, str), f"the private Phase 7 {name} column is missing" + return value + + +def _invocation() -> Any: + anonymizer = build_synthetic_anonymizer({}) + return anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)).invocation + + +def _dispatch( + *, + invocation: str = "invocation-current", + attempt: str = "attempt-current", + row: str = "row-current", +) -> _Dispatch: + return _Dispatch( + _InvocationId(invocation), + _DISPATCH_TASK, + _AttemptId(attempt), + _RowToken(row), + ) + + +def _identity_factory(*values: str) -> Callable[[], str]: + iterator = iter(values) + return iterator.__next__ + + +def _single_name_scope() -> tuple[_ScopeManifest, tuple[object, ...]]: + manifest, handoffs = _compiled_scope( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ) + return manifest, cast(tuple[object, ...], handoffs) + + +def _empty_scope() -> tuple[_ScopeManifest, tuple[object, ...]]: + plan, _backend, execution = _qualified_phase6(("Nothing sensitive",), (("target-0",),), {}) + compiled = _compile_phase7(plan, execution, plan.coherence_scopes) + assert isinstance(compiled, _Phase7Plan) + assert len(compiled.manifests) == 1 + assert compiled.manifests[0].slots == () + return compiled.manifests[0], cast(tuple[object, ...], execution.handoffs) + + +def _related_scope() -> tuple[_ScopeManifest, tuple[object, ...]]: + plan, _backend, execution, raw_cluster = _person_relation_fixture() + cluster = cast(_ClusterId, raw_cluster) + module = importlib.import_module("anonymizer.engine.execution.phase7_admission") + selector = module._ClusterRoleSelector + relation = module._RelationDeclaration( + "email_from_name/v1", + ( + selector("cluster_role/v1", cluster, "person_given_name"), + selector("cluster_role/v1", cluster, "person_family_name"), + ), + selector("cluster_role/v1", cluster, "email_address"), + ) + compiled = _compile_phase7(plan, execution, (_CoherenceScope(_ids(plan)),), (relation,)) + assert isinstance(compiled, _Phase7Plan) + return compiled.manifests[0], cast(tuple[object, ...], execution.handoffs) + + +def _same_role_scope() -> tuple[_ScopeManifest, tuple[object, ...]]: + return _compiled_scope( + ("Alice", "Bob"), + (("target-0",), ("target-1",)), + { + "target-0": (_Proposal("Alice", "first_name", "person-a"),), + "target-1": (_Proposal("Bob", "first_name", "person-b"),), + }, + combined_scope=True, + ) + + +def _request(frame: pd.DataFrame) -> dict[str, Any]: + value = frame.iloc[0][_column("COL_PHASE7_CANDIDATE_REQUEST")] + assert isinstance(value, str) + parsed = json.loads(value) + assert isinstance(parsed, dict) + return parsed + + +def _success_response(values_by_role: dict[str, str]) -> _Response: + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + request = _request(frame) + assignments = [ + {"slot_token": slot["slot_token"], "value": values_by_role[slot["role"]]} + for slot in reversed(request["slots"]) + ] + output = frame.copy() + output[_column("COL_PHASE7_CANDIDATE_BUNDLE")] = [{"assignments": assignments}] + output["_anonymizer_record_id"] = ["backend-randomized-id"] + return WorkflowRunResult(output, []) + + return respond + + +def _backend(adapter: _ScriptedAdapter, *identities: str) -> Any: + backend_type = getattr(_backend_module(), "_Phase7NddBackend") + return backend_type( + cast(NddAdapter, adapter), + _invocation(), + identity_factory=_identity_factory(*identities), + ) + + +def _propose( + backend: Any, + manifest: object, + handoffs: object, + dispatch: object, +) -> object: + return backend.propose_scope(manifest, handoffs, _load_phase7_contract(), dispatch) + + +def _status(result: object) -> str: + status = getattr(result, "status", None) + value = getattr(status, "value", None) + assert isinstance(value, str) + return value + + +def _reason(result: object) -> str | None: + reason = getattr(result, "reason", None) + if reason is None: + return None + value = getattr(reason, "value", None) + assert isinstance(value, str) + return value + + +def _assignment_pairs(result: object) -> tuple[tuple[object, str], ...]: + assignments = getattr(result, "assignments", None) + assert isinstance(assignments, tuple) + return tuple((assignment.token, assignment.value) for assignment in assignments) + + +def test_phase7_zero_slot_scope_is_no_work_without_an_attempt_or_adapter_call() -> None: + manifest, handoffs = _empty_scope() + adapter = _ScriptedAdapter(lambda _frame, _columns: pytest.fail("empty scope called the adapter")) + backend = _backend(adapter) + + result = _propose(backend, manifest, handoffs, None) + + assert _status(result) == "no_work" + assert _assignment_pairs(result) == () + assert adapter.calls == [] + + +def test_phase7_nonempty_scope_uses_one_private_single_row_adapter_call() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + backend = _backend(adapter, "task-token", "slot-token") + + result = _propose(backend, manifest, handoffs, _dispatch()) + + assert _status(result) == "candidate" + assert len(adapter.calls) == 1 + assert len(adapter.calls[0]) == 1 + assert adapter.workflow_names == ["phase7-candidate-planning"] + assert adapter.private_depth == 0 + + +def test_phase7_two_nonempty_scopes_each_cross_the_adapter_once() -> None: + plan, _phase6_backend, execution = _qualified_phase6( + ("Alice", "Bob"), + (("target-0",), ("target-1",)), + { + "target-0": (_Proposal("Alice", "first_name", "person-a"),), + "target-1": (_Proposal("Bob", "first_name", "person-b"),), + }, + ) + compiled = _compile_phase7(plan, execution, plan.coherence_scopes) + assert isinstance(compiled, _Phase7Plan) + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + backend = _backend(adapter, "task-a", "slot-a", "task-b", "slot-b") + + results = tuple( + _propose( + backend, + manifest, + execution.handoffs, + _dispatch(attempt=f"attempt-{index}", row=f"row-{index}"), + ) + for index, manifest in enumerate(compiled.manifests) + ) + + assert tuple(_status(result) for result in results) == ("candidate", "candidate") + assert len(adapter.calls) == 2 + assert all(len(frame) == 1 for frame in adapter.calls) + + +def test_phase7_workframe_contains_only_opaque_correlations_and_governed_request_material() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + backend = _backend(adapter, "task-token", "slot-token") + + _propose(backend, manifest, handoffs, _dispatch()) + + frame = adapter.calls[0] + assert tuple(frame.columns) == ( + execution_constants.COL_TARGET_WORK_ID, + _column("COL_PHASE7_INVOCATION_ID"), + execution_constants.COL_TASK_ID, + execution_constants.COL_ATTEMPT_ID, + _column("COL_PHASE7_CANDIDATE_REQUEST"), + RECORD_ID_COLUMN, + ) + assert frame.iloc[0][execution_constants.COL_TARGET_WORK_ID] == "row-current" + assert frame.iloc[0][_column("COL_PHASE7_INVOCATION_ID")] == "invocation-current" + assert frame.iloc[0][execution_constants.COL_TASK_ID] == "task-token" + assert frame.iloc[0][execution_constants.COL_ATTEMPT_ID] == "attempt-current" + request = _request(frame) + assert set(request) == {"schema_version", "slots", "required_distinct_pairs", "relations"} + assert request["schema_version"] == "phase7-workframe/v1" + assert request["slots"] == [ + { + "slot_token": "slot-token", + "role": "person_given_name", + "format": "unicode_person_name/v1", + "mask": "none/v1", + "source_values": ["Alice"], + } + ] + serialized = json.dumps(request, sort_keys=True) + assert "target-0" not in serialized + assert "source_id" not in serialized + assert "mention" not in serialized + assert "cluster" not in serialized + assert frame.iloc[0][RECORD_ID_COLUMN] == "row-current" + + +def test_phase7_adapter_tracking_identity_is_opaque_and_not_content_derived() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + backend = _backend(adapter, "task-token", "slot-token") + + _propose(backend, manifest, handoffs, _dispatch()) + + frame = adapter.calls[0] + real_adapter = object.__new__(NddAdapter) + attached = real_adapter._attach_record_ids(frame) + assert attached[RECORD_ID_COLUMN].tolist() == ["row-current"] + assert "Alice" not in attached[RECORD_ID_COLUMN].iloc[0] + + +def test_phase7_ndd_declaration_uses_the_replacement_model_and_governed_request_column() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + backend = _backend(adapter, "task-token", "slot-token") + + _propose(backend, manifest, handoffs, _dispatch()) + + assert len(adapter.columns[0]) == 1 + column = adapter.columns[0][0] + assert column.name == _column("COL_PHASE7_CANDIDATE_BUNDLE") + assert _column("COL_PHASE7_CANDIDATE_REQUEST") in str(column.prompt) + assert column.model_alias == _invocation().selected_models.replace.replacement_generator + + +def test_phase7_hydrates_permuted_assignments_by_ephemeral_slot_token() -> None: + manifest, handoffs = _related_scope() + values = { + "person_given_name": "Mira", + "person_family_name": "Stone", + "email_address": "mira.stone@example.com", + } + adapter = _ScriptedAdapter(_success_response(values)) + backend = _backend(adapter, "task-token", "slot-a", "slot-b", "slot-c") + + result = _propose(backend, manifest, handoffs, _dispatch()) + + assert _status(result) == "candidate" + assert _assignment_pairs(result) == tuple((slot.id, values[slot.role]) for slot in manifest.slots) + + +def test_phase7_same_role_slots_hydrate_by_token_instead_of_role_or_order() -> None: + manifest, handoffs = _same_role_scope() + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + slots = _request(frame)["slots"] + output = frame.copy() + output[_column("COL_PHASE7_CANDIDATE_BUNDLE")] = [ + { + "assignments": [ + {"slot_token": slots[1]["slot_token"], "value": "Nora"}, + {"slot_token": slots[0]["slot_token"], "value": "Mira"}, + ] + } + ] + return WorkflowRunResult(output, []) + + result = _propose( + _backend(_ScriptedAdapter(respond), "task-token", "slot-a", "slot-b"), + manifest, + handoffs, + _dispatch(), + ) + + assert _assignment_pairs(result) == ((manifest.slots[0].id, "Mira"), (manifest.slots[1].id, "Nora")) + + +def test_phase7_backend_ids_and_output_column_order_are_not_identity() -> None: + manifest, handoffs = _single_name_scope() + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + request = _request(frame) + output = pd.DataFrame( + { + "_anonymizer_record_id": ["untrusted-random-backend-id"], + _column("COL_PHASE7_CANDIDATE_BUNDLE"): [ + {"assignments": [{"slot_token": request["slots"][0]["slot_token"], "value": "Mira"}]} + ], + execution_constants.COL_ATTEMPT_ID: ["attempt-current"], + execution_constants.COL_TASK_ID: ["task-token"], + _column("COL_PHASE7_INVOCATION_ID"): ["invocation-current"], + execution_constants.COL_TARGET_WORK_ID: ["row-current"], + } + ) + return WorkflowRunResult(output, []) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "candidate" + assert _assignment_pairs(result) == ((manifest.slots[0].id, "Mira"),) + + +@pytest.mark.parametrize("fault", ["missing", "duplicate", "foreign"]) +def test_phase7_rejects_nonexact_slot_token_hydration_as_invocation_inconsistent(fault: str) -> None: + manifest, handoffs = _single_name_scope() + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + token = _request(frame)["slots"][0]["slot_token"] + assignments: list[dict[str, str]] = [] + if fault != "missing": + assignments.append({"slot_token": "foreign-slot" if fault == "foreign" else token, "value": "Mira"}) + if fault == "duplicate": + assignments.append({"slot_token": token, "value": "Other"}) + output = frame.copy() + output[_column("COL_PHASE7_CANDIDATE_BUNDLE")] = [{"assignments": assignments}] + return WorkflowRunResult(output, []) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + assert _reason(result) == "evidence_unattributable" + assert _assignment_pairs(result) == () + + +@pytest.mark.parametrize( + ("column_name", "value"), + [ + ("COL_PHASE7_INVOCATION_ID", "invocation-foreign"), + ("COL_TASK_ID", "task-foreign"), + ("COL_ATTEMPT_ID", "attempt-stale"), + ("COL_TARGET_WORK_ID", "row-foreign"), + ], +) +def test_phase7_rejects_foreign_or_stale_row_correlations(column_name: str, value: str) -> None: + manifest, handoffs = _single_name_scope() + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + token = _request(frame)["slots"][0]["slot_token"] + output = frame.copy() + output[_column(column_name)] = [value] + output[_column("COL_PHASE7_CANDIDATE_BUNDLE")] = [{"assignments": [{"slot_token": token, "value": "Mira"}]}] + return WorkflowRunResult(output, []) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + assert _assignment_pairs(result) == () + + +@pytest.mark.parametrize("column_name", ["COL_PHASE7_INVOCATION_ID", "COL_ATTEMPT_ID", "COL_TARGET_WORK_ID"]) +def test_phase7_missing_scalar_correlations_fail_closed(column_name: str) -> None: + manifest, handoffs = _single_name_scope() + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + token = _request(frame)["slots"][0]["slot_token"] + output = frame.copy() + output[_column(column_name)] = [pd.NA] + output[_column("COL_PHASE7_CANDIDATE_BUNDLE")] = [{"assignments": [{"slot_token": token, "value": "Mira"}]}] + return WorkflowRunResult(output, []) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + assert _reason(result) == "evidence_unattributable" + + +def test_phase7_rejects_duplicate_success_rows_even_when_payloads_match() -> None: + manifest, handoffs = _single_name_scope() + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + token = _request(frame)["slots"][0]["slot_token"] + output = pd.concat([frame, frame], ignore_index=True) + output[_column("COL_PHASE7_CANDIDATE_BUNDLE")] = [ + {"assignments": [{"slot_token": token, "value": "Mira"}]}, + {"assignments": [{"slot_token": token, "value": "Mira"}]}, + ] + return WorkflowRunResult(output, []) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + + +def test_phase7_trusted_call_token_attributes_exactly_one_complete_scope_task_failure() -> None: + manifest, handoffs = _single_name_scope() + failure = FailedRecord("untrusted-backend-record", "phase7-candidate-planning", "dropped") + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + return WorkflowRunResult(frame.iloc[0:0].copy(), [failure], (_FailedRowEvidence("row-current", failure),)) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "task_failed" + assert _reason(result) == "backend_failed" + assert _assignment_pairs(result) == () + + +def test_phase7_rejects_a_non_failed_record_even_with_matching_private_evidence() -> None: + manifest, handoffs = _single_name_scope() + malformed = object() + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + return WorkflowRunResult( + frame.iloc[0:0].copy(), + [cast(FailedRecord, malformed)], + (_FailedRowEvidence("row-current", cast(FailedRecord, malformed)),), + ) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + assert _reason(result) == "evidence_unattributable" + + +def test_phase7_rejects_missing_failed_evidence_correlation_without_raising() -> None: + manifest, handoffs = _single_name_scope() + failure = FailedRecord("untrusted-backend-record", "phase7-candidate-planning", "dropped") + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + return WorkflowRunResult( + frame.iloc[0:0].copy(), + [failure], + (_FailedRowEvidence(cast(str, pd.NA), failure),), + ) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + assert _reason(result) == "evidence_unattributable" + + +@pytest.mark.parametrize( + "fault", + ["unattributed", "foreign", "stale", "duplicate", "duplicate_records", "success_plus_failure"], +) +def test_phase7_ambiguous_failed_record_evidence_causes_global_inconsistency(fault: str) -> None: + manifest, handoffs = _single_name_scope() + failure = FailedRecord("untrusted-backend-record", "phase7-candidate-planning", "dropped") + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + failed = [failure] + evidence: tuple[_FailedRowEvidence, ...] = () + output = frame.iloc[0:0].copy() + if fault == "foreign": + evidence = (_FailedRowEvidence("row-foreign", failure),) + elif fault == "stale": + evidence = (_FailedRowEvidence("row-from-stale-attempt", failure),) + elif fault == "duplicate": + evidence = (_FailedRowEvidence("row-current", failure), _FailedRowEvidence("row-current", failure)) + elif fault == "duplicate_records": + failed = [failure, failure] + evidence = (_FailedRowEvidence("row-current", failure),) + elif fault == "success_plus_failure": + token = _request(frame)["slots"][0]["slot_token"] + output = frame.copy() + output[_column("COL_PHASE7_CANDIDATE_BUNDLE")] = [{"assignments": [{"slot_token": token, "value": "Mira"}]}] + evidence = (_FailedRowEvidence("row-current", failure),) + return WorkflowRunResult(output, failed, evidence) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + assert _reason(result) == "evidence_unattributable" + assert _assignment_pairs(result) == () + + +def test_phase7_failed_row_evidence_without_a_failure_is_inconsistent() -> None: + manifest, handoffs = _single_name_scope() + failure = FailedRecord("untrusted-backend-record", "phase7-candidate-planning", "dropped") + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + return WorkflowRunResult(frame, [], (_FailedRowEvidence("row-current", failure),)) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + + +def test_phase7_adapter_exception_after_dispatch_poisoned_as_transport_loss() -> None: + manifest, handoffs = _single_name_scope() + + def fail(_frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + raise RuntimeError("SECRET-CANDIDATE-and-original") + + adapter = _ScriptedAdapter(fail) + result = _propose(_backend(adapter, "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "poisoned" + assert _reason(result) is None + assert "SECRET" not in repr(result) + assert len(adapter.calls) == 1 + + +def test_phase7_rejects_an_over_limit_workframe_before_adapter_execution() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(lambda _frame, _columns: pytest.fail("over-limit workframe reached adapter")) + backend = _backend(adapter, "x" * 20_000, "slot-token") + + result = _propose(backend, manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + assert _reason(result) == "limit_exceeded" + assert adapter.calls == [] + + +def test_phase7_planner_replays_an_admitted_manifest_copy_without_redispatch() -> None: + manifest, handoffs = _single_name_scope() + admitted_copy = replace(manifest) + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + backend = _backend(adapter, "task-token", "slot-token") + + first = _propose(backend, manifest, handoffs, _dispatch()) + replay = _propose(backend, admitted_copy, handoffs, _dispatch()) + + assert replay is first + assert _status(replay) == "candidate" + assert len(adapter.calls) == 1 + + +def test_phase7_nonidentical_terminal_replay_cannot_read_or_rewrite_planned_result() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + backend = _backend(adapter, "task-token", "slot-token") + + accepted = _propose(backend, manifest, handoffs, _dispatch()) + stale = _propose(backend, replace(manifest), handoffs, _dispatch(attempt="later", row="later-row")) + exact = _propose(backend, replace(manifest), handoffs, _dispatch()) + + assert _status(accepted) == "candidate" + assert _status(stale) == "poisoned" + assert exact is accepted + assert len(adapter.calls) == 1 + + +def test_phase7_nonidentical_replay_before_acceptance_poison_the_reservation() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + backend: Any + + def barrier(stage: str) -> None: + if stage == "reserve": + conflicting = _propose(backend, replace(manifest), handoffs, _dispatch(attempt="other", row="other")) + assert _status(conflicting) == "poisoned" + + backend = getattr(_backend_module(), "_Phase7NddBackend")( + cast(NddAdapter, adapter), _invocation(), identity_factory=_identity_factory("task", "slot"), barrier=barrier + ) + result = _propose(backend, manifest, handoffs, _dispatch()) + + assert _status(result) == "poisoned" + assert adapter.calls == [] + + +def test_phase7_cancellation_at_reserve_aborts_before_adapter_dispatch() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(lambda _frame, _columns: pytest.fail("cancelled scope dispatched")) + backend: Any + + def barrier(stage: str) -> None: + if stage == "reserve": + assert backend.cancel_scope(manifest) is not None + + backend = getattr(_backend_module(), "_Phase7NddBackend")(cast(NddAdapter, adapter), _invocation(), barrier=barrier) + result = _propose(backend, manifest, handoffs, _dispatch()) + + assert _status(result) == "aborted" + assert adapter.calls == [] + assert _propose(backend, manifest, handoffs, _dispatch()) == result + + +def test_phase7_post_dispatch_untrusted_cancellation_poison_rejects_late_success() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + backend: Any + + def barrier(stage: str) -> None: + if stage == "dispatch": + assert backend.cancel_scope(manifest, trusted_stop=False) is not None + + backend = getattr(_backend_module(), "_Phase7NddBackend")( + cast(NddAdapter, adapter), _invocation(), identity_factory=_identity_factory("task", "slot"), barrier=barrier + ) + result = _propose(backend, manifest, handoffs, _dispatch()) + + assert _status(result) == "poisoned" + # The dispatch barrier is post-dispatch: cancellation makes the returned + # adapter success late evidence, not a reason to suppress the first call. + assert len(adapter.calls) == 1 + + +def test_phase7_crash_after_dispatch_poison_replays_without_a_second_effect() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + + def crash() -> None: + raise ConnectionError + + backend = getattr(_backend_module(), "_Phase7NddBackend")( + cast(NddAdapter, adapter), + _invocation(), + identity_factory=_identity_factory("task", "slot"), + crash_after_dispatch=crash, + ) + result = _propose(backend, manifest, handoffs, _dispatch()) + + assert _status(result) == "poisoned" + assert len(adapter.calls) == 1 + assert _propose(backend, manifest, handoffs, _dispatch()) is result + + +def test_phase7_workframe_byte_ceiling_accepts_exactly_limit_and_rejects_one_over() -> None: + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + limit = dict(contract.byte_limits)["max_workframe_bytes_per_scope"] + row = { + execution_constants.COL_TARGET_WORK_ID: "row-current", + _column("COL_PHASE7_INVOCATION_ID"): "invocation-current", + execution_constants.COL_TASK_ID: "task-token", + execution_constants.COL_ATTEMPT_ID: "attempt-current", + _column("COL_PHASE7_CANDIDATE_REQUEST"): "", + RECORD_ID_COLUMN: "row-current", + } + fixed_bytes = len(json.dumps(row, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode("utf-8")) + exact_request = "x" * (limit - fixed_bytes) + dataframe = _backend_module()._candidate_dataframe( + _dispatch(), + "task-token", + exact_request, + max_bytes=limit, + ) + rejected = _backend_module()._candidate_dataframe( + _dispatch(), + "task-token", + exact_request + "x", + max_bytes=limit, + ) + + assert isinstance(dataframe, pd.DataFrame) + assert rejected is None + + +@pytest.mark.parametrize( + "identities", + [ + ("", "slot-token"), + ("row-current", "slot-token"), + ("task-token", "task-token"), + ("task-token", cast(str, None)), + ], +) +def test_phase7_invalid_or_colliding_generated_identity_fails_before_adapter( + identities: tuple[str, str], +) -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(lambda _frame, _columns: pytest.fail("invalid identity reached adapter")) + + result = _propose(_backend(adapter, *identities), manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + assert _reason(result) == "limit_exceeded" + assert adapter.calls == [] + + +def test_phase7_structural_output_parse_failure_is_content_free_and_inconsistent() -> None: + manifest, handoffs = _single_name_scope() + + def respond(frame: pd.DataFrame, _columns: list[Any]) -> WorkflowRunResult: + output = frame.copy() + output[_column("COL_PHASE7_CANDIDATE_BUNDLE")] = [ + {"assignments": [{"slot_token": "slot-token", "value": b"SECRET-CANDIDATE"}]} + ] + return WorkflowRunResult(output, []) + + result = _propose(_backend(_ScriptedAdapter(respond), "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + assert _reason(result) == "evidence_unattributable" + assert "SECRET" not in repr(result) + + +def test_phase7_p5_rejects_an_invalid_candidate_before_publication() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Alice"})) + backend = _backend(adapter, "task-token", "slot-token") + result = _propose(backend, manifest, handoffs, _dispatch()) + + assert _status(result) == "invocation_inconsistent" + assert getattr(result, "assignments") == () + assert _propose(backend, manifest, handoffs, _dispatch()) is result + assert len(adapter.calls) == 1 + + +def test_phase7_reentrant_close_returns_the_absorbing_terminal_result() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + backend: Any + + def barrier(stage: str) -> None: + if stage == "cleanup": + backend.close() + + backend = getattr(_backend_module(), "_Phase7NddBackend")( + cast(NddAdapter, adapter), + _invocation(), + identity_factory=_identity_factory("task", "slot"), + barrier=barrier, + ) + result = _propose(backend, manifest, handoffs, _dispatch()) + replay = _propose(backend, manifest, handoffs, _dispatch()) + + assert _status(result) == "candidate" + assert replay is result + assert len(adapter.calls) == 1 + + +def test_phase7_hydrated_valid_candidate_can_cross_the_unchanged_p5_boundary() -> None: + manifest, handoffs = _single_name_scope() + adapter = _ScriptedAdapter(_success_response({"person_given_name": "Mira"})) + result = _propose(_backend(adapter, "task-token", "slot-token"), manifest, handoffs, _dispatch()) + + p5_result = _validate_scope_bundle( + manifest, + handoffs, + getattr(result, "assignments"), + _load_phase7_contract(), + ) + + assert isinstance(p5_result, _ValidatedBundle) + + +def test_phase7_backend_has_no_direct_datadesigner_execution_path() -> None: + source = inspect.getsource(_backend_module()) + + assert "DataDesigner.create" not in source + assert "DataDesigner.preview" not in source + assert ".create(" not in source + assert ".preview(" not in source diff --git a/tests/engine/execution/test_phase7_p5_reference_conformance.py b/tests/engine/execution/test_phase7_p5_reference_conformance.py new file mode 100644 index 00000000..07ec0398 --- /dev/null +++ b/tests/engine/execution/test_phase7_p5_reference_conformance.py @@ -0,0 +1,206 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +from typing import cast + +import pytest + +from anonymizer.engine.execution.mention_resolution import _ClusterId +from anonymizer.engine.execution.phase7_admission import _Phase7Plan +from anonymizer.engine.execution.phase7_validation import ( + _BundleRejected, + _CandidateAssignment, + _ValidatedBundle, +) +from tests.engine.execution.phase7_reference_model import ( + _canonical_value as _reference_canonical_value, +) +from tests.engine.execution.phase7_reference_model import ( + _digit_mask_valid as _reference_digit_mask_valid, +) +from tests.engine.execution.phase7_reference_model import ( + _format_valid as _reference_format_valid, +) +from tests.engine.execution.phase7_reference_model import ( + case_by_name, + reduce_reference, +) +from tests.engine.execution.test_phase7_admission import ( + _compile_phase7, + _person_relation_fixture, + _Proposal, +) +from tests.engine.execution.test_phase7_application import _apply, _outputs, _patch_tuple, _validated +from tests.engine.execution.test_phase7_validation import ( + _assignments_for_roles, + _code, + _compiled_scope, + _validate, +) + + +@pytest.mark.parametrize( + "value", + ["\u2003 Ａ-lí_ce \u2003", "Straße", "É９", "e\u0301", "-_ .", "𐐀lice"], +) +def test_p5_canonicalization_matches_the_independent_p4_oracle(value: str) -> None: + from anonymizer.engine.execution.phase7_validation import _canonicalize_value + + assert _canonicalize_value(value) == (_reference_canonical_value(value) or None) + + +@pytest.mark.parametrize( + ("format_name", "value"), + [ + ("unicode_person_name/v1", "Élodie Marie-José."), + ("unicode_person_name/v1", "A\u2028B"), + ("username_ascii/v1", "alice_01"), + ("username_ascii/v1", "álîce"), + ("telephone_ascii/v1", "+1 (555) 010-0200"), + ("telephone_ascii/v1", "1+234567"), + ("email_addr_spec_ascii/v1", "nova.vale@example.test"), + ("email_addr_spec_ascii/v1", "nova@-example.test"), + ], +) +def test_p5_format_predicates_match_the_independent_p4_oracle(format_name: str, value: str) -> None: + from anonymizer.engine.execution.phase7_validation import _matches_format + + assert _matches_format(format_name, value) is _reference_format_valid(format_name, value) + + +@pytest.mark.parametrize( + ("source", "candidate"), + [("555-0100", "555-0199"), ("555-0100", "555 0199"), ("+１ (555)-0100", "+2 (212)-9999")], +) +def test_p5_digit_mask_matches_the_independent_p4_oracle(source: str, candidate: str) -> None: + from anonymizer.engine.execution.phase7_validation import _matches_mask + + assert _matches_mask("digit_literal/v1", source, candidate) is _reference_digit_mask_valid(source, candidate) + + +@pytest.mark.parametrize( + ("case_name", "text", "proposals", "values"), + [ + ( + "owner-distinct_slots_same_canonical_value", + "Alice Adams", + (_Proposal("Alice", "first_name", "c0"), _Proposal("Adams", "last_name", "c0")), + ("Nova", "Ｎｏｖａ"), + ), + ( + "owner-candidate_matches_own_original", + "Alice", + (_Proposal("Alice", "first_name", "c0"),), + (" Alice ",), + ), + ( + "owner-candidate_matches_other_slot_original", + "Alice Adams", + (_Proposal("Alice", "first_name", "c0"), _Proposal("Adams", "last_name", "c0")), + ("Adams", "Vale"), + ), + ], +) +def test_p5_scope_validation_reasons_match_the_independent_p4_oracle( + case_name: str, + text: str, + proposals: tuple[_Proposal, ...], + values: tuple[str, ...], +) -> None: + manifest, handoffs = _compiled_scope((text,), (("target-0",),), {"target-0": proposals}) + assignments = tuple( + _CandidateAssignment(slot.id, value) for slot, value in zip(manifest.slots, values, strict=True) + ) + production = _validate(manifest, handoffs, assignments) + reference = reduce_reference(case_by_name(case_name)) + + assert isinstance(production, _BundleRejected) + assert _code(production) == reference.reason_codes[0] + + +@pytest.mark.parametrize( + ("case_name", "text", "proposals", "values"), + [ + ( + "shared-slot-reuse", + "Alice and Alicia", + (_Proposal("Alice", "first_name", "c0"), _Proposal("Alicia", "first_name", "c0")), + ("Nova",), + ), + ( + "owner-valid_phone_source_mask", + "555-0100", + (_Proposal("555-0100", "phone_number", "c0"),), + ("555-0199",), + ), + ( + "anchored-non-cascading-application", + "Alice met Nova", + (_Proposal("Alice", "first_name", "c0"), _Proposal("Nova", "last_name", "c1")), + ("Nova Blake", "Vale"), + ), + ], +) +def test_p5_reuse_mask_and_application_outputs_match_the_independent_p4_oracle( + case_name: str, + text: str, + proposals: tuple[_Proposal, ...], + values: tuple[str, ...], +) -> None: + bundle, _backend = _validated((text,), (("target-0",),), {"target-0": proposals}, values) + production = _apply(bundle, _patch_tuple(bundle)) + reference = reduce_reference(case_by_name(case_name)) + + assert reference.reason_codes == (None,) + assert _outputs(production)["target-0"][0] == reference.outputs[0][1] + + +def test_p5_email_relation_acceptance_and_rejection_match_the_independent_p4_oracle() -> None: + plan, _backend, execution, cluster_value = _person_relation_fixture() + cluster = cast(_ClusterId, cluster_value) + from anonymizer.engine.execution import phase7_admission as admission + + relation = admission._RelationDeclaration( + "email_from_name/v1", + ( + admission._ClusterRoleSelector("cluster_role/v1", cluster, "person_given_name"), + admission._ClusterRoleSelector("cluster_role/v1", cluster, "person_family_name"), + ), + admission._ClusterRoleSelector("cluster_role/v1", cluster, "email_address"), + ) + compiled = _compile_phase7(plan, execution, plan.coherence_scopes, (relation,)) + assert isinstance(compiled, _Phase7Plan) + manifest = compiled.manifests[0] + + accepted = _validate( + manifest, + execution.handoffs, + _assignments_for_roles( + manifest, + { + "person_given_name": "Nova", + "person_family_name": "Vale", + "email_address": "nova.vale@example.test", + }, + ), + ) + rejected = _validate( + manifest, + execution.handoffs, + _assignments_for_roles( + manifest, + { + "person_given_name": "Nova", + "person_family_name": "Vale", + "email_address": "other@example.test", + }, + ), + ) + accepted_reference = reduce_reference(case_by_name("owner-valid_given_family_email_relation")) + rejected_reference = reduce_reference(case_by_name("owner-email_local_part_omits_name")) + + assert isinstance(accepted, _ValidatedBundle) + assert _outputs(_apply(accepted, _patch_tuple(accepted)))["target-0"][0] == accepted_reference.outputs[0][1] + assert _code(rejected) == rejected_reference.reason_codes[0] diff --git a/tests/engine/execution/test_phase7_planner_ledger.py b/tests/engine/execution/test_phase7_planner_ledger.py new file mode 100644 index 00000000..7c072e24 --- /dev/null +++ b/tests/engine/execution/test_phase7_planner_ledger.py @@ -0,0 +1,87 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Unit coverage for the bounded private Phase 7 planner ledger.""" + +from __future__ import annotations + +import copy +import pickle + +import pytest + +from anonymizer.engine.execution.phase7_planner_ledger import ( + _PlannerLedger, + _PlannerSnapshot, + _PlannerState, + _Reservation, +) + + +def test_reservation_is_identity_only_and_terminal_publication_is_one_shot() -> None: + ledger: _PlannerLedger[str] = _PlannerLedger() + scope = object() + reservation, replay = ledger.reserve(scope) + + assert isinstance(reservation, _Reservation) + assert replay is None + assert not ledger.owns(scope, _Reservation()) + assert ledger.terminal(scope, reservation, _PlannerSnapshot(_PlannerState.PLANNED, "bundle")) + assert not ledger.terminal(scope, reservation, _PlannerSnapshot(_PlannerState.PLANNED, "other")) + + replay_reservation, replay = ledger.reserve(scope) + assert replay_reservation is None + assert replay == _PlannerSnapshot(_PlannerState.PLANNED, "bundle") + + +def test_close_preserves_accepted_publication_for_exact_replay_and_rejects_new_work() -> None: + ledger: _PlannerLedger[str] = _PlannerLedger() + scope = object() + reservation, _ = ledger.reserve(scope, ("dispatch",)) + assert isinstance(reservation, _Reservation) + assert ledger.terminal(scope, reservation, _PlannerSnapshot(_PlannerState.PLANNED, "bundle")) + + ledger.close() + + replay_reservation, replay = ledger.reserve(scope, ("dispatch",)) + assert replay_reservation is None + assert replay == _PlannerSnapshot(_PlannerState.PLANNED, "bundle") + with pytest.raises(RuntimeError): + ledger.reserve(object(), ("new-dispatch",)) + assert not ledger.terminal(scope, reservation, _PlannerSnapshot(_PlannerState.PLANNED, "other")) + + +def test_close_poison_active_reservations_and_rejects_later_access_or_escape() -> None: + ledger: _PlannerLedger[str] = _PlannerLedger() + scope = object() + reservation, _ = ledger.reserve(scope) + assert isinstance(reservation, _Reservation) + + with pytest.raises(TypeError): + copy.copy(ledger) + with pytest.raises(TypeError): + pickle.dumps(ledger) + + ledger.close() + + assert ledger.current(scope) == _PlannerSnapshot(_PlannerState.POISONED) + replay_reservation, replay = ledger.reserve(scope) + assert replay_reservation is None + assert replay == _PlannerSnapshot(_PlannerState.POISONED) + assert not ledger.terminal(scope, reservation, _PlannerSnapshot(_PlannerState.PLANNED, "bundle")) + + +def test_verified_cleanup_retires_accepted_values_without_rewriting_terminal_state() -> None: + ledger: _PlannerLedger[str] = _PlannerLedger() + scope = object() + reservation, _ = ledger.reserve(scope) + assert isinstance(reservation, _Reservation) + assert ledger.terminal(scope, reservation, _PlannerSnapshot(_PlannerState.PLANNED, "bundle")) + + ledger.close() + ledger.discard_values() + + assert ledger.current(scope) == _PlannerSnapshot(_PlannerState.PLANNED) + replay_reservation, replay = ledger.reserve(scope) + assert replay_reservation is None + assert replay == _PlannerSnapshot(_PlannerState.PLANNED) diff --git a/tests/engine/execution/test_phase7_reference_conformance.py b/tests/engine/execution/test_phase7_reference_conformance.py new file mode 100644 index 00000000..a2dd091a --- /dev/null +++ b/tests/engine/execution/test_phase7_reference_conformance.py @@ -0,0 +1,182 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +from collections.abc import Callable + +import pytest + +from anonymizer.engine.execution.graph import _CoherenceScope, _DatumId +from anonymizer.engine.execution.phase7_admission import _Phase7Plan, _ScopeManifest +from tests.engine.execution.phase7_reference_model import ( + ReferenceManifest, + case_by_name, + reduce_reference, +) +from tests.engine.execution.test_phase7_admission import ( + _compile_phase7, + _ids, + _phase7_module, + _Proposal, + _qualified_phase6, + _rejection_code, +) + +_LABELS = ("first_name", "last_name", "email", "phone_number") +_SOURCES = ("Alice", "Adams", "alice@example.com", "555-0100") + + +@pytest.mark.parametrize("slot_count", range(5)) +def test_p3_sealed_manifest_matches_the_independent_zero_to_four_slot_oracle(slot_count: int) -> None: + labels = _LABELS[:slot_count] + sources = _SOURCES[:slot_count] + text = " ".join(sources) if sources else "plain-0" + plan, _backend, execution = _qualified_phase6( + (text,), + (("target-0",),), + {"target-0": tuple(_Proposal(source, label, "c0") for label, source in zip(labels, sources, strict=True))}, + ) + production = _compile_phase7(plan, execution, plan.coherence_scopes) + reference = reduce_reference(case_by_name(f"future-slots-{slot_count}")) + module = _phase7_module() + + assert isinstance(production, module._Phase7Plan) + assert module._is_admitted_phase7_plan(production) + assert _production_manifest_shape(production.manifests) == _reference_manifest_shape(reference.manifests) + + +def test_p3_sealed_manifests_match_two_independent_reference_scopes() -> None: + plan, _backend, execution = _qualified_phase6( + ("Alice Adams", "Bob Stone"), + (("target-0",), ("target-1",)), + { + "target-0": ( + _Proposal("Alice", "first_name", "c0"), + _Proposal("Adams", "last_name", "c0"), + ), + "target-1": ( + _Proposal("Bob", "first_name", "c1"), + _Proposal("Stone", "last_name", "c1"), + ), + }, + ) + + production = _compile_phase7(plan, execution, plan.coherence_scopes) + reference = reduce_reference(case_by_name("independent-scopes-2-2")) + + assert isinstance(production, _Phase7Plan) + assert _production_manifest_shape(production.manifests) == _reference_manifest_shape(reference.manifests) + + +@pytest.mark.parametrize( + ("case_name", "production_scopes", "expected"), + [ + ( + "admission-empty-scope", + lambda ids: (_CoherenceScope(()), _CoherenceScope(ids)), + "empty_scope", + ), + ( + "admission-duplicate-scope", + lambda ids: (_CoherenceScope(ids), _CoherenceScope(tuple(reversed(ids)))), + "duplicate_scope", + ), + ( + "admission-duplicate-member", + lambda ids: ( + _CoherenceScope((ids[0], ids[0], ids[1])), + _CoherenceScope((ids[2], ids[3])), + ), + "duplicate_scope_member", + ), + ( + "admission-unknown-datum", + lambda ids: ( + _CoherenceScope((ids[0], ids[1], ids[2])), + _CoherenceScope((_DatumId("foreign"),)), + ), + "unknown_scope_datum", + ), + ( + "admission-coverage-gap", + lambda ids: (_CoherenceScope((ids[0], ids[1], ids[2])),), + "scope_coverage_gap", + ), + ( + "admission-overlap", + lambda ids: ( + _CoherenceScope((ids[0], ids[1])), + _CoherenceScope((ids[1], ids[2], ids[3])), + ), + "scope_overlap", + ), + ( + "admission-nesting", + lambda ids: (_CoherenceScope(ids), _CoherenceScope((ids[0], ids[1]))), + "unsupported_scope_nesting", + ), + ], +) +def test_p3_admission_rejection_matches_the_independent_oracle( + case_name: str, + production_scopes: Callable[[tuple[_DatumId, ...]], tuple[_CoherenceScope, ...]], + expected: str, +) -> None: + plan, _backend, execution = _qualified_phase6( + ("one", "two", "three", "four"), + (("target-0",), ("target-1",), ("target-2",), ("target-3",)), + {}, + ) + scopes = production_scopes(_ids(plan)) + + production = _compile_phase7(plan, execution, scopes) + reference = reduce_reference(case_by_name(case_name)) + + assert reference.admission == expected + assert _rejection_code(production) == expected + + +def _production_manifest_shape(manifests: tuple[_ScopeManifest, ...]) -> tuple[object, ...]: + normalized = [] + for manifest in manifests: + slots = manifest.slots + positions = {slot.id: index for index, slot in enumerate(slots)} + normalized.append( + ( + len(manifest.members), + tuple((slot.role, len(slot.mention_ids)) for slot in slots), + tuple((positions[pair.left], positions[pair.right]) for pair in manifest.required_pairs), + tuple( + ( + relation.version, + tuple(positions[slot_id] for slot_id in relation.upstream), + positions[relation.downstream], + ) + for relation in manifest.relations + ), + ) + ) + return tuple(normalized) + + +def _reference_manifest_shape(manifests: tuple[ReferenceManifest, ...]) -> tuple[object, ...]: + normalized = [] + for manifest in manifests: + positions = {slot.key: index for index, slot in enumerate(manifest.slots)} + normalized.append( + ( + len(manifest.members), + tuple((slot.role, len(slot.mention_indexes)) for slot in manifest.slots), + tuple((positions[left], positions[right]) for left, right in manifest.required_pairs), + tuple( + ( + version, + tuple(positions[slot_key] for slot_key in upstream), + positions[downstream], + ) + for version, upstream, downstream in manifest.relations + ), + ) + ) + return tuple(normalized) diff --git a/tests/engine/execution/test_phase7_reference_model.py b/tests/engine/execution/test_phase7_reference_model.py new file mode 100644 index 00000000..33fceb5a --- /dev/null +++ b/tests/engine/execution/test_phase7_reference_model.py @@ -0,0 +1,292 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import ast +import hashlib +import importlib +import json +from dataclasses import fields, replace +from pathlib import Path + +import pytest + +from tests.engine.execution.phase7_reference_model import ( + MAX_EXOGENOUS_OBSERVATIONS, + OWNER_CASE_IDS, + ReferenceCase, + ReferenceEvent, + ReferenceEventKind, + canonical_corpus_bytes, + canonical_events, + case_by_name, + events_commute, + finite_reference_cases, + owner_case_outcome, + reduce_reference, + reference_manifest, +) + + +def test_phase7_reference_model_defines_timestamp_free_declarations_and_events() -> None: + model = importlib.import_module("tests.engine.execution.phase7_reference_model") + + required = { + "ReferenceCase", + "ReferenceDatum", + "ReferenceDeclaration", + "ReferenceEvent", + "ReferenceEventKind", + "ReferenceMention", + "ReferenceRelation", + "ReferenceScope", + "canonical_events", + "events_commute", + } + + assert required.issubset(vars(model)), "Phase 7 reference grammar is incomplete" + assert tuple(kind.value for kind in model.ReferenceEventKind) == model.EVENT_ALPHABET + + +def test_phase7_reference_model_exposes_an_independent_finite_reducer() -> None: + model = importlib.import_module("tests.engine.execution.phase7_reference_model") + + required = { + "canonical_corpus_bytes", + "corpus_document", + "finite_reference_cases", + "reduce_reference", + "reference_manifest", + } + + assert required.issubset(vars(model)), "Phase 7 reference reducer or corpus generator is missing" + + +def test_phase7_reference_person_name_format_matches_the_p0_zs_only_contract() -> None: + model = importlib.import_module("tests.engine.execution.phase7_reference_model") + + assert model._format_valid("unicode_person_name/v1", "A\u00a0B") + assert not model._format_valid("unicode_person_name/v1", "A\u2028B") + assert not model._format_valid("unicode_person_name/v1", "A\u2029B") + + +def test_phase7_reference_model_preserves_all_owner_contract_cases() -> None: + cases = {case.owner_case: case for case in finite_reference_cases() if case.owner_case is not None} + expected = { + "valid_empty_scope_zero_dispatch": "planned_empty", + "valid_single_given_name": "planned", + "valid_given_family_email_relation": "planned", + "valid_phone_source_mask": "planned", + "unknown_contract_version": "contract_invalid", + "contract_digest_mismatch": "digest_mismatch", + "missing_detector_disposition": "detector_universe_incomplete", + "unknown_role": "unsupported_role", + "unknown_relation": "unsupported_constraint", + "unknown_mask": "unsupported_mask", + "unsupported_detector_label": "unsupported_label", + "selector_resolves_zero_slots": "selector_missing", + "selector_resolves_multiple_slots": "selector_ambiguous", + "relation_crosses_scopes": "cross_scope_relation", + "email_relation_wrong_roles": "relation_role_mismatch", + "distinct_slots_same_canonical_value": "canonical_collision", + "candidate_matches_own_original": "candidate_matches_original", + "candidate_matches_other_slot_original": "candidate_matches_original", + "email_local_part_omits_name": "relation_failed", + "count_limits_exact": "planned", + "count_limits_one_over": "limit_exceeded", + "byte_limits_exact": "planned", + "byte_limits_one_over": "limit_exceeded", + "runtime_capability_missing": "missing_capability", + "trusted_task_failure": "failed", + "unattributable_failure": "inconsistent_global_embargo", + "cleanup_attestation_verified": "release_eligible", + "cleanup_attestation_missing": "inconsistent_global_embargo", + "cleanup_attestation_contradictory": "inconsistent_global_embargo", + "redact_policy_role_bearing_scope": "blocked_zero_effects", + } + + assert tuple(cases) == OWNER_CASE_IDS + assert {case_id: owner_case_outcome(case) for case_id, case in cases.items()} == expected + + +@pytest.mark.parametrize("slot_count", range(5)) +def test_future_policy_enumerates_zero_through_four_slots_and_all_required_pairs(slot_count: int) -> None: + result = reduce_reference(case_by_name(f"future-slots-{slot_count}")) + + assert result.admission == "admitted" + assert result.scope_outcomes == ("planned",) + assert len(result.manifests[0].slots) == slot_count + assert len(result.manifests[0].required_pairs) == slot_count * (slot_count - 1) // 2 + assert result.dispatch_count == int(slot_count > 0) + assert result.attempt_count == int(slot_count > 0) + assert result.released_groups == (0,) + + +@pytest.mark.parametrize("slot_count", range(5)) +def test_current_empty_policy_blocks_every_role_bearing_scope_without_dispatch(slot_count: int) -> None: + result = reduce_reference(case_by_name(f"current-empty-policy-slots-{slot_count}")) + + assert result.dispatch_count == 0 + assert result.attempt_count == 0 + if slot_count == 0: + assert result.scope_outcomes == ("planned",) + assert result.released_groups == (0,) + else: + assert result.scope_outcomes == ("blocked",) + assert result.released_groups == () + + +def test_independent_scopes_have_separate_slots_pairs_tasks_and_release() -> None: + result = reduce_reference(case_by_name("independent-scopes-2-2")) + + assert result.scope_outcomes == ("planned", "planned") + assert tuple(len(manifest.slots) for manifest in result.manifests) == (2, 2) + assert tuple(len(manifest.required_pairs) for manifest in result.manifests) == (1, 1) + assert result.dispatch_count == 2 + assert result.released_groups == (0, 1) + assert tuple(task[:2] for task in result.task_outcomes if task[0] == "scope") == ( + ("scope", "0"), + ("scope", "1"), + ) + + +def test_phase7_reference_lifecycle_is_absorbing_and_release_is_fail_closed() -> None: + expected = { + "dispatch-rejected": ("failed", "completed", (), 0), + "backend-exception": ("failed", "completed", (), 1), + "contradictory-candidate-evidence": ("inconsistent", "inconsistent", (), 1), + "cancel-before-dispatch": ("cancelled", "cancelled", (), 0), + "dispatch-cancel-without-stop": ("lost", "lost", (), 1), + "dispatch-cancel-trusted-stop": ("cancelled", "cancelled", (), 1), + "late-candidate-after-stop": ("cancelled", "cancelled", (), 1), + "late-candidate-after-loss": ("lost", "lost", (), 1), + "foreign-candidate-before-acceptance": ("inconsistent", "inconsistent", (), 1), + "partial-candidate": ("inconsistent", "inconsistent", (), 1), + "planned-then-foreign-is-absorbing": ("planned", "completed", (0,), 1), + "finalization-failure": ("planned", "inconsistent", (), 1), + "cleanup-failure": ("planned", "inconsistent", (), 1), + "teardown-failure-after-acceptance": ("planned", "completed", (0,), 1), + "release-then-cancel-is-absorbing": ("planned", "completed", (0,), 1), + } + + actual = { + name: ( + reduce_reference(case_by_name(name)).scope_outcomes[0], + reduce_reference(case_by_name(name)).invocation, + reduce_reference(case_by_name(name)).released_groups, + reduce_reference(case_by_name(name)).dispatch_count, + ) + for name in expected + } + + assert actual == expected + + +def test_phase4_group_and_dependency_outcomes_define_the_only_legal_release_set() -> None: + atomic = reduce_reference(case_by_name("atomic-group-member-failure")) + dependent = reduce_reference(case_by_name("dependent-datum-withheld")) + independent = reduce_reference(case_by_name("independent-scope-local-failure")) + + assert atomic.released_groups == () + assert dependent.released_groups == () + assert independent.released_groups == (1,) + assert independent.released_datums == ("d1",) + + +def test_anchored_application_is_non_cascading() -> None: + result = reduce_reference(case_by_name("anchored-non-cascading-application")) + + assert result.outputs == (("d0", "Nova Blake met Vale"),) + assert result.released_groups == (0,) + + +def test_canonicalization_collapses_only_commuting_independent_observations() -> None: + case = case_by_name("independent-scopes-1-1") + first_dispatch, first_candidate, second_dispatch, second_candidate = case.events[:4] + + assert events_commute(first_dispatch, second_dispatch, case.declaration) + assert canonical_events((first_dispatch, second_dispatch), case.declaration) == canonical_events( + (second_dispatch, first_dispatch), case.declaration + ) + assert not events_commute(first_dispatch, first_candidate, case.declaration) + assert canonical_events((first_dispatch, first_candidate), case.declaration) != canonical_events( + (first_candidate, first_dispatch), case.declaration + ) + assert reduce_reference(case) == reduce_reference( + replace(case, events=(second_dispatch, second_candidate, first_dispatch, first_candidate, *case.events[4:])) + ) + + +def test_reference_model_rejects_a_trace_over_the_frozen_bound() -> None: + case = case_by_name("future-slots-1") + excessive = replace( + case, + events=tuple( + ReferenceEvent(ReferenceEventKind.CANCELLATION) for _index in range(MAX_EXOGENOUS_OBSERVATIONS + 1) + ), + ) + + with pytest.raises(AssertionError, match="16-observation"): + reduce_reference(excessive) + + +def test_reference_model_forbidden_import_and_input_boundary_is_structural() -> None: + source_path = Path(__file__).with_name("phase7_reference_model.py") + tree = ast.parse(source_path.read_text(encoding="utf-8")) + imported_modules = { + node.module for node in ast.walk(tree) if isinstance(node, ast.ImportFrom) and node.module is not None + } | {alias.name for node in ast.walk(tree) if isinstance(node, ast.Import) for alias in node.names} + + assert not any( + module == forbidden or module.startswith(f"{forbidden}.") + for module in imported_modules + for forbidden in ("anonymizer", "pandas", "data_designer", "datadesigner") + ) + assert tuple(field.name for field in fields(ReferenceCase)) == ( + "name", + "declaration", + "events", + "owner_case", + ) + + +def test_phase7_reference_manifest_freezes_exact_counts_serialization_and_digest() -> None: + frozen = json.loads(Path(__file__).with_name("phase7_reference_manifest.json").read_text(encoding="utf-8")) + generated = reference_manifest() + + assert generated == frozen + assert generated["reference_model_version"] == "phase7-reference-model/v1" + assert generated["generator_version"] == "phase7-finite-envelope/v1" + assert generated["graph_count"] == 54 + assert generated["case_count"] == 83 + assert generated["canonical_trace_count"] == 78 + assert generated["actual_event_count"] == 549 + assert generated["owner_case_count"] == 30 + assert generated["max_exogenous_observations"] == 16 + assert generated["digest"] == hashlib.sha256(canonical_corpus_bytes()).hexdigest() + assert not canonical_corpus_bytes().endswith(b"\n") + + +def test_corpus_serialization_is_invariant_to_case_and_commuting_schedule_order( + monkeypatch: pytest.MonkeyPatch, +) -> None: + model = importlib.import_module("tests.engine.execution.phase7_reference_model") + cases = finite_reference_cases() + baseline = canonical_corpus_bytes() + + monkeypatch.setattr(model, "finite_reference_cases", lambda: tuple(reversed(cases))) + assert canonical_corpus_bytes() == baseline + + target = next(case for case in cases if case.name == "independent-scopes-1-1") + first_dispatch, first_candidate, second_dispatch, second_candidate = target.events[:4] + reordered = replace( + target, + events=(second_dispatch, second_candidate, first_dispatch, first_candidate, *target.events[4:]), + ) + monkeypatch.setattr( + model, + "finite_reference_cases", + lambda: tuple(reordered if case.name == target.name else case for case in cases), + ) + assert canonical_corpus_bytes() == baseline diff --git a/tests/engine/execution/test_phase7_reference_mutations.py b/tests/engine/execution/test_phase7_reference_mutations.py new file mode 100644 index 00000000..612fb3ba --- /dev/null +++ b/tests/engine/execution/test_phase7_reference_mutations.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib.util +import sys +from dataclasses import asdict +from pathlib import Path +from types import ModuleType + +import pytest + +from tests.engine.execution.phase7_reference_model import case_by_name, reduce_reference + +_Mutation = tuple[str, tuple[tuple[str, str], ...], str] +_MUTATIONS: tuple[_Mutation, ...] = ( + ( + "representative-datum-accounting", + (("if not set(group).issubset(eligible):", "if group and group[0] not in eligible:"),), + "atomic-group-member-failure", + ), + ( + "label-remapping", + (('"first_name": "person_given_name",', '"first_name": "person_family_name",'),), + "future-slots-2", + ), + ( + "text-identity", + ( + ( + "structural_key = (scope_index, mention.cluster, role)", + "structural_key = (scope_index, mention.source, role)", + ), + ), + "equal-text-distinct-clusters", + ), + ( + "position-identity", + ( + ( + "structural_key = (scope_index, mention.cluster, role)", + 'structural_key = (scope_index, f"{mention.datum}:{mention.start}", role)', + ), + ), + "shared-slot-reuse", + ), + ( + "partial-acceptance", + ( + ("if set(keys) != expected:", "if set(keys) - expected:"), + ("value = assignment_by_slot[slot.key]", 'value = assignment_by_slot.get(slot.key, "Placeholder")'), + ), + "partial-candidate", + ), + ( + "distinct-slot-aliasing", + (("if canonical[left] == canonical[right]:", "if False and canonical[left] == canonical[right]:"),), + "owner-distinct_slots_same_canonical_value", + ), + ( + "cascading-application", + ( + ("for mention, slot_key in reversed(ordered):", "for mention, slot_key in ordered:"), + ( + """ if output[mention.start : mention.end] != mention.source: + return None + output = output[: mention.start] + assignment_by_slot[slot_key] + output[mention.end :] +""", + """ output = output.replace(mention.source, assignment_by_slot[slot_key]) +""", + ), + ), + "anchored-non-cascading-application", + ), + ( + "late-resurrection", + ( + ( + 'if state != "reserved" or not dispatched[scope_index] or event.attempt != attempts[scope_index]:', + 'if state not in {"reserved", "cancelled", "lost"} or not dispatched[scope_index] or event.attempt != attempts[scope_index]:', + ), + ), + "late-candidate-after-loss", + ), + ( + "skipped-cleanup", + ( + ( + 'global_inconsistent = global_inconsistent or cleanup != "verified"', + "global_inconsistent = global_inconsistent or False", + ), + ( + 'and cleanup == "verified"', + 'and cleanup in {"verified", "failed", "unconfirmed", "contradictory"}', + ), + ), + "cleanup-failure", + ), +) + + +@pytest.mark.parametrize( + ("name", "replacements", "witness"), _MUTATIONS, ids=lambda value: value if isinstance(value, str) else None +) +def test_frozen_phase7_corpus_kills_every_required_mutation( + name: str, + replacements: tuple[tuple[str, str], ...], + witness: str, + tmp_path: Path, +) -> None: + baseline = asdict(reduce_reference(case_by_name(witness))) + mutant = _load_mutant(name, replacements, tmp_path) + + observed = asdict(mutant.reduce_reference(mutant.case_by_name(witness))) + + assert observed != baseline, f"required Phase 7 mutation survived: {name}" + + +def _load_mutant(name: str, replacements: tuple[tuple[str, str], ...], tmp_path: Path) -> ModuleType: + source_path = Path(__file__).with_name("phase7_reference_model.py") + source = source_path.read_text(encoding="utf-8") + for original, replacement in replacements: + assert source.count(original) == 1, f"mutation seam drifted for {name}" + source = source.replace(original, replacement) + mutant_path = tmp_path / f"phase7_reference_model_{name.replace('-', '_')}.py" + mutant_path.write_text(source, encoding="utf-8") + module_name = f"tests.engine.execution._phase7_mutant_{name.replace('-', '_')}" + spec = importlib.util.spec_from_file_location(module_name, mutant_path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[module_name] = module + spec.loader.exec_module(module) + return module diff --git a/tests/engine/execution/test_phase7_runtime.py b/tests/engine/execution/test_phase7_runtime.py new file mode 100644 index 00000000..713a9a3b --- /dev/null +++ b/tests/engine/execution/test_phase7_runtime.py @@ -0,0 +1,359 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Focused lifecycle coverage for the private Phase 7 coordinator.""" + +from __future__ import annotations + +from dataclasses import dataclass, replace + +import pytest + +from anonymizer.engine.execution.accounting_outcomes import ( + _GroupReleased, + _GroupWithheld, + _TaskCancelled, + _TaskFailed, + _TaskLost, + _TaskSucceeded, +) +from anonymizer.engine.execution.accounting_plan import _DatumTaskSubject +from anonymizer.engine.execution.graph import _CoherenceScope +from anonymizer.engine.execution.phase7_admission import _Phase7Plan +from anonymizer.engine.execution.phase7_application import _AppliedDatum +from anonymizer.engine.execution.phase7_contract import _load_phase7_contract, _Phase7StableSubstituteContract +from anonymizer.engine.execution.phase7_ndd_backend import _Phase7NddResult, _Phase7NddStatus +from anonymizer.engine.execution.phase7_runtime import ( + _Phase7CleanupAttestation, + _Phase7Runtime, + _ScopePlanState, +) +from tests.engine.execution.test_phase7_admission import _compile_phase7, _Proposal, _qualified_phase6 + + +@dataclass +class _Backend: + calls: int = 0 + closed: int = 0 + discarded: int = 0 + attest_cleanup: bool = True + discard_fails: bool = False + stale_cleanup_identity: bool = False + trusted_stop_receipt: object | None = None + echo_dispatch_as_stop_receipt: bool = False + result: _Phase7NddStatus = _Phase7NddStatus.TASK_FAILED + + def propose_scope(self, manifest: object, handoffs: object, contract: object, dispatch: object) -> _Phase7NddResult: + del manifest, handoffs, contract + assert dispatch is not None + self.calls += 1 + return _Phase7NddResult( + self.result, + trusted_stop_receipt=dispatch if self.echo_dispatch_as_stop_receipt else self.trusted_stop_receipt, + ) + + def close(self) -> None: + self.closed += 1 + + def discard_values(self) -> None: + if self.discard_fails: + raise RuntimeError + self.discarded += 1 + + def cleanup_attestation(self, cleanup_identity: object) -> object: + if not self.attest_cleanup: + return None + identity = object() if self.stale_cleanup_identity else cleanup_identity + return _Phase7CleanupAttestation("phase7-cleanup-attestation/v1", True, 0, 0, True, 0, False, identity) + + +def test_empty_manifest_is_verified_no_work_then_cleaned_without_backend_dispatch() -> None: + phase6, _phase6_backend, execution = _qualified_phase6(("plain",), (("target-0",),), {}) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + backend = _Backend() + + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + result = _Phase7Runtime(backend).run(phase6, execution, plan, contract) + + assert tuple(outcome.state for outcome in result.scopes) == (_ScopePlanState.PLANNED,) + assert backend.calls == 0 + assert backend.closed == 1 + assert backend.discarded == 1 + assert result.cleanup.verified + assert result.cleanup.active_reservation_count == 0 + assert result.cleanup.provisional_bundle_reference_count == 0 + # The runtime consumes the exact compiler-issued Phase 4 task plan; it + # does not fabricate a detached scope-only accounting shell. + assert len(result.phase4.accounting.tasks) == len(plan.accounting.tasks) + assert result.phase4.accounting.tasks[:-2] == execution.accounting.tasks + assert not hasattr(result, "bundles") + planned, applied = result.phase4.accounting.tasks[-2:] + assert isinstance(planned, _TaskSucceeded) + assert not hasattr(planned.candidate, "assignments") + assert plan.scope_tasks == (planned.task,) + assert plan.application_tasks == (applied.task,) + assert isinstance(applied.task.subject, _DatumTaskSubject) + assert isinstance(applied, _TaskSucceeded) + assert isinstance(applied.candidate, _AppliedDatum) + assert applied.candidate.output == "plain" + assert result.released == (applied.candidate,) + released_group = result.phase4.accounting.groups[0] + assert isinstance(released_group, _GroupReleased) + assert released_group.outputs == ((applied.candidate.datum_id, applied.candidate),) + + +def test_application_exception_fails_only_its_owned_datum_and_still_cleans_up( + monkeypatch: pytest.MonkeyPatch, +) -> None: + phase6, _phase6_backend, execution = _qualified_phase6( + ("first", "second"), + (("target-0",), ("target-1",)), + {}, + ) + plan = _compile_phase7( + phase6, + execution, + (_CoherenceScope(tuple(datum.id for datum in phase6.accounting.datums)),), + ) + assert isinstance(plan, _Phase7Plan) + backend = _Backend() + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + + def apply_datum(_bundle: object, _patches: object, datum_id: object) -> _AppliedDatum: + if datum_id == plan.accounting.datums[0].id: + raise RuntimeError("private canary") + assert datum_id == plan.accounting.datums[1].id + return _AppliedDatum(plan.accounting.datums[1].id, "second", False) + + monkeypatch.setattr( + "anonymizer.engine.execution.phase7_runtime._apply_substitute_datum", + apply_datum, + raising=False, + ) + + result = _Phase7Runtime(backend).run(phase6, execution, plan, contract) + + first_application, second_application = ( + next(outcome for outcome in result.phase4.accounting.tasks if outcome.task == task) + for task in plan.application_tasks + ) + assert isinstance(first_application, _TaskFailed) + assert isinstance(second_application, _TaskSucceeded) + assert result.released == (second_application.candidate,) + assert isinstance(result.phase4.accounting.groups[0], _GroupWithheld) + assert isinstance(result.phase4.accounting.groups[1], _GroupReleased) + assert backend.closed == 1 + assert backend.discarded == 1 + assert result.cleanup.verified + + +def test_reconstructed_phase6_execution_is_rejected_before_any_planner_dispatch() -> None: + phase6, _phase6_backend, execution = _qualified_phase6( + ("Alice",), (("target-0",),), {"target-0": (_Proposal("Alice", "first_name", "person"),)} + ) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + backend = _Backend() + incomplete = type(execution)(execution.accounting, execution.released, ()) + + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + with pytest.raises(TypeError): + _Phase7Runtime(backend).run(phase6, incomplete, plan, contract) + assert backend.calls == 0 + + +def test_phase6_terminals_must_match_the_exact_compiler_expanded_prefix() -> None: + phase6, _phase6_backend, execution = _qualified_phase6(("plain",), (("target-0",),), {}) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + # This is still a sealed Phase 7-shaped value, but its Phase 4 expansion + # no longer carries the admitted Phase 6 prefix at its front. + malformed = replace(plan, accounting=replace(plan.accounting, tasks=plan.accounting.tasks[::-1])) + backend = _Backend() + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + + with pytest.raises(TypeError): + _Phase7Runtime(backend).run(phase6, execution, malformed, contract) + assert backend.calls == 0 + + +def test_missing_cleanup_evidence_embargoes_the_private_phase4_handoff() -> None: + phase6, _phase6_backend, execution = _qualified_phase6(("plain",), (("target-0",),), {}) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + backend = _Backend() + + backend.discard_fails = True + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + + result = _Phase7Runtime(backend).run(phase6, execution, plan, contract) + + assert not result.cleanup.verified + assert result.phase4.global_embargo + + +@pytest.mark.parametrize( + ("backend_status", "expected", "embargo"), + [ + (_Phase7NddStatus.ABORTED, _ScopePlanState.LOST, True), + (_Phase7NddStatus.POISONED, _ScopePlanState.LOST, True), + (_Phase7NddStatus.INVOCATION_INCONSISTENT, _ScopePlanState.INCONSISTENT, True), + ], +) +def test_terminal_backend_lifecycle_evidence_is_absorbing_and_embargoed( + backend_status: _Phase7NddStatus, expected: _ScopePlanState, embargo: bool +) -> None: + phase6, _phase6_backend, execution = _qualified_phase6( + ("Alice",), (("target-0",),), {"target-0": (_Proposal("Alice", "first_name", "person"),)} + ) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + backend = _Backend(result=backend_status) + + result = _Phase7Runtime(backend).run(phase6, execution, plan, contract) + + assert result.scopes[0].state is expected + assert backend.calls == 1 + assert result.phase4.global_embargo is embargo + + +def test_unverified_abort_after_dispatch_is_lost_without_stop_acknowledgement() -> None: + phase6, _phase6_backend, execution = _qualified_phase6( + ("Alice",), (("target-0",),), {"target-0": (_Proposal("Alice", "first_name", "person"),)} + ) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + result = _Phase7Runtime(_Backend(result=_Phase7NddStatus.ABORTED)).run(phase6, execution, plan, contract) + + lost = next(outcome for outcome in result.phase4.accounting.tasks if outcome.task == plan.scope_tasks[0]) + assert isinstance(lost, _TaskLost) + assert {cause.code.value for cause in lost.causes} == {"transport_lost"} + assert result.phase4.global_embargo + + +def test_backend_echoed_dispatch_is_not_trusted_stop_evidence() -> None: + phase6, _phase6_backend, execution = _qualified_phase6( + ("Alice",), (("target-0",),), {"target-0": (_Proposal("Alice", "first_name", "person"),)} + ) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + + result = _Phase7Runtime(_Backend(result=_Phase7NddStatus.ABORTED, echo_dispatch_as_stop_receipt=True)).run( + phase6, execution, plan, contract + ) + + assert result.scopes[0].state is _ScopePlanState.LOST + scope_outcome = next(outcome for outcome in result.phase4.accounting.tasks if outcome.task == plan.scope_tasks[0]) + assert isinstance(scope_outcome, _TaskLost) + assert result.phase4.global_embargo + + +def test_independently_verified_stop_receipt_acknowledges_cancellation() -> None: + phase6, _phase6_backend, execution = _qualified_phase6( + ("Alice",), (("target-0",),), {"target-0": (_Proposal("Alice", "first_name", "person"),)} + ) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + + receipt = object() + observed: list[object] = [] + + def verify_stop(candidate: object, dispatch: object) -> bool: + observed.extend((candidate, dispatch)) + return candidate is receipt and getattr(dispatch, "task", None) is plan.scope_tasks[0] + + result = _Phase7Runtime( + _Backend(result=_Phase7NddStatus.ABORTED, trusted_stop_receipt=receipt), + trusted_stop_receipt_verified=verify_stop, + ).run(phase6, execution, plan, contract) + + assert len(observed) == 2 + cancelled = next(outcome for outcome in result.phase4.accounting.tasks if outcome.task == plan.scope_tasks[0]) + assert isinstance(cancelled, _TaskCancelled) + assert {cause.code.value for cause in cancelled.causes} == {"cancellation", "stop_acknowledged"} + assert result.phase4.global_embargo + + +def test_cleanup_attestation_must_be_complete_and_verified() -> None: + phase6, _phase6_backend, execution = _qualified_phase6(("plain",), (("target-0",),), {}) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + backend = _Backend(attest_cleanup=False) + + result = _Phase7Runtime(backend).run(phase6, execution, plan, contract) + + assert not result.cleanup.verified + assert result.phase4.global_embargo + + +def test_cleanup_attestation_must_bind_the_current_finalization_identity() -> None: + phase6, _phase6_backend, execution = _qualified_phase6(("plain",), (("target-0",),), {}) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + + result = _Phase7Runtime(_Backend(stale_cleanup_identity=True)).run(phase6, execution, plan, contract) + + assert not result.cleanup.verified + assert result.phase4.global_embargo + + +def test_accepted_cancellation_before_dispatch_has_zero_planner_attempts() -> None: + phase6, _phase6_backend, execution = _qualified_phase6( + ("Alice",), (("target-0",),), {"target-0": (_Proposal("Alice", "first_name", "person"),)} + ) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + backend = _Backend() + + result = _Phase7Runtime(backend, cancellation_requested=lambda: True).run(phase6, execution, plan, contract) + + assert backend.calls == 0 + assert result.scopes[0].state is _ScopePlanState.CANCELLED + assert result.phase4.global_embargo + + +def test_accepted_cancellation_after_dispatch_without_verified_stop_is_lost() -> None: + phase6, _phase6_backend, execution = _qualified_phase6( + ("Alice",), (("target-0",),), {"target-0": (_Proposal("Alice", "first_name", "person"),)} + ) + plan = _compile_phase7(phase6, execution, phase6.coherence_scopes) + assert isinstance(plan, _Phase7Plan) + contract = _load_phase7_contract() + assert isinstance(contract, _Phase7StableSubstituteContract) + observed = [False] + + class _CancellingBackend(_Backend): + def propose_scope( + self, manifest: object, handoffs: object, contract: object, dispatch: object + ) -> _Phase7NddResult: + observed[0] = True + return super().propose_scope(manifest, handoffs, contract, dispatch) + + backend = _CancellingBackend() + result = _Phase7Runtime(backend, cancellation_requested=lambda: observed[0]).run(phase6, execution, plan, contract) + + assert backend.calls == 1 + assert result.scopes[0].state is _ScopePlanState.LOST + scope_outcome = next(outcome for outcome in result.phase4.accounting.tasks if outcome.task == plan.scope_tasks[0]) + assert isinstance(scope_outcome, _TaskLost) + assert result.phase4.global_embargo diff --git a/tests/engine/execution/test_phase7_validation.py b/tests/engine/execution/test_phase7_validation.py new file mode 100644 index 00000000..dda472be --- /dev/null +++ b/tests/engine/execution/test_phase7_validation.py @@ -0,0 +1,527 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import importlib +import importlib.util +import itertools +import pickle +from dataclasses import FrozenInstanceError, replace +from types import ModuleType +from typing import cast + +import pytest + +from anonymizer.engine.execution.graph import _CoherenceScope +from anonymizer.engine.execution.mention_resolution import _ClusterId +from anonymizer.engine.execution.phase6_runtime import _Phase6SubstituteHandoff +from anonymizer.engine.execution.phase7_admission import _Phase7Plan, _ScopeManifest +from anonymizer.engine.execution.phase7_contract import _load_phase7_contract +from anonymizer.engine.execution.phase7_validation import ( + _BundleRejected, + _CandidateAssignment, + _ValidatedBundle, +) +from tests.engine.execution.test_phase7_admission import ( + _compile_phase7, + _ids, + _person_relation_fixture, + _Proposal, + _qualified_phase6, +) + + +def _validation_module() -> ModuleType: + module_name = "anonymizer.engine.execution.phase7_validation" + assert importlib.util.find_spec(module_name) is not None, "the private Phase 7 validation module is missing" + return importlib.import_module(module_name) + + +def _compiled_scope( + texts: tuple[str, ...], + scopes: tuple[tuple[str, ...], ...], + proposals: dict[str, tuple[_Proposal, ...]], + *, + combined_scope: bool = False, +) -> tuple[_ScopeManifest, tuple[_Phase6SubstituteHandoff, ...]]: + plan, _backend, execution = _qualified_phase6(texts, scopes, proposals) + declared = (_CoherenceScope(_ids(plan)),) if combined_scope else plan.coherence_scopes + compiled = _compile_phase7(plan, execution, declared) + assert isinstance(compiled, _Phase7Plan) + assert len(compiled.manifests) == 1 + return compiled.manifests[0], execution.handoffs + + +def _assignments_for_roles(manifest: _ScopeManifest, values: dict[str, str]) -> tuple[_CandidateAssignment, ...]: + module = _validation_module() + assignment_type = getattr(module, "_CandidateAssignment") + slots = getattr(manifest, "slots") + return tuple(assignment_type(slot.id, values[slot.role]) for slot in slots) + + +def _validate( + manifest: _ScopeManifest, + handoffs: object, + assignments: object, +) -> _ValidatedBundle | _BundleRejected: + module = _validation_module() + validator = getattr(module, "_validate_scope_bundle", None) + assert callable(validator), "the private Phase 7 complete-bundle validator is missing" + return validator(manifest, handoffs, assignments, _load_phase7_contract()) + + +def _code(result: object) -> str: + code = getattr(result, "code", None) + assert code is not None, "malformed Phase 7 candidate did not return a typed rejection" + value = getattr(code, "value", None) + assert isinstance(value, str) + return value + + +def _validated_values(result: object) -> tuple[str, ...]: + assignments = getattr(result, "assignments", None) + assert isinstance(assignments, tuple) + return tuple(getattr(assignment, "value") for assignment in assignments) + + +def _all_permutations(values: tuple[_CandidateAssignment, ...]) -> tuple[tuple[_CandidateAssignment, ...], ...]: + return tuple(itertools.permutations(values)) + + +def test_phase7_canonicalizer_exactly_matches_the_p0_unicode_algorithm() -> None: + module = _validation_module() + canonicalize = getattr(module, "_canonicalize_value", None) + assert callable(canonicalize), "the single private Phase 7 canonicalizer is missing" + + assert canonicalize("\u2003 Ａ-lí_ce \u2003") == "alíce" + assert canonicalize("Straße") == "strasse" + assert canonicalize("É９") == "é9" + assert canonicalize("e\u0301") == "é" + assert canonicalize("-_ .") is None + assert canonicalize(None) is None + + +@pytest.mark.parametrize( + ("format_name", "value", "expected"), + [ + pytest.param("unicode_person_name/v1", "Élodie Marie-José.", True, id="person-unicode"), + pytest.param("unicode_person_name/v1", "A" * 128, True, id="person-128-bytes"), + pytest.param("unicode_person_name/v1", "A" * 129, False, id="person-129-bytes"), + pytest.param("unicode_person_name/v1", "é" * 64, True, id="person-128-multibyte-bytes"), + pytest.param("unicode_person_name/v1", "é" * 65, False, id="person-130-multibyte-bytes"), + pytest.param("unicode_person_name/v1", "A\u00a0B", True, id="person-zs"), + pytest.param("unicode_person_name/v1", "A\u2028B", False, id="person-zl"), + pytest.param("unicode_person_name/v1", "A\u2029B", False, id="person-zp"), + pytest.param("unicode_person_name/v1", "A_1", False, id="person-disallowed"), + pytest.param("username_ascii/v1", "a", True, id="username-min"), + pytest.param("username_ascii/v1", "a" * 64, True, id="username-max"), + pytest.param("username_ascii/v1", "a" * 65, False, id="username-over"), + pytest.param("username_ascii/v1", "_alice", False, id="username-left-boundary"), + pytest.param("username_ascii/v1", "alice-", False, id="username-right-boundary"), + pytest.param("username_ascii/v1", "álîce", False, id="username-nonascii"), + pytest.param("telephone_ascii/v1", "+1 (555) 010-0200", True, id="phone-valid"), + pytest.param("telephone_ascii/v1", "123456", False, id="phone-six-digits"), + pytest.param("telephone_ascii/v1", "1234567890123456", False, id="phone-sixteen-digits"), + pytest.param("telephone_ascii/v1", "1+234567", False, id="phone-nonleading-plus"), + pytest.param("telephone_ascii/v1", "+123+4567", False, id="phone-two-plus"), + pytest.param("telephone_ascii/v1", "123/4567", False, id="phone-invalid-character"), + pytest.param("email_addr_spec_ascii/v1", "a.b+tag@example-domain.com", True, id="email-valid"), + pytest.param("email_addr_spec_ascii/v1", ".alice@example.com", False, id="email-leading-dot"), + pytest.param("email_addr_spec_ascii/v1", "alice..x@example.com", False, id="email-double-dot"), + pytest.param("email_addr_spec_ascii/v1", "alice@-example.com", False, id="email-label-hyphen"), + pytest.param("email_addr_spec_ascii/v1", "alice@example.c", False, id="email-short-tld"), + pytest.param("email_addr_spec_ascii/v1", "álîce@example.com", False, id="email-nonascii"), + pytest.param("unknown/v1", "Alice", False, id="unknown-format"), + ], +) +def test_phase7_format_predicates_are_total_and_closed(format_name: str, value: object, expected: bool) -> None: + validator = getattr(_validation_module(), "_matches_format", None) + assert callable(validator), "the private Phase 7 format validator is missing" + + assert validator(format_name, value) is expected + + +def test_phase7_email_format_enforces_every_exact_byte_and_label_boundary() -> None: + validator = getattr(_validation_module(), "_matches_format", None) + assert callable(validator) + exact = f"{'a' * 64}@{'b' * 63}.{'c' * 63}.{'d' * 61}" + + assert len(exact.encode("utf-8")) == 254 + assert validator("email_addr_spec_ascii/v1", exact) + assert not validator("email_addr_spec_ascii/v1", f"{exact}e") + assert not validator("email_addr_spec_ascii/v1", f"{'a' * 65}@example.com") + assert not validator("email_addr_spec_ascii/v1", f"alice@{'b' * 64}.com") + assert validator("email_addr_spec_ascii/v1", f"alice@example.{'z' * 63}") + assert not validator("email_addr_spec_ascii/v1", f"alice@example.{'z' * 64}") + + +@pytest.mark.parametrize( + ("mask_name", "source", "candidate", "expected"), + [ + pytest.param("none/v1", "anything", "different", True, id="none"), + pytest.param("digit_literal/v1", "+１ (555)-0100", "+2 (212)-9999", True, id="nfkc-source"), + pytest.param("digit_literal/v1", "555-0100", "212-9999", True, id="changed-digits"), + pytest.param("digit_literal/v1", "555-0100", "212 9999", False, id="changed-literal"), + pytest.param("digit_literal/v1", "555-0100", "212-999", False, id="length"), + pytest.param("digit_literal/v1", "555-0100", "２12-9999", True, id="nfkc-candidate"), + pytest.param("unknown/v1", "555-0100", "212-9999", False, id="unknown-mask"), + ], +) +def test_phase7_source_masks_are_total_and_exact( + mask_name: str, + source: object, + candidate: object, + expected: bool, +) -> None: + validator = getattr(_validation_module(), "_matches_mask", None) + assert callable(validator), "the private Phase 7 source-mask validator is missing" + + assert validator(mask_name, source, candidate) is expected + + +def test_phase7_bundle_requires_every_exact_opaque_slot_token_once() -> None: + manifest, handoffs = _compiled_scope( + ("Alice Adams",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"), _Proposal("Adams", "last_name", "person"))}, + ) + module = _validation_module() + assignment_type = getattr(module, "_CandidateAssignment") + assignments = _assignments_for_roles( + manifest, + {"person_given_name": "Nova", "person_family_name": "Vale"}, + ) + + accepted = _validate(manifest, handoffs, assignments) + assert isinstance(accepted, getattr(module, "_ValidatedBundle")) + assert _validated_values(accepted) == ("Nova", "Vale") + assert _validated_values(_validate(manifest, handoffs, tuple(reversed(assignments)))) == ("Nova", "Vale") + assert _code(_validate(manifest, handoffs, assignments[:-1])) == "partial_bundle" + assert _code(_validate(manifest, handoffs, (*assignments, assignments[0]))) == "duplicate_slot" + duplicate_divergent = (*assignments, assignment_type(assignments[0].token, "Mira")) + assert _code(_validate(manifest, handoffs, duplicate_divergent)) == "duplicate_slot" + + foreign_manifest, _foreign_handoffs = _compiled_scope( + ("Mira",), + (("target-0",),), + {"target-0": (_Proposal("Mira", "first_name", "other"),)}, + ) + foreign = assignment_type(foreign_manifest.slots[0].id, "Tess") + assert _code(_validate(manifest, handoffs, (foreign, *assignments[1:]))) == "foreign_slot" + assert _code(_validate(manifest, handoffs, (*assignments, foreign))) == "foreign_slot" + + +@pytest.mark.parametrize( + "malformed", + [None, [], (), (object(),), (("not", "an assignment"),)], + ids=["none", "list", "empty", "object", "pair"], +) +def test_phase7_bundle_validation_is_total_for_malformed_untrusted_input(malformed: object) -> None: + manifest, handoffs = _compiled_scope( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ) + + result = _validate(manifest, handoffs, malformed) + + assert _code(result) in {"invalid_input", "partial_bundle"} + + +def test_phase7_bundle_validation_rejects_nontext_empty_and_unencodable_values_without_repair() -> None: + manifest, handoffs = _compiled_scope( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ) + assignment_type = getattr(_validation_module(), "_CandidateAssignment") + token = manifest.slots[0].id + + assert _code(_validate(manifest, handoffs, (assignment_type(token, 7),))) == "invalid_input" + assert _code(_validate(manifest, handoffs, (assignment_type(token, ""),))) == "candidate_matches_original" + assert _code(_validate(manifest, handoffs, (assignment_type(token, " -_. "),))) == "candidate_matches_original" + assert _code(_validate(manifest, handoffs, (assignment_type(token, "\ud800"),))) == "candidate_matches_original" + + +def test_phase7_candidate_must_differ_from_every_original_across_the_complete_scope() -> None: + manifest, handoffs = _compiled_scope( + ("Alice", "Bob"), + (("target-0",), ("target-1",)), + { + "target-0": (_Proposal("Alice", "first_name", "alice"),), + "target-1": (_Proposal("Bob", "first_name", "bob"),), + }, + combined_scope=True, + ) + module = _validation_module() + assignment_type = getattr(module, "_CandidateAssignment") + assignments = tuple( + assignment_type(slot.id, "Nova" if index == 0 else "A-l.i.c.e") for index, slot in enumerate(manifest.slots) + ) + + assert _code(_validate(manifest, handoffs, assignments)) == "candidate_matches_original" + + +def test_phase7_candidate_collision_checks_do_not_cross_scope_boundaries() -> None: + plan, _backend, execution = _qualified_phase6( + ("Alice", "Bob"), + (("target-0",), ("target-1",)), + { + "target-0": (_Proposal("Alice", "first_name", "alice"),), + "target-1": (_Proposal("Bob", "first_name", "bob"),), + }, + ) + compiled = _compile_phase7(plan, execution, plan.coherence_scopes) + assert isinstance(compiled, _Phase7Plan) + manifest = compiled.manifests[0] + assignment_type = getattr(_validation_module(), "_CandidateAssignment") + + accepted = _validate(manifest, execution.handoffs, (assignment_type(manifest.slots[0].id, "Bob"),)) + + assert isinstance(accepted, getattr(_validation_module(), "_ValidatedBundle")) + + +def test_phase7_every_compiled_required_pair_is_enforced() -> None: + manifest, handoffs = _compiled_scope( + ("Alice Adams Brenda Chloe",), + (("target-0",),), + { + "target-0": ( + _Proposal("Alice", "first_name", "a"), + _Proposal("Adams", "last_name", "a"), + _Proposal("Brenda", "first_name", "b"), + _Proposal("Chloe", "first_name", "c"), + ), + }, + ) + module = _validation_module() + assignment_type = getattr(module, "_CandidateAssignment") + baseline = ("Nova", "Vale", "Mira", "Tess") + positions = {slot.id: index for index, slot in enumerate(manifest.slots)} + + assert len(manifest.required_pairs) == 6 + for pair in manifest.required_pairs: + values = list(baseline) + values[positions[pair.left]] = "S-am" + values[positions[pair.right]] = "S.am" + assignments = tuple(assignment_type(slot.id, values[index]) for index, slot in enumerate(manifest.slots)) + assert _code(_validate(manifest, handoffs, assignments)) == "canonical_collision" + + +def test_phase7_shared_slot_accepts_one_value_and_rejects_divergent_duplicate_assignments() -> None: + manifest, handoffs = _compiled_scope( + ("Alice Alicia",), + (("target-0",),), + { + "target-0": ( + _Proposal("Alice", "first_name", "person"), + _Proposal("Alicia", "first_name", "person"), + ) + }, + ) + module = _validation_module() + assignment_type = getattr(module, "_CandidateAssignment") + token = manifest.slots[0].id + one = (assignment_type(token, "Nova"),) + + assert len(manifest.slots) == 1 + assert len(manifest.slots[0].mention_ids) == 2 + assert isinstance(_validate(manifest, handoffs, one), getattr(module, "_ValidatedBundle")) + assert _code(_validate(manifest, handoffs, (*one, assignment_type(token, "Nova")))) == "duplicate_slot" + assert _code(_validate(manifest, handoffs, (*one, assignment_type(token, "Vale")))) == "duplicate_slot" + + +def test_phase7_reused_slot_checks_the_source_mask_for_every_bound_mention() -> None: + manifest, handoffs = _compiled_scope( + ("555-0100 and (555)0100",), + (("target-0",),), + { + "target-0": ( + _Proposal("555-0100", "phone_number", "phone"), + _Proposal("(555)0100", "phone_number", "phone"), + ) + }, + ) + assignment = _assignments_for_roles(manifest, {"voice_phone_number": "212-9999"}) + + assert _code(_validate(manifest, handoffs, assignment)) == "relation_failed" + + +def test_phase7_email_relation_is_a_canonical_local_part_bundle_predicate() -> None: + plan, _backend, execution, cluster_value = _person_relation_fixture() + cluster = cast(_ClusterId, cluster_value) + admission = importlib.import_module("anonymizer.engine.execution.phase7_admission") + relation = admission._RelationDeclaration( + "email_from_name/v1", + ( + admission._ClusterRoleSelector("cluster_role/v1", cluster, "person_given_name"), + admission._ClusterRoleSelector("cluster_role/v1", cluster, "person_family_name"), + ), + admission._ClusterRoleSelector("cluster_role/v1", cluster, "email_address"), + ) + compiled = _compile_phase7(plan, execution, plan.coherence_scopes, (relation,)) + assert isinstance(compiled, _Phase7Plan) + manifest = compiled.manifests[0] + + accepted = _assignments_for_roles( + manifest, + { + "person_given_name": "Nova", + "person_family_name": "Vale", + "email_address": "v-ale@example.com", + }, + ) + domain_only = _assignments_for_roles( + manifest, + { + "person_given_name": "Nova", + "person_family_name": "Vale", + "email_address": "opaque@nova.example.com", + }, + ) + omitted = _assignments_for_roles( + manifest, + { + "person_given_name": "Nova", + "person_family_name": "Vale", + "email_address": "opaque@example.com", + }, + ) + + assert isinstance( + _validate(manifest, execution.handoffs, accepted), getattr(_validation_module(), "_ValidatedBundle") + ) + assert _code(_validate(manifest, execution.handoffs, domain_only)) == "relation_failed" + assert _code(_validate(manifest, execution.handoffs, omitted)) == "relation_failed" + + +def test_phase7_reachable_candidate_byte_limit_accepts_exact_and_rejects_one_over() -> None: + source = "1" * 15 + " " * 241 + exact = "2" * 15 + " " * 241 + manifest, handoffs = _compiled_scope( + (source,), + (("target-0",),), + {"target-0": (_Proposal(source, "phone_number", "phone"),)}, + ) + assignment_type = getattr(_validation_module(), "_CandidateAssignment") + token = manifest.slots[0].id + + assert len(exact.encode("utf-8")) == 256 + assert isinstance( + _validate(manifest, handoffs, (assignment_type(token, exact),)), + getattr(_validation_module(), "_ValidatedBundle"), + ) + assert _code(_validate(manifest, handoffs, (assignment_type(token, f"{exact} "),))) == "limit_exceeded" + + +def test_phase7_empty_manifest_accepts_only_the_complete_empty_bundle() -> None: + manifest, handoffs = _compiled_scope(("plain text",), (("target-0",),), {}) + module = _validation_module() + + accepted = _validate(manifest, handoffs, ()) + + assert isinstance(accepted, getattr(module, "_ValidatedBundle")) + assert accepted.assignments == () + assert _code(_validate(manifest, handoffs, None)) == "invalid_input" + + +def test_phase7_validation_rejects_stale_manifest_and_handoff_inputs() -> None: + manifest, handoffs = _compiled_scope( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ) + assignments = _assignments_for_roles(manifest, {"person_given_name": "Nova"}) + stale = replace(manifest, members=()) + + assert _code(_validate(stale, handoffs, assignments)) == "invalid_input" + assert _code(_validate(manifest, (), assignments)) == "invalid_input" + assert _code(_validate(manifest, (object(),), assignments)) == "invalid_input" + + +def test_phase7_validated_bundle_is_private_immutable_and_value_order_is_permutation_invariant() -> None: + manifest, handoffs = _compiled_scope( + ("Alice Adams alice@example.com",), + (("target-0",),), + { + "target-0": ( + _Proposal("Alice", "first_name", "person"), + _Proposal("Adams", "last_name", "person"), + _Proposal("alice@example.com", "email", "person"), + ) + }, + ) + assignments = _assignments_for_roles( + manifest, + { + "person_given_name": "Nova", + "person_family_name": "Vale", + "email_address": "nova@example.com", + }, + ) + baseline = _validate(manifest, handoffs, assignments) + + for permutation in _all_permutations(assignments): + assert _validated_values(_validate(manifest, handoffs, permutation)) == _validated_values(baseline) + with pytest.raises(FrozenInstanceError): + setattr(baseline, "assignments", ()) + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(baseline) + assert "Nova" not in repr(baseline) + + +def test_phase7_validated_bundle_recursively_rejects_nested_manifest_and_source_mutations() -> None: + manifest, handoffs = _compiled_scope( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ) + assignments = _assignments_for_roles(manifest, {"person_given_name": "Nova"}) + validated = _validate(manifest, handoffs, assignments) + module = _validation_module() + admitted = getattr(module, "_is_validated_bundle", None) + assert callable(admitted) + assert isinstance(validated, _ValidatedBundle) + assert admitted(validated) + + stale_manifest = replace(manifest, members=()) + assert not admitted(replace(validated, manifest=stale_manifest)) + + handoff = handoffs[0] + resolved = handoff.resolved + resolved_mention = resolved.mentions[0] + stale_mention = replace(resolved_mention.mention, source_slice="Mallory") + stale_resolved_mention = replace(resolved_mention, mention=stale_mention) + stale_resolved = replace(resolved, mentions=(stale_resolved_mention, *resolved.mentions[1:])) + stale_handoff = replace(handoff, resolved=stale_resolved) + stale_bundle = replace(validated, handoffs=(stale_handoff, *handoffs[1:])) + + assert not admitted(stale_bundle) + assert _code(_validate(manifest, (stale_handoff, *handoffs[1:]), assignments)) == "invalid_input" + + +def test_phase7_validation_has_no_planner_or_runtime_effects() -> None: + plan, backend, execution = _qualified_phase6( + ("Alice",), + (("target-0",),), + {"target-0": (_Proposal("Alice", "first_name", "person"),)}, + ) + compiled = _compile_phase7(plan, execution, plan.coherence_scopes) + assert isinstance(compiled, _Phase7Plan) + manifest = compiled.manifests[0] + assignments = _assignments_for_roles(manifest, {"person_given_name": "Nova"}) + calls_before = tuple(backend.calls) + + result = _validate(manifest, execution.handoffs, assignments) + + assert isinstance(result, getattr(_validation_module(), "_ValidatedBundle")) + assert tuple(backend.calls) == calls_before + assert backend.planner_effect_count == 0 + assert not { + "_AccountingLedger", + "_observe_context_boundary", + "_build_context_workframes", + "NddAdapter", + "DataDesigner", + } & set(vars(_validation_module())) diff --git a/tests/engine/execution/test_typed_task_accounting.py b/tests/engine/execution/test_typed_task_accounting.py new file mode 100644 index 00000000..8075bca7 --- /dev/null +++ b/tests/engine/execution/test_typed_task_accounting.py @@ -0,0 +1,439 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +from dataclasses import replace +from typing import cast + +import pytest + +from anonymizer.engine.execution.accounting_admission import _AccountingAdmissionCode, _compile_accounting_plan +from anonymizer.engine.execution.accounting_ledger import _AccountingLedger +from anonymizer.engine.execution.accounting_outcomes import ( + _DatumBlocked, + _DatumFailed, + _DatumQualified, + _GroupReleased, + _GroupWithheld, + _InvocationCancelled, + _InvocationCompleted, + _InvocationLost, + _StageBlocked, + _StageFailed, + _StageSucceeded, + _TaskBlocked, + _TaskCancelled, + _TaskFailed, + _TaskInconsistent, + _TaskLost, + _TaskSucceeded, +) +from anonymizer.engine.execution.accounting_plan import ( + _AccountingLimits, + _AccountingPlan, + _DatumTaskSubject, + _is_admitted_accounting_plan, + _ScopeTaskSubject, + _StageId, + _TaskKey, + _TaskPredecessor, +) +from anonymizer.engine.execution.graph import _DatumDependency, _DatumId, _TextDatum, _trivial_graph +from anonymizer.engine.execution.graph_runtime import ( + _AccountingGraphAdmissionError, + _AccountingGraphRuntime, + _FrameExecutionBackend, +) +from anonymizer.engine.execution.invocation import _CompiledInvocation +from tests.engine.execution.phase4_reference_model import ( + ReferenceMixedDeclaration, + ReferenceMixedTaskKey, + ReferenceTaskOutcome, + reduce_mixed_schedule, +) + + +def test_typed_task_accounting_test_infrastructure() -> None: + task = _TaskKey(_StageId("protect"), _DatumTaskSubject(_DatumId("fabricated-datum"))) + + assert task.stage.value == "protect" + + +def test_task_key_rejects_any_subject_outside_the_closed_sum() -> None: + with pytest.raises(TypeError, match="task subject"): + _TaskKey(_StageId("protect"), cast(_DatumTaskSubject, None)) + + +def test_accounting_admission_emits_only_datum_owned_tasks() -> None: + graph = _trivial_graph((_TextDatum(_DatumId("fabricated-datum"), "fabricated text"),)) + result = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=1, max_datum_bytes=64, max_graph_bytes=64), + ) + + assert isinstance(result, _AccountingPlan), "typed datum task admission is missing" + assert result.tasks == (_TaskKey(_StageId("protect"), _DatumTaskSubject(_DatumId("fabricated-datum"))),) + assert not hasattr(result.tasks[0], "datum_id") + + +def test_plan_appends_one_compiler_owned_task_for_each_datum_stage() -> None: + graph = _trivial_graph( + ( + _TextDatum(_DatumId("datum-0"), "first"), + _TextDatum(_DatumId("datum-1"), "second"), + ) + ) + plan = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=2, max_datum_bytes=64, max_graph_bytes=128, max_stages=2), + ) + assert isinstance(plan, _AccountingPlan) + + expanded = plan.with_datum_stage(_StageId("apply")) + + assert expanded.tasks == ( + *plan.tasks, + _TaskKey(_StageId("apply"), _DatumTaskSubject(_DatumId("datum-0"))), + _TaskKey(_StageId("apply"), _DatumTaskSubject(_DatumId("datum-1"))), + ) + assert expanded.stages == (*plan.stages, _StageId("apply")) + assert _is_admitted_accounting_plan(expanded) + with pytest.raises(TypeError, match="datum stage"): + expanded.with_datum_stage(_StageId("apply")) + + +def test_plan_adds_one_opaque_scope_task_for_each_declared_scope() -> None: + graph = _trivial_graph((_TextDatum(_DatumId("fabricated-datum"), "fabricated text"),)) + plan = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=1, max_datum_bytes=64, max_graph_bytes=64), + ) + assert isinstance(plan, _AccountingPlan) + first_scope = _ScopeTaskSubject() + zero_mention_scope = _ScopeTaskSubject() + mixed = plan.with_scope_tasks(_StageId("scope-plan"), (first_scope, zero_mention_scope)) + + scope_tasks = tuple(task for task in mixed.tasks if isinstance(task.subject, _ScopeTaskSubject)) + assert scope_tasks == ( + _TaskKey(_StageId("scope-plan"), first_scope), + _TaskKey(_StageId("scope-plan"), zero_mention_scope), + ) + assert first_scope is not zero_mention_scope + assert plan.with_scope_tasks(_StageId("scope-plan"), ()) is plan + + +def test_scope_task_capability_is_bound_into_the_plan_proof() -> None: + graph = _trivial_graph((_TextDatum(_DatumId("fabricated-datum"), "fabricated text"),)) + plan = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=1, max_datum_bytes=64, max_graph_bytes=64), + ) + assert isinstance(plan, _AccountingPlan) + mixed = plan.with_scope_tasks(_StageId("scope-plan"), (_ScopeTaskSubject(),)) + forged = replace( + mixed, + tasks=(*mixed.tasks[:-1], _TaskKey(_StageId("scope-plan"), _ScopeTaskSubject())), + ) + + assert _is_admitted_accounting_plan(mixed) + assert not _is_admitted_accounting_plan(forged) + + +def test_scope_task_is_ready_without_implicit_datum_edges_and_stays_out_of_datum_reduction() -> None: + graph = _trivial_graph((_TextDatum(_DatumId("fabricated-datum"), "fabricated text"),)) + plan = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=1, max_datum_bytes=64, max_graph_bytes=64), + ) + assert isinstance(plan, _AccountingPlan) + mixed = plan.with_scope_tasks(_StageId("scope-plan"), (_ScopeTaskSubject(),)) + ledger = _AccountingLedger[str](mixed, identity_factory=iter((f"opaque-{index}" for index in range(9))).__next__) + ledger.open() + try: + ready = ledger.ready_tasks() + except (AttributeError, KeyError, ValueError): + ready = () + + assert ready == mixed.tasks + datum_task, scope_task = ready + ledger.accept_success(ledger.dispatch(datum_task), "protected datum") + ledger.accept_success(ledger.dispatch(scope_task), "scope receipt") + result = ledger.finish() + + assert tuple(type(outcome) for outcome in result.tasks) == (_TaskSucceeded, _TaskSucceeded) + assert result.datums == (_DatumQualified(_DatumId("fabricated-datum"), "protected datum"),) + assert isinstance(result.groups[0], _GroupReleased) + assert result.groups[0].outputs == ((_DatumId("fabricated-datum"), "protected datum"),) + + +def test_scope_task_readiness_is_governed_only_by_explicit_predecessors() -> None: + graph = _trivial_graph((_TextDatum(_DatumId("fabricated-datum"), "fabricated text"),)) + plan = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=1, max_datum_bytes=64, max_graph_bytes=64), + ) + assert isinstance(plan, _AccountingPlan) + mixed = plan.with_scope_tasks(_StageId("scope-plan"), (_ScopeTaskSubject(),)) + datum_task, scope_task = mixed.tasks + mixed = mixed.with_task_predecessors((_TaskPredecessor(scope_task, datum_task),)) + ledger = _AccountingLedger[str](mixed, identity_factory=iter((f"opaque-{index}" for index in range(9))).__next__) + ledger.open() + + assert ledger.ready_tasks() == (scope_task,) + ledger.accept_failure(ledger.dispatch(scope_task)) + assert ledger.ready_tasks() == () + result = ledger.finish() + + assert isinstance(result.tasks[0], _TaskBlocked) + assert isinstance(result.tasks[1], _TaskFailed) + assert isinstance(result.datums[0], _DatumBlocked) + assert isinstance(result.groups[0], _GroupWithheld) + + +def test_scope_task_admission_rejects_duplicate_ownership_and_invalid_predecessors() -> None: + graph = _trivial_graph((_TextDatum(_DatumId("fabricated-datum"), "fabricated text"),)) + plan = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=1, max_datum_bytes=64, max_graph_bytes=64), + ) + assert isinstance(plan, _AccountingPlan) + scope = _ScopeTaskSubject() + + with pytest.raises(TypeError, match="scope task subjects"): + plan.with_scope_tasks(_StageId("scope-plan"), (scope, scope)) + + mixed = plan.with_scope_tasks(_StageId("scope-plan"), (scope,)) + with pytest.raises(TypeError, match="scope task subjects"): + mixed.with_scope_tasks(_StageId("scope-plan"), (scope,)) + + datum_task, scope_task = mixed.tasks + unknown = _TaskKey(_StageId("scope-plan"), _ScopeTaskSubject()) + invalid = ( + (_TaskPredecessor(scope_task, scope_task),), + (_TaskPredecessor(unknown, datum_task),), + (_TaskPredecessor(scope_task, datum_task), _TaskPredecessor(scope_task, datum_task)), + (_TaskPredecessor(scope_task, datum_task), _TaskPredecessor(datum_task, scope_task)), + ) + for predecessors in invalid: + with pytest.raises(TypeError, match="task predecessor"): + mixed.with_task_predecessors(predecessors) + + +@pytest.mark.parametrize( + ("terminal", "task_type", "group_type", "invocation_type"), + [ + pytest.param("failure", _TaskFailed, _GroupReleased, _InvocationCompleted, id="known-failure-is-local"), + pytest.param("cancellation", _TaskCancelled, _GroupWithheld, _InvocationCancelled, id="cancellation-embargoes"), + pytest.param("loss", _TaskLost, _GroupWithheld, _InvocationLost, id="loss-embargoes"), + pytest.param("missing", _TaskInconsistent, _GroupReleased, _InvocationCompleted, id="missing-is-local"), + ], +) +def test_scope_task_terminal_evidence_is_conserved_without_entering_datum_reduction( + terminal: str, + task_type: type[object], + group_type: type[object], + invocation_type: type[object], +) -> None: + graph = _trivial_graph((_TextDatum(_DatumId("fabricated-datum"), "fabricated text"),)) + plan = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=1, max_datum_bytes=64, max_graph_bytes=64), + ) + assert isinstance(plan, _AccountingPlan) + mixed = plan.with_scope_tasks(_StageId("scope-plan"), (_ScopeTaskSubject(),)) + datum_task, scope_task = mixed.tasks + ledger = _AccountingLedger[str]( + mixed, + identity_factory=iter((f"opaque-{index}" for index in range(12))).__next__, + ) + ledger.open() + ledger.accept_success(ledger.dispatch(datum_task), "protected datum") + if terminal == "failure": + ledger.accept_failure(ledger.dispatch(scope_task)) + elif terminal == "cancellation": + ledger.request_cancellation() + elif terminal == "loss": + ledger.mark_transport_lost(ledger.dispatch(scope_task)) + else: + dispatch = ledger.dispatch(scope_task) + ledger.reconcile((dispatch,), (), trusted_run_record=True) + + result = ledger.finish() + + assert len(result.tasks) == len(mixed.tasks) == 2 + assert isinstance(result.tasks[0], _TaskSucceeded) + assert isinstance(result.tasks[1], task_type) + assert result.datums == (_DatumQualified(_DatumId("fabricated-datum"), "protected datum"),) + assert isinstance(result.groups[0], group_type) + assert isinstance(result.invocation, invocation_type) + + +@pytest.mark.parametrize( + ("predecessor_direction", "scope_outcome"), + [ + pytest.param("scope-before-datum", ReferenceTaskOutcome.SUCCEEDED, id="scope-predecessor-succeeds"), + pytest.param("scope-before-datum", ReferenceTaskOutcome.FAILED, id="scope-predecessor-fails"), + pytest.param("datum-before-scope", ReferenceTaskOutcome.FAILED, id="scope-failure-does-not-block-release"), + ], +) +def test_mixed_plan_matches_independent_reference_model( + predecessor_direction: str, + scope_outcome: ReferenceTaskOutcome, +) -> None: + graph = _trivial_graph( + ( + _TextDatum(_DatumId("a"), "fabricated a"), + _TextDatum(_DatumId("b"), "fabricated b"), + ) + ) + graph = replace(graph, dependencies=(_DatumDependency(_DatumId("a"), _DatumId("b")),)) + plan = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=2, max_datum_bytes=64, max_graph_bytes=64), + ) + assert isinstance(plan, _AccountingPlan) + scope_subject = _ScopeTaskSubject() + mixed = plan.with_scope_tasks(_StageId("scope-plan"), (scope_subject,)) + datum_a, datum_b, scope = mixed.tasks + predecessor = ( + _TaskPredecessor(scope, datum_a) + if predecessor_direction == "scope-before-datum" + else _TaskPredecessor(datum_a, scope) + ) + mixed = mixed.with_task_predecessors((predecessor,)) + scope_key: ReferenceMixedTaskKey = ("scope-plan", ("scope", "scope-zero-mentions")) + datum_a_key: ReferenceMixedTaskKey = ("protect", ("datum", "a")) + datum_b_key: ReferenceMixedTaskKey = ("protect", ("datum", "b")) + reference_predecessor = ( + (scope_key, datum_a_key) if predecessor_direction == "scope-before-datum" else (datum_a_key, scope_key) + ) + schedule = ( + ((scope_key, scope_outcome),) + if predecessor_direction == "scope-before-datum" and scope_outcome is ReferenceTaskOutcome.FAILED + else ( + (scope_key, ReferenceTaskOutcome.SUCCEEDED), + (datum_a_key, ReferenceTaskOutcome.SUCCEEDED), + (datum_b_key, ReferenceTaskOutcome.SUCCEEDED), + ) + if predecessor_direction == "scope-before-datum" + else ( + (datum_a_key, ReferenceTaskOutcome.SUCCEEDED), + (scope_key, scope_outcome), + (datum_b_key, ReferenceTaskOutcome.SUCCEEDED), + ) + ) + reference = reduce_mixed_schedule( + ReferenceMixedDeclaration( + datum_ids=("a", "b"), + datum_stages=("protect",), + scope_tasks=(("scope-plan", "scope-zero-mentions"),), + datum_dependencies=(("a", "b"),), + task_predecessors=(reference_predecessor,), + atomic_groups=(("a",), ("b",)), + ), + schedule, + ) + production_by_reference = { + datum_a_key: datum_a, + datum_b_key: datum_b, + scope_key: scope, + } + ledger = _AccountingLedger[str]( + mixed, + identity_factory=iter((f"opaque-{index}" for index in range(24))).__next__, + ) + ledger.open() + observed_frontiers = [] + for reference_task, outcome in schedule: + observed_frontiers.append(tuple(_as_reference_key(task, scope_subject) for task in ledger.ready_tasks())) + dispatch = ledger.dispatch(production_by_reference[reference_task]) + if outcome is ReferenceTaskOutcome.SUCCEEDED: + ledger.accept_success(dispatch, f"candidate-{len(observed_frontiers)}") + else: + ledger.accept_failure(dispatch) + result = ledger.finish() + + assert tuple(observed_frontiers) == reference.ready_frontiers + assert ( + tuple((_as_reference_key(outcome.task, scope_subject), _task_outcome(outcome)) for outcome in result.tasks) + == reference.tasks + ) + assert tuple((outcome.datum_id.value, _datum_outcome(outcome)) for outcome in result.datums) == reference.datums + assert tuple((outcome.stage.value, _stage_outcome(outcome)) for outcome in result.stages) == reference.stages + assert ( + frozenset( + frozenset(datum_id.value for datum_id, _candidate in group.outputs) + for group in result.groups + if isinstance(group, _GroupReleased) + ) + == reference.released_groups + ) + + +def test_datum_only_graph_runtime_rejects_mixed_plan_before_backend_effects() -> None: + graph = _trivial_graph((_TextDatum(_DatumId("fabricated-datum"), "fabricated text"),)) + plan = _compile_accounting_plan( + graph, + limits=_AccountingLimits(max_datums=1, max_datum_bytes=64, max_graph_bytes=64), + ) + assert isinstance(plan, _AccountingPlan) + mixed = plan.with_scope_tasks(_StageId("scope-plan"), (_ScopeTaskSubject(),)) + effects: list[str] = [] + + class _Backend: + def run(self, *_args: object, **_kwargs: object) -> object: + effects.append("run") + raise AssertionError("backend must not run") + + runtime = _AccountingGraphRuntime(cast(_FrameExecutionBackend, _Backend())) + with pytest.raises(_AccountingGraphAdmissionError) as raised: + runtime.run( + mixed, + invocation=cast(_CompiledInvocation, object()), + data_summary=None, + preview_num_records=None, + hydrate=lambda _datum, _row: "unused", + ) + + assert raised.value.code is _AccountingAdmissionCode.UNSUPPORTED_TASK_CARDINALITY + assert effects == [] + + +def _as_reference_key(task: _TaskKey, scope_subject: _ScopeTaskSubject) -> ReferenceMixedTaskKey: + if isinstance(task.subject, _DatumTaskSubject): + subject = ("datum", task.subject.datum_id.value) + else: + assert task.subject is scope_subject + subject = ("scope", "scope-zero-mentions") + return task.stage.value, subject + + +def _task_outcome(outcome: object) -> ReferenceTaskOutcome: + if isinstance(outcome, _TaskSucceeded): + return ReferenceTaskOutcome.SUCCEEDED + if isinstance(outcome, _TaskFailed): + return ReferenceTaskOutcome.FAILED + if isinstance(outcome, _TaskBlocked): + return ReferenceTaskOutcome.BLOCKED + raise AssertionError("unexpected task outcome in mixed differential") + + +def _datum_outcome(outcome: object) -> ReferenceTaskOutcome: + if isinstance(outcome, _DatumQualified): + return ReferenceTaskOutcome.SUCCEEDED + if isinstance(outcome, _DatumFailed): + return ReferenceTaskOutcome.FAILED + if isinstance(outcome, _DatumBlocked): + return ReferenceTaskOutcome.BLOCKED + raise AssertionError("unexpected datum outcome in mixed differential") + + +def _stage_outcome(outcome: object) -> ReferenceTaskOutcome: + if isinstance(outcome, _StageSucceeded): + return ReferenceTaskOutcome.SUCCEEDED + if isinstance(outcome, _StageFailed): + return ReferenceTaskOutcome.FAILED + if isinstance(outcome, _StageBlocked): + return ReferenceTaskOutcome.BLOCKED + raise AssertionError("unexpected stage outcome in mixed differential") diff --git a/tests/engine/test_llm_replace_workflow.py b/tests/engine/test_llm_replace_workflow.py index f6ae372b..7ad75661 100644 --- a/tests/engine/test_llm_replace_workflow.py +++ b/tests/engine/test_llm_replace_workflow.py @@ -167,6 +167,7 @@ def test_generate_map_only_preserves_original_anonymizer_row_order_with_mixed_ro input_df, model_configs=stub_model_configs, selected_models=stub_replace_model_selection, + preview_num_records=3, ) assert result.dataframe[COL_TEXT].tolist() == ["No entities here", "Alice works at Acme", "Still no entities"] @@ -175,6 +176,7 @@ def test_generate_map_only_preserves_original_anonymizer_row_order_with_mixed_ro {"replacements": [{"original": "Alice", "label": "first_name", "synthetic": "Maya"}]}, {"replacements": []}, ] + assert adapter.run_workflow.call_args.kwargs["preview_num_records"] == 1 def test_generate_map_only_strips_internal_prompt_columns( diff --git a/tests/engine/test_ndd_adapter.py b/tests/engine/test_ndd_adapter.py index 8e92a138..25f2d919 100644 --- a/tests/engine/test_ndd_adapter.py +++ b/tests/engine/test_ndd_adapter.py @@ -4,6 +4,8 @@ from __future__ import annotations import logging +import pickle +from pathlib import Path from types import SimpleNamespace from typing import cast from unittest.mock import Mock @@ -13,11 +15,13 @@ from data_designer.config.column_configs import LLMTextColumnConfig from data_designer.config.column_types import ColumnConfigT from data_designer.config.models import ModelConfig +from data_designer.config.run_config import RunConfig from data_designer.interface.data_designer import DataDesigner from anonymizer.engine.ndd import adapter as ndd_adapter from anonymizer.engine.ndd.adapter import RECORD_ID_COLUMN, NddAdapter from anonymizer.interface.errors import AnonymizerWorkflowError +from anonymizer.measurement import MeasurementCollector, measurement_session _FORBIDDEN_BACKEND_STRINGS = ("Data Designer", "DataDesigner", "data_designer", "DD") @@ -190,7 +194,7 @@ def preview(self, _config_builder: object, *, num_records: int) -> SimpleNamespa adapter = NddAdapter(data_designer=cast(DataDesigner, UsageDataDesigner())) - with pytest.raises(AnonymizerWorkflowError, match="boom"): + with pytest.raises(AnonymizerWorkflowError, match="Workflow failed"): adapter.run_workflow( input_df, model_configs=[_make_model_config()], @@ -202,6 +206,58 @@ def preview(self, _config_builder: object, *, num_records: int) -> SimpleNamespa assert adapter.total_input_tokens == 7 +def test_private_execution_uses_ephemeral_artifacts_and_suppresses_active_collector(tmp_path: Path) -> None: + private_canary = "PRIVATE-CORRELATION-CANARY" + raw_canary = "RAW-ROW-alice@example.test" + durable_root = tmp_path / "durable" + durable_root.mkdir() + + class PrivateDataDesigner: + _artifact_path = durable_root + run_config = RunConfig(async_trace=True) + + def set_run_config(self, run_config: RunConfig) -> None: + self.run_config = run_config + + def create(self, _builder: object, **kwargs: object) -> SimpleNamespace: + assert self.run_config.async_trace is False + artifact_root = Path(cast(str, kwargs.get("artifact_path", self._artifact_path))) + assert artifact_root != self._artifact_path + artifact = artifact_root / "captured.txt" + artifact.write_text(f"{private_canary}\n{raw_canary}") + output = pd.DataFrame( + { + "text": [raw_canary], + "__anonymizer_private_row_correlation__": [private_canary], + RECORD_ID_COLUMN: ["record-a"], + "output": ["protected"], + } + ) + return SimpleNamespace(load_dataset=lambda: output, task_traces=[]) + + data_designer = PrivateDataDesigner() + adapter = NddAdapter(data_designer=cast(DataDesigner, data_designer)) + collector = MeasurementCollector() + with measurement_session(collector), adapter.private_execution(): + result = adapter.run_workflow( + pd.DataFrame( + { + "text": [raw_canary], + "__anonymizer_private_row_correlation__": [private_canary], + RECORD_ID_COLUMN: ["record-a"], + } + ), + model_configs=[_make_model_config()], + columns=_make_columns(), + workflow_name="private-protection", + ) + + assert result.dataframe.iloc[0]["output"] == "protected" + assert data_designer.run_config.async_trace is True + assert not any(durable_root.rglob("*")) + assert collector.records == [] + + def test_detect_missing_records_returns_missing_ids() -> None: adapter = NddAdapter(data_designer=Mock(spec=DataDesigner)) input_df = adapter._attach_record_ids(pd.DataFrame({"text": ["a", "b", "c"]})) @@ -217,6 +273,39 @@ def test_detect_missing_records_returns_missing_ids() -> None: assert failed_records[0].step == "replace-workflow" +def test_run_workflow_attributes_missing_rows_only_by_private_correlation() -> None: + input_df = pd.DataFrame( + { + "text": ["same", "same"], + "__anonymizer_private_row_correlation__": ["opaque-a", "opaque-b"], + RECORD_ID_COLUMN: ["public-a", "public-b"], + }, + index=pd.Index([None, None]), + ) + + class DroppingDataDesigner: + def create(self, _builder: object, **_kwargs: object) -> SimpleNamespace: + output = input_df.iloc[[1]].copy() + return SimpleNamespace(load_dataset=lambda: output, task_traces=[]) + + adapter = NddAdapter(data_designer=cast(DataDesigner, DroppingDataDesigner())) + + with adapter.private_execution(): + result = adapter.run_workflow( + input_df, + model_configs=[_make_model_config()], + columns=_make_columns(), + workflow_name="replace-workflow", + ) + + assert [record.record_id for record in result.failed_records] == ["public-a"] + assert result.failed_row_tokens == ("opaque-a",) + assert "opaque-a" not in repr(result) + assert "opaque-a" not in repr(result.failed_row_evidence[0]) + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(result.failed_row_evidence[0]) + + def test_detect_missing_records_for_preview_subset_has_no_false_failures() -> None: adapter = NddAdapter(data_designer=Mock(spec=DataDesigner)) full_input_df = adapter._attach_record_ids(pd.DataFrame({"text": ["a", "b", "c"]})) @@ -255,25 +344,21 @@ class DataDesignerRuntimeError(Exception): preview_num_records=3, ) - assert isinstance(exc_info.value.__cause__, DataDesignerRuntimeError) - assert "endpoint unreachable" in str(exc_info.value) + assert exc_info.value.__cause__ is None + assert exc_info.value.__context__ is None + assert str(exc_info.value) == "Workflow failed" warning_records = _unique_records(caplog, level=logging.WARNING, message_contains="Workflow failed") assert len(warning_records) == 1 warning_msg = warning_records[0].getMessage() assert "3" in warning_msg - assert "test-model-alias" in warning_msg - assert "endpoint unreachable" in warning_msg + assert "test-model-alias" not in warning_msg + assert "endpoint unreachable" not in warning_msg assert "Check endpoint reachability" not in warning_msg assert "detect-workflow" not in warning_msg _assert_no_backend_reference(warning_msg) - debug_records = _unique_records(caplog, level=logging.DEBUG, message_contains="failure context") - assert len(debug_records) == 1 - debug_msg = debug_records[0].getMessage() - assert "detect-workflow" in debug_msg - assert "output" in debug_msg - _assert_no_backend_reference(debug_msg) + assert not _unique_records(caplog, level=logging.DEBUG, message_contains="failure context") def test_create_exception_wraps_in_workflow_error_and_logs( @@ -299,25 +384,21 @@ class DataDesignerRuntimeError(Exception): preview_num_records=None, ) - assert isinstance(exc_info.value.__cause__, DataDesignerRuntimeError) - assert "quota exceeded" in str(exc_info.value) + assert exc_info.value.__cause__ is None + assert exc_info.value.__context__ is None + assert str(exc_info.value) == "Workflow failed" warning_records = _unique_records(caplog, level=logging.WARNING, message_contains="Workflow failed") assert len(warning_records) == 1 warning_msg = warning_records[0].getMessage() assert "2" in warning_msg - assert "test-model-alias" in warning_msg - assert "quota exceeded" in warning_msg + assert "test-model-alias" not in warning_msg + assert "quota exceeded" not in warning_msg assert "Check endpoint reachability" not in warning_msg assert "replace-workflow" not in warning_msg _assert_no_backend_reference(warning_msg) - debug_records = _unique_records(caplog, level=logging.DEBUG, message_contains="failure context") - assert len(debug_records) == 1 - debug_msg = debug_records[0].getMessage() - assert "replace-workflow" in debug_msg - assert "output" in debug_msg - _assert_no_backend_reference(debug_msg) + assert not _unique_records(caplog, level=logging.DEBUG, message_contains="failure context") def test_load_dataset_exception_wraps_in_workflow_error_and_logs( @@ -345,26 +426,22 @@ class DataDesignerRuntimeError(Exception): preview_num_records=None, ) - assert isinstance(exc_info.value.__cause__, DataDesignerRuntimeError) - assert "corrupt parquet" in str(exc_info.value) + assert exc_info.value.__cause__ is None + assert exc_info.value.__context__ is None + assert str(exc_info.value) == "Workflow failed" warning_records = _unique_records(caplog, level=logging.WARNING, message_contains="Workflow failed") assert len(warning_records) == 1 warning_msg = warning_records[0].getMessage() assert "2" in warning_msg - assert "test-model-alias" in warning_msg - assert "corrupt parquet" in warning_msg + assert "test-model-alias" not in warning_msg + assert "corrupt parquet" not in warning_msg assert "Check local storage" not in warning_msg assert "Check endpoint reachability" not in warning_msg assert "replace-workflow" not in warning_msg _assert_no_backend_reference(warning_msg) - debug_records = _unique_records(caplog, level=logging.DEBUG, message_contains="failure context") - assert len(debug_records) == 1 - debug_msg = debug_records[0].getMessage() - assert "replace-workflow" in debug_msg - assert "output" in debug_msg - _assert_no_backend_reference(debug_msg) + assert not _unique_records(caplog, level=logging.DEBUG, message_contains="failure context") def test_detect_missing_records_short_circuit_warns_when_input_missing_id( diff --git a/tests/engine/test_rewrite_workflow.py b/tests/engine/test_rewrite_workflow.py index 5c5bd84e..44cb2630 100644 --- a/tests/engine/test_rewrite_workflow.py +++ b/tests/engine/test_rewrite_workflow.py @@ -33,6 +33,7 @@ COL_WEIGHTED_LEAKAGE_RATE, ) from anonymizer.engine.ndd.adapter import RECORD_ID_COLUMN, FailedRecord, WorkflowRunResult +from anonymizer.engine.replace.llm_replace_workflow import LlmReplaceResult from anonymizer.engine.rewrite.rewrite_workflow import RewriteWorkflow, _detection_valid_fraction _REPLACE_PATCH = "anonymizer.engine.rewrite.rewrite_workflow.LlmReplaceWorkflow" @@ -164,7 +165,7 @@ def _standard_side_effect( def _mock_replace(mock_cls: Mock, replace_df: pd.DataFrame) -> None: """Configure a patched LlmReplaceWorkflow class to return replace_df.""" - mock_cls.return_value.generate_map_only.return_value = Mock(dataframe=replace_df, failed_records=[]) + mock_cls.return_value.generate_map_only.return_value = LlmReplaceResult(dataframe=replace_df, failed_records=[]) # --------------------------------------------------------------------------- @@ -303,7 +304,7 @@ def test_failed_records_accumulated_across_steps( ] with patch(_REPLACE_PATCH) as mock_replace_cls: - mock_replace_cls.return_value.generate_map_only.return_value = Mock( + mock_replace_cls.return_value.generate_map_only.return_value = LlmReplaceResult( dataframe=stub_replace_df, failed_records=[FailedRecord(record_id="d", step="replace-map-generation", reason="timeout")], ) diff --git a/tests/engine/test_workflow_utils.py b/tests/engine/test_workflow_utils.py index 10619de8..2cdcb8f7 100644 --- a/tests/engine/test_workflow_utils.py +++ b/tests/engine/test_workflow_utils.py @@ -11,6 +11,7 @@ from pydantic import BaseModel from anonymizer.engine.ndd.adapter import RECORD_ID_COLUMN +from anonymizer.engine.private_row_verification import PRIVATE_CORRELATION_COLUMN from anonymizer.engine.rewrite.workflow_utils import derive_seed_columns, select_seed_cols @@ -71,6 +72,20 @@ def test_derive_always_includes_record_id() -> None: assert RECORD_ID_COLUMN in seed +def test_derive_preserves_private_runtime_correlation() -> None: + columns = [ + LLMStructuredColumnConfig( + name="_out", + prompt="No column refs", + model_alias="test", + output_format=_StubOutput, + ), + ] + df = pd.DataFrame({RECORD_ID_COLUMN: ["r1"], PRIVATE_CORRELATION_COLUMN: ["opaque"]}) + + assert derive_seed_columns(columns, df) == [RECORD_ID_COLUMN, PRIVATE_CORRELATION_COLUMN] + + def test_derive_ignores_columns_missing_from_df() -> None: """Required columns not present in the dataframe are silently skipped.""" diff --git a/tests/fixtures/streaming/POLICY.md b/tests/fixtures/streaming/POLICY.md new file mode 100644 index 00000000..28eb6cda --- /dev/null +++ b/tests/fixtures/streaming/POLICY.md @@ -0,0 +1,83 @@ + + + +# Synthetic streaming fixtures + +Fixtures in this directory are synthetic test data only. Do not commit customer +data, private captures, production corpus excerpts, credentials, or provider +responses. Provider-backed characterization requires separate written approval. + +Streaming tests and the internal characterization runner use local `Redact`, +`Annotate`, or `Hash` strategies only. Reports must be aggregate and +privacy-safe: they may contain counts, byte totals, durations, and boolean +outcomes, but never source content, protected content, entity values, prompts, +provider text, engine identifiers, or per-row traces. + +## Intake validation corpus + +The `intake_*` fixtures are synthetic validation probes, not captured Intake +traffic and not claims of production format support. They exercise only the +closed fields declared by the test adapter. Unknown content-bearing fields fail +closed. + +Their shapes are derived from immutable NeMo Platform sources: + +- [Intake ingest-format reference](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/references/ingest-formats.md) +- [ATIF domain and validation rules](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/ingest/atif_domain.py) +- [Chat-completion ingest model](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/ingest/chat_completions.py) +- [OTLP/HTTP protobuf receiver](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/src/nmp/intake/spans/ingest/otlp.py) +- [Local CHAIN-to-LLM OTLP example](https://github.com/NVIDIA-NeMo/nemo-platform/blob/e1057736703bb8b167a4bd9013cea0caae2df63a/services/intake/examples/send_otel_sample.py) + +OTLP validation uses real `ExportTraceServiceRequest` protobuf bytes. The probe +withholds the complete batch if any span is invalid or any selected segment +does not produce a successful Plan A outcome. This is stricter than Intake's +current HTTP-200 response with per-span errors and is a proposed adapter policy, +not established Intake behavior. + +`tests/streaming/test_intake_dogfood.py` is an opt-in integration test for an +operator-owned Intake deployment. Set `ANONYMIZER_INTAKE_DOGFOOD_BASE_URL` to +the service origin to enable it. Its default before-Intake profile sends only +protected requests and checks that every declared synthetic PII value is absent +from the outbound bytes. Raw-format and partial-write characterization requires +the additional explicit `ANONYMIZER_INTAKE_DOGFOOD_ALLOW_RAW=1` opt-in and is +valid only for an isolated deployment approved to receive raw synthetic data. +The protected-only profile also wires invalid OTLP to a real Intake emitter and +checks that adapter rejection prevents the emitter call and leaves no stored +session. + +The protected-delivery probes keep retry ownership outside Anonymizer. A +pre-connect failure raises a bounded, cause-free test error; it does not alter +the protected bytes or create an Intake row. The probe then sends that same +protected byte string successfully and verifies one stable public row. A +commit-then-forget approximation separately resends exact protected bytes to +characterize an ambiguous delivery. +Intake's public read model collapses repeated ATIF, chat-completion, and OTLP +records for the fixed fixture identities. The chat contract requires a +positive, representable, non-future integer `response.created` and preserves it +unchanged because Intake uses that value as the span start time; if Intake +cannot accept it, Intake assigns a new ingestion time and exposes an exact +retry as another row. This does not establish +transactional idempotency or describe the number of physical ClickHouse writes. +A source adapter must retain the exact protected bytes for retry and must reject +a chat item that lacks a stable creation time. + +The test owns only its synthetic requests and unique identifiers: it does not +provision, configure, stop, or clean up Intake, ClickHouse, containers, or data. +Its persisted synthetic rows remain under the operator's retention policy. + +## Sandbox session export + +`tests/streaming/sandbox_session_export.py` is a test-only adapter for a +completed, successful Sandbox Codex run. It reads the run's copied prompt and +Codex JSONL output, maps the reviewed message, shell-command, and file-change +items to ATIF v1.0, and fails closed on other completed item types. Raw session +artifacts remain on the producer side of the provisional before-Intake boundary. + +Set `ANONYMIZER_SANDBOX_DOGFOOD_RUN_DIR` with +`ANONYMIZER_INTAKE_DOGFOOD_BASE_URL` to exercise the opt-in live path. The test +consumes an operator-owned run created from the synthetic +`sandbox_agent_prompt.md` fixture; it does not launch, configure, stop, or +remove a Sandbox session. Only protected ATIF bytes cross into Intake. The +current test uses declared synthetic PII and deterministic local detection, so +it validates the execution and data boundary but does not establish +provider-backed detection quality or production Sandbox support. diff --git a/tests/fixtures/streaming/complete_trace.json b/tests/fixtures/streaming/complete_trace.json new file mode 100644 index 00000000..7992d15c --- /dev/null +++ b/tests/fixtures/streaming/complete_trace.json @@ -0,0 +1,49 @@ +{ + "schema_version": "synthetic-agent-trace/v1", + "trace_id": "trace-2026-0001", + "metadata": { + "environment": "test", + "retention_class": "synthetic-only", + "sequence": 17 + }, + "messages": [ + { + "id": "msg-001", + "parent_id": null, + "role": "user", + "sequence": 0, + "content": "Email alice@example.test about the synthetic account." + }, + { + "id": "msg-002", + "parent_id": "msg-001", + "role": "assistant", + "sequence": 1, + "content": "I will query the synthetic account.", + "tool_calls": [ + { + "id": "call-001", + "type": "function", + "name": "lookup_customer", + "arguments": { + "customer_email": "alice@example.test", + "case_note": "The synthetic account belongs to Alice Example." + } + } + ] + }, + { + "id": "msg-003", + "parent_id": "msg-002", + "role": "tool", + "sequence": 2, + "tool_call_id": "call-001", + "name": "lookup_customer", + "tool_result": { + "status": "ok", + "content": "Customer alice@example.test has phone +1-555-0100.", + "account_email": "alice@example.test" + } + } + ] +} diff --git a/tests/fixtures/streaming/complete_trace.jsonl b/tests/fixtures/streaming/complete_trace.jsonl new file mode 100644 index 00000000..e62e2b33 --- /dev/null +++ b/tests/fixtures/streaming/complete_trace.jsonl @@ -0,0 +1 @@ +{"schema_version":"synthetic-agent-trace/v1","trace_id":"trace-2026-0001","metadata":{"environment":"test","retention_class":"synthetic-only","sequence":17},"messages":[{"id":"msg-001","parent_id":null,"role":"user","sequence":0,"content":"Email alice@example.test about the synthetic account."},{"id":"msg-002","parent_id":"msg-001","role":"assistant","sequence":1,"content":"I will query the synthetic account.","tool_calls":[{"id":"call-001","type":"function","name":"lookup_customer","arguments":{"customer_email":"alice@example.test","case_note":"The synthetic account belongs to Alice Example."}}]},{"id":"msg-003","parent_id":"msg-002","role":"tool","sequence":2,"tool_call_id":"call-001","name":"lookup_customer","tool_result":{"status":"ok","content":"Customer alice@example.test has phone +1-555-0100.","account_email":"alice@example.test"}}]} diff --git a/tests/fixtures/streaming/intake_atif_v10.json b/tests/fixtures/streaming/intake_atif_v10.json new file mode 100644 index 00000000..65859546 --- /dev/null +++ b/tests/fixtures/streaming/intake_atif_v10.json @@ -0,0 +1,53 @@ +{ + "schema_version": "ATIF-v1.0", + "session_id": "atif-session-v10", + "trajectory_id": "trajectory-v10", + "agent": { + "name": "validation-agent", + "version": "1.0.0", + "model_name": "validation-model" + }, + "steps": [ + { + "step_id": 1, + "timestamp": "2026-08-19T00:00:00Z", + "source": "user", + "message": "Contact Alice at alice@example.test" + }, + { + "step_id": 2, + "timestamp": "2026-08-19T00:00:01Z", + "source": "agent", + "message": "I will look up Alice", + "model_name": "validation-model", + "tool_calls": [ + { + "tool_call_id": "call-v10-1", + "function_name": "lookup_contact", + "arguments": { + "email": "alice@example.test" + } + } + ], + "observation": { + "results": [ + { + "source_call_id": "call-v10-1", + "content": "Alice works at Acme" + } + ] + }, + "metrics": { + "prompt_tokens": 12, + "completion_tokens": 8, + "cost_usd": 0.001 + } + } + ], + "final_metrics": { + "total_prompt_tokens": 12, + "total_completion_tokens": 8, + "total_cost_usd": 0.001, + "total_steps": 2 + } +} diff --git a/tests/fixtures/streaming/intake_atif_v17.json b/tests/fixtures/streaming/intake_atif_v17.json new file mode 100644 index 00000000..980efcc5 --- /dev/null +++ b/tests/fixtures/streaming/intake_atif_v17.json @@ -0,0 +1,62 @@ +{ + "schema_version": "ATIF-v1.7", + "session_id": "atif-session-v17", + "trajectory_id": "trajectory-v17", + "agent": { + "name": "validation-agent", + "version": "1.7.0", + "model_name": "validation-model" + }, + "steps": [ + { + "step_id": 1, + "timestamp": "2026-08-19T00:01:00Z", + "source": "user", + "message": [ + { + "type": "text", + "text": "Email Bob at bob@example.test" + } + ] + }, + { + "step_id": 2, + "timestamp": "2026-08-19T00:01:01Z", + "source": "agent", + "message": "I found Bob", + "reasoning_content": "Search Acme records for Bob", + "model_name": "validation-model", + "tool_calls": [ + { + "tool_call_id": "call-v17-1", + "function_name": "search_directory", + "arguments": { + "query": "Bob at Acme", + "limit": 5 + }, + "extra": { + "invocation_index": 1 + } + } + ], + "observation": { + "results": [ + { + "source_call_id": "call-v17-1", + "content": [ + { + "type": "text", + "text": "Bob uses bob@example.test" + } + ] + } + ] + } + } + ], + "notes": "Validation trajectory for Bob", + "evaluation_context": { + "evaluation_id": "sdk-validation", + "test_case_id": "atif-v17" + } +} diff --git a/tests/fixtures/streaming/intake_chat_completion.json b/tests/fixtures/streaming/intake_chat_completion.json new file mode 100644 index 00000000..73dd661e --- /dev/null +++ b/tests/fixtures/streaming/intake_chat_completion.json @@ -0,0 +1,69 @@ +{ + "request": { + "model": "gpt-validation", + "messages": [ + { + "role": "system", + "content": "Keep answers brief" + }, + { + "role": "user", + "content": "Find Carol at carol@example.test" + }, + { + "role": "assistant", + "content": null, + "tool_calls": [ + { + "id": "call-chat-1", + "type": "function", + "function": { + "name": "lookup_contact", + "arguments": "{\"email\":\"carol@example.test\"}" + } + } + ] + }, + { + "role": "tool", + "tool_call_id": "call-chat-1", + "content": "Carol works at Acme" + } + ], + "temperature": 0.2, + "provider_extension": { + "region": "us-west", + "request_class": "validation" + } + }, + "response": { + "id": "chatcmpl-validation-1", + "object": "chat.completion", + "created": 1767225600, + "model": "gpt-validation-2026-08-19", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": "Carol can be reached at carol@example.test" + }, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 32, + "completion_tokens": 9, + "total_tokens": 41 + }, + "provider_response_id": "provider-validation-1" + }, + "session_id": "chat-session-1", + "trace_id": "chat-trace-1", + "provider": "validation-provider", + "cost_usd": 0.0003, + "evaluation_context": { + "evaluation_id": "sdk-validation", + "test_case_id": "chat-completion" + } +} diff --git a/tests/fixtures/streaming/intake_local_otlp_trace.json b/tests/fixtures/streaming/intake_local_otlp_trace.json new file mode 100644 index 00000000..264dda40 --- /dev/null +++ b/tests/fixtures/streaming/intake_local_otlp_trace.json @@ -0,0 +1,42 @@ +{ + "resource_attributes": { + "service.name": "intake-spans-smoke" + }, + "scope": { + "name": "nmp.intake.spans.sample", + "version": "1.0.0" + }, + "spans": [ + { + "trace_id": "00000000000000000000000000000001", + "span_id": "0000000000000001", + "name": "sample-chain", + "start_time_unix_nano": 1787097600000000000, + "end_time_unix_nano": 1787097601000000000, + "attributes": { + "openinference.span.kind": "CHAIN", + "gen_ai.conversation.id": "sample-session" + } + }, + { + "trace_id": "00000000000000000000000000000001", + "span_id": "0000000000000002", + "parent_span_id": "0000000000000001", + "name": "sample-llm", + "start_time_unix_nano": 1787097600100000000, + "end_time_unix_nano": 1787097600900000000, + "attributes": { + "openinference.span.kind": "LLM", + "gen_ai.agent.name": "sdk-validation-agent", + "gen_ai.conversation.id": "sample-session", + "gen_ai.system": "openai", + "gen_ai.request.model": "gpt-validation", + "gen_ai.usage.input_tokens": 12, + "gen_ai.usage.output_tokens": 8, + "gen_ai.usage.total_tokens": 20, + "input.value": "{\"messages\":[{\"role\":\"user\",\"content\":\"Email Dave at dave@example.test\"}]}", + "output.value": "{\"choices\":[{\"message\":{\"role\":\"assistant\",\"content\":\"Dave works at Acme\"}}]}" + } + } + ] +} diff --git a/tests/fixtures/streaming/openshell_process_activity.jsonl b/tests/fixtures/streaming/openshell_process_activity.jsonl new file mode 100644 index 00000000..c43d1044 --- /dev/null +++ b/tests/fixtures/streaming/openshell_process_activity.jsonl @@ -0,0 +1,2 @@ +{"class_uid":1007,"class_name":"Process Activity","category_uid":1,"category_name":"System Activity","activity_id":1,"activity_name":"Launch","type_uid":100701,"type_name":"Process Activity: Launch","time":1786464000000,"severity_id":1,"severity":"Informational","status_id":1,"status":"Success","message":"Process started for alice@example.test","metadata":{"version":"1.7.0","product":{"name":"OpenShell Sandbox Supervisor","vendor_name":"OpenShell","version":"0.9.0"},"profiles":["security_control","container","host"],"uid":"sandbox-alpha"},"device":{"hostname":"alice-host.example.test","os":{"name":"Linux"}},"container":{"name":"alice-workspace","uid":"sandbox-alpha","image":{"name":"registry.example.test/alice/agent:latest"}},"process":{"name":"alice-agent","pid":42,"cmd_line":"python /work/alice@example.test/main.py"},"actor":{"process":{"name":"openshell-sandbox","pid":1}},"launch_type_id":1,"launch_type":"Spawn","action_id":1,"action":"Allowed","disposition_id":1,"disposition":"Allowed"} +{"class_uid":1007,"class_name":"Process Activity","category_uid":1,"category_name":"System Activity","activity_id":2,"activity_name":"Terminate","type_uid":100702,"type_name":"Process Activity: Terminate","time":1786464001000,"severity_id":1,"severity":"Informational","status_id":1,"status":"Success","message":"alice-agent exited after contacting alice@example.test","metadata":{"version":"1.7.0","product":{"name":"OpenShell Sandbox Supervisor","vendor_name":"OpenShell","version":"0.9.0"},"profiles":["security_control","container","host"],"uid":"sandbox-alpha"},"device":{"hostname":"alice-host.example.test","os":{"name":"Linux"}},"container":{"name":"alice-workspace","uid":"sandbox-alpha","image":{"name":"registry.example.test/alice/agent:latest"}},"process":{"name":"alice-agent","pid":42,"cmd_line":"python /work/alice@example.test/main.py"},"exit_code":0} diff --git a/tests/fixtures/streaming/sandbox_agent_prompt.md b/tests/fixtures/streaming/sandbox_agent_prompt.md new file mode 100644 index 00000000..56f1ee6f --- /dev/null +++ b/tests/fixtures/streaming/sandbox_agent_prompt.md @@ -0,0 +1,16 @@ + + + +Run a small synthetic contact-resolution task for an integration test. + +The contact is fictional test data: + +- Name: Mira Testperson +- Email: mira.sandbox@example.test +- Phone: 555-0109 +- Organization: Acme Validation Lab + +Create `contact-summary.txt` in the current workspace with one sentence that +contains all four values exactly. Then read the file back and finish with a +short confirmation that repeats the same four values. Do not access the +network, inspect unrelated files, or use any real personal data. diff --git a/tests/interface/cli/test_cli_errors.py b/tests/interface/cli/test_cli_errors.py index fbac0c5a..a2c278d4 100644 --- a/tests/interface/cli/test_cli_errors.py +++ b/tests/interface/cli/test_cli_errors.py @@ -10,7 +10,7 @@ import pytest from anonymizer.interface.cli.main import app -from anonymizer.interface.errors import AnonymizerIOError, InvalidConfigError +from anonymizer.interface.errors import AnonymizerIOError, AnonymizerWorkflowError, InvalidConfigError def test_invalid_source_exits(tmp_path: Path) -> None: @@ -54,6 +54,7 @@ def csv_file(tmp_path: Path) -> Path: [ InvalidConfigError("bad config"), AnonymizerIOError("io error"), + AnonymizerWorkflowError("Anonymization pipeline failed."), ValueError("bad value"), ], ) diff --git a/tests/interface/test_anonymizer_interface.py b/tests/interface/test_anonymizer_interface.py index d0a7db56..d2062eaa 100644 --- a/tests/interface/test_anonymizer_interface.py +++ b/tests/interface/test_anonymizer_interface.py @@ -30,12 +30,20 @@ from anonymizer.engine.detection.detection_workflow import EntityDetectionResult, EntityDetectionWorkflow from anonymizer.engine.ndd.adapter import FailedRecord from anonymizer.engine.ndd.model_loader import load_default_model_providers, validate_model_alias_references +from anonymizer.engine.private_row_verification import PRIVATE_CORRELATION_COLUMN from anonymizer.engine.replace.replace_runner import ReplacementResult, ReplacementWorkflow from anonymizer.engine.rewrite.rewrite_workflow import RewriteResult, RewriteWorkflow from anonymizer.interface.anonymizer import Anonymizer, _resolve_model_providers from anonymizer.interface.errors import InvalidConfigError, InvalidInputError +def test_workflow_error_is_part_of_the_top_level_public_error_contract() -> None: + import anonymizer + + assert hasattr(anonymizer, "AnonymizerWorkflowError") + assert issubclass(anonymizer.AnonymizerWorkflowError, anonymizer.AnonymizerError) + + @pytest.fixture def stub_input(tmp_path: Path) -> AnonymizerInput: csv_path = tmp_path / "input.csv" @@ -48,19 +56,35 @@ def _make_anonymizer( replace_return: ReplacementResult | None = None, rewrite_return: RewriteResult | None = None, ) -> tuple[Anonymizer, Mock, Mock, Mock]: + def preserve_accepted_detections(dataframe: pd.DataFrame, result_dataframe: pd.DataFrame) -> pd.DataFrame: + output = result_dataframe.copy() + if PRIVATE_CORRELATION_COLUMN in dataframe and len(output) == len(dataframe): + output[PRIVATE_CORRELATION_COLUMN] = dataframe[PRIVATE_CORRELATION_COLUMN].to_numpy() + if COL_FINAL_ENTITIES not in output and len(output) == len(dataframe): + output[COL_FINAL_ENTITIES] = dataframe[COL_FINAL_ENTITIES].to_numpy() + return output + detection_workflow = Mock(spec=EntityDetectionWorkflow) - detection_workflow.run.return_value = detection_return or EntityDetectionResult( + detection_result = detection_return or EntityDetectionResult( dataframe=pd.DataFrame({COL_TEXT: ["Alice works at Acme"], COL_FINAL_ENTITIES: [{"entities": []}]}), failed_records=[], ) + detection_workflow.run.side_effect = lambda dataframe, **_kwargs: EntityDetectionResult( + dataframe=preserve_accepted_detections(dataframe, detection_result.dataframe), + failed_records=detection_result.failed_records, + ) _replace_df = pd.DataFrame( {COL_TEXT: ["Alice works at Acme"], COL_REPLACED_TEXT: ["[REDACTED] works at [REDACTED]"]} ) replace_runner = Mock(spec=ReplacementWorkflow) - replace_runner.run.return_value = replace_return or ReplacementResult( + replace_result = replace_return or ReplacementResult( dataframe=_replace_df, failed_records=[], ) + replace_runner.run.side_effect = lambda dataframe, **_kwargs: ReplacementResult( + dataframe=preserve_accepted_detections(dataframe, replace_result.dataframe), + failed_records=replace_result.failed_records, + ) _rewrite_df = pd.DataFrame( { COL_TEXT: ["Alice works at Acme"], @@ -73,10 +97,14 @@ def _make_anonymizer( } ) rewrite_runner = Mock(spec=RewriteWorkflow) - rewrite_runner.run.return_value = rewrite_return or RewriteResult( + rewrite_result = rewrite_return or RewriteResult( dataframe=_rewrite_df, failed_records=[], ) + rewrite_runner.run.side_effect = lambda dataframe, **_kwargs: RewriteResult( + dataframe=preserve_accepted_detections(dataframe, rewrite_result.dataframe), + failed_records=rewrite_result.failed_records, + ) anonymizer = Anonymizer( detection_workflow=detection_workflow, replace_runner=replace_runner, @@ -289,7 +317,7 @@ def test_run_exposes_trace_dataframe_and_filters_internal_columns( assert COL_REPLACEMENT_MAP in result.trace_dataframe.columns assert COL_DETECTED_ENTITIES not in result.dataframe.columns assert COL_REPLACEMENT_MAP not in result.dataframe.columns - assert set(result.dataframe.columns) == {"text", "text_replaced", "text_with_spans"} + assert set(result.dataframe.columns) == {"text", "text_replaced", "text_with_spans", COL_FINAL_ENTITIES} def test_preview_exposes_trace_dataframe_for_display( @@ -456,9 +484,7 @@ def test_run_with_colliding_output_column_renames_input_and_warns( "bio_replaced__input": ["pre-existing"], } ) - anonymizer, _, _, _ = _make_anonymizer( - replace_return=ReplacementResult(dataframe=replace_df, failed_records=[]), - ) + anonymizer, _, _, _ = _make_anonymizer(replace_return=ReplacementResult(dataframe=replace_df, failed_records=[])) with caplog.at_level("WARNING", logger="anonymizer"): result = anonymizer.run( @@ -496,6 +522,10 @@ def test_run_with_text_column_matching_static_output_preserves_both_columns( } ) anonymizer, _, _, _ = _make_anonymizer( + detection_return=EntityDetectionResult( + dataframe=pd.DataFrame({COL_TEXT: ["Alice bio text"], COL_FINAL_ENTITIES: [entities_payload]}), + failed_records=[], + ), replace_return=ReplacementResult(dataframe=replace_df, failed_records=[]), ) @@ -713,7 +743,7 @@ def test_run_rewrite_uses_combined_runner_when_enabled(stub_input: AnonymizerInp config = AnonymizerConfig(rewrite=Rewrite(use_combined_graph=True)) anonymizer, _, _, rewrite_runner = _make_anonymizer() combined_runner = Mock(spec=RewriteWorkflow) - combined_runner.run.return_value = rewrite_runner.run.return_value + combined_runner.run.side_effect = rewrite_runner.run.side_effect anonymizer._combined_rewrite_runner = combined_runner anonymizer.run(config=config, data=stub_input) diff --git a/tests/interface/test_anonymizer_logging.py b/tests/interface/test_anonymizer_logging.py index 3b0c40b8..7fcbb573 100644 --- a/tests/interface/test_anonymizer_logging.py +++ b/tests/interface/test_anonymizer_logging.py @@ -78,6 +78,7 @@ def _make_logging_anonymizer( { COL_TEXT: ["Alice works at Acme", "Bob likes cats"], COL_REPLACED_TEXT: ["[REDACTED] works at [REDACTED]", "[REDACTED] likes cats"], + COL_FINAL_ENTITIES: entities, } ) replace_runner = Mock(spec=ReplacementWorkflow) @@ -93,6 +94,7 @@ def _make_logging_anonymizer( "leakage_mass": [0.3, 0.1], "any_high_leaked": [False, False], "needs_human_review": [False, False], + COL_FINAL_ENTITIES: entities, } ) rewrite_runner = Mock(spec=RewriteWorkflow) @@ -175,6 +177,24 @@ def test_run_logs_failure_counts(stub_input: AnonymizerInput, caplog: pytest.Log assert "2 total failures" in messages +def test_run_preserves_debug_input_and_failure_diagnostics( + stub_input: AnonymizerInput, caplog: pytest.LogCaptureFixture +) -> None: + anonymizer = _make_logging_anonymizer( + detection_failures=[FailedRecord(record_id="r1", step="detection", reason="timeout")], + ) + + with caplog.at_level(logging.DEBUG, logger="anonymizer"): + anonymizer.run(config=AnonymizerConfig(replace=Redact()), data=stub_input) + + messages = caplog.text + assert "input text lengths: min=" in messages + assert "detection config: threshold=0.30" in messages + assert "1 record(s) failed during pipeline processing." in messages + assert "r1" not in messages + assert "timeout" not in messages + + def test_run_without_replacement_skips_replace_logs( stub_input: AnonymizerInput, caplog: pytest.LogCaptureFixture ) -> None: @@ -290,7 +310,8 @@ def test_evaluate_debug_logs_config_and_failures_without_sensitive_context( assert "active evaluation judges: entity_coverage_judge" in messages assert "evaluation models:" in messages assert "1 evaluation failed record(s)" in messages - assert "r1 (entity-coverage-judge: timeout)" in messages + assert "r1" not in messages + assert "timeout" not in messages assert "confidential dataset description" not in messages @@ -452,6 +473,14 @@ def test_preview_set_preview_num_records_not_capped(stub_input: AnonymizerInput) """When num_records < available rows, preview_num_records is forwarded as-is.""" anonymizer = _make_logging_anonymizer() config = AnonymizerConfig(replace=Redact()) + detection = _mock_method(anonymizer._detection_workflow.run) + replacement = _mock_method(anonymizer._replace_runner.run) + detection.side_effect = lambda dataframe, **_kwargs: EntityDetectionResult( + dataframe=detection.return_value.dataframe.iloc[: len(dataframe)].copy(), failed_records=[] + ) + replacement.side_effect = lambda dataframe, **_kwargs: ReplacementResult( + dataframe=replacement.return_value.dataframe.iloc[: len(dataframe)].copy(), failed_records=[] + ) # stub_input has 2 rows; num_records=1 fits, so no clamping anonymizer.preview(config=config, data=stub_input, num_records=1) @@ -511,6 +540,7 @@ def test_preview_with_large_input_only_loads_preview_rows(tmp_path: Path, caplog { COL_TEXT: [f"Name{i} works here" for i in range(num_preview)], COL_REPLACED_TEXT: ["[REDACTED] works here" for _ in range(num_preview)], + COL_FINAL_ENTITIES: entities, } ) replace_runner = Mock(spec=ReplacementWorkflow) diff --git a/tests/interface/test_anonymizer_telemetry.py b/tests/interface/test_anonymizer_telemetry.py index aaaf0efc..a992d9bd 100644 --- a/tests/interface/test_anonymizer_telemetry.py +++ b/tests/interface/test_anonymizer_telemetry.py @@ -17,9 +17,11 @@ from anonymizer.engine.constants import COL_FINAL_ENTITIES, COL_REPLACED_TEXT, COL_REWRITTEN_TEXT, COL_TEXT from anonymizer.engine.detection.detection_workflow import EntityDetectionResult, EntityDetectionWorkflow from anonymizer.engine.ndd.adapter import FailedRecord +from anonymizer.engine.private_row_verification import PRIVATE_CORRELATION_COLUMN from anonymizer.engine.replace.replace_runner import ReplacementResult, ReplacementWorkflow from anonymizer.engine.rewrite.rewrite_workflow import RewriteResult, RewriteWorkflow from anonymizer.interface.anonymizer import Anonymizer +from anonymizer.interface.errors import AnonymizerWorkflowError from anonymizer.telemetry import ( NOT_APPLICABLE, AnonymizerEvent, @@ -40,20 +42,36 @@ def _make_anonymizer( replace_return: ReplacementResult | None = None, rewrite_return: RewriteResult | None = None, ) -> tuple[Anonymizer, Mock, Mock, Mock]: + def preserve_accepted_detections(dataframe: pd.DataFrame, result_dataframe: pd.DataFrame) -> pd.DataFrame: + output = result_dataframe.copy() + if PRIVATE_CORRELATION_COLUMN in dataframe and len(output) == len(dataframe): + output[PRIVATE_CORRELATION_COLUMN] = dataframe[PRIVATE_CORRELATION_COLUMN].to_numpy() + if COL_FINAL_ENTITIES not in output and len(output) == len(dataframe): + output[COL_FINAL_ENTITIES] = dataframe[COL_FINAL_ENTITIES].to_numpy() + return output + detection_workflow = Mock(spec=EntityDetectionWorkflow) - detection_workflow.run.return_value = detection_return or EntityDetectionResult( + detection_result = detection_return or EntityDetectionResult( dataframe=pd.DataFrame({COL_TEXT: ["Alice works at Acme"], COL_FINAL_ENTITIES: [{"entities": []}]}), failed_records=[], ) + detection_workflow.run.side_effect = lambda dataframe, **_kwargs: EntityDetectionResult( + dataframe=preserve_accepted_detections(dataframe, detection_result.dataframe), + failed_records=detection_result.failed_records, + ) _replace_df = pd.DataFrame( {COL_TEXT: ["Alice works at Acme"], COL_REPLACED_TEXT: ["[REDACTED] works at [REDACTED]"]} ) _replace_df.attrs["original_text_column"] = "text" replace_runner = Mock(spec=ReplacementWorkflow) - replace_runner.run.return_value = replace_return or ReplacementResult( + replace_result = replace_return or ReplacementResult( dataframe=_replace_df, failed_records=[], ) + replace_runner.run.side_effect = lambda dataframe, **_kwargs: ReplacementResult( + dataframe=preserve_accepted_detections(dataframe, replace_result.dataframe), + failed_records=replace_result.failed_records, + ) _rewrite_df = pd.DataFrame( { COL_TEXT: ["Alice works at Acme"], @@ -67,10 +85,14 @@ def _make_anonymizer( ) _rewrite_df.attrs["original_text_column"] = "text" rewrite_runner = Mock(spec=RewriteWorkflow) - rewrite_runner.run.return_value = rewrite_return or RewriteResult( + rewrite_result = rewrite_return or RewriteResult( dataframe=_rewrite_df, failed_records=[], ) + rewrite_runner.run.side_effect = lambda dataframe, **_kwargs: RewriteResult( + dataframe=preserve_accepted_detections(dataframe, rewrite_result.dataframe), + failed_records=rewrite_result.failed_records, + ) anonymizer = Anonymizer( detection_workflow=detection_workflow, replace_runner=replace_runner, @@ -163,11 +185,16 @@ def test_run_emits_error_event_and_reraises( stub_input: AnonymizerInput, ) -> None: anonymizer, detection_wf, _, _ = _make_anonymizer() - detection_wf.run.side_effect = RuntimeError("kaboom") + secret = "provider failure containing synthetic-secret@example.test" + detection_wf.run.side_effect = RuntimeError(secret) - with pytest.raises(RuntimeError, match="kaboom"): + with pytest.raises(AnonymizerWorkflowError, match="Anonymization pipeline failed") as exc_info: anonymizer.run(config=AnonymizerConfig(replace=Redact()), data=stub_input) + assert secret not in str(exc_info.value) + assert exc_info.value.__cause__ is None + assert exc_info.value.__context__ is None + assert len(captured_events) == 1 assert captured_events[0].task_status == TaskStatusEnum.ERROR @@ -199,6 +226,25 @@ def test_preview_emits_task_preview( assert captured_events[0].task == TaskEnum.PREVIEW assert captured_events[0].task_status == TaskStatusEnum.COMPLETED + def test_preview_emits_error_event_and_raises_public_sanitized_error( + self, + captured_events: list[AnonymizerEvent], + stub_input: AnonymizerInput, + ) -> None: + anonymizer, detection_wf, _, _ = _make_anonymizer() + secret = "provider failure containing synthetic-secret@example.test" + detection_wf.run.side_effect = RuntimeError(secret) + + with pytest.raises(AnonymizerWorkflowError, match="Anonymization pipeline failed") as exc_info: + anonymizer.preview(config=AnonymizerConfig(replace=Redact()), data=stub_input, num_records=5) + + assert secret not in str(exc_info.value) + assert exc_info.value.__cause__ is None + assert exc_info.value.__context__ is None + assert len(captured_events) == 1 + assert captured_events[0].task == TaskEnum.PREVIEW + assert captured_events[0].task_status == TaskStatusEnum.ERROR + # ============================================================================= # Opt-out diff --git a/tests/interface/test_phase6_public_compatibility.py b/tests/interface/test_phase6_public_compatibility.py new file mode 100644 index 00000000..5b3999ff --- /dev/null +++ b/tests/interface/test_phase6_public_compatibility.py @@ -0,0 +1,556 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +from collections.abc import Callable +from pathlib import Path +from unittest.mock import Mock, patch + +import pandas as pd +import pytest + +from anonymizer.config.anonymizer_config import AnonymizerConfig, AnonymizerInput +from anonymizer.config.replace_strategies import Annotate, Hash, Redact, ReplaceMethod, Substitute +from anonymizer.engine.constants import ( + COL_ATTRIBUTE_FIDELITY_INVALID_ENTITIES, + COL_ATTRIBUTE_FIDELITY_VALID, + COL_DETECTED_ENTITIES, + COL_ENTITIES_BY_VALUE, + COL_ENTITY_COVERAGE, + COL_FINAL_ENTITIES, + COL_MISSED_ENTITIES, + COL_RELATIONAL_CONSISTENCY_INVALID_RELATIONS, + COL_RELATIONAL_CONSISTENCY_VALID, + COL_REPLACEMENT_APPLICATION, + COL_REPLACEMENT_MAP, + COL_TAGGED_TEXT, + COL_TARGET_WORK_ID, + COL_TEXT, + COL_TYPE_FIDELITY_INVALID_REPLACEMENTS, + COL_TYPE_FIDELITY_VALID, +) +from anonymizer.engine.detection.detection_workflow import EntityDetectionResult, EntityDetectionWorkflow +from anonymizer.engine.execution.phase6_runtime import _CandidateProposal +from anonymizer.engine.ndd.adapter import FailedRecord, NddAdapter +from anonymizer.engine.replace.llm_replace_workflow import LlmReplaceWorkflow, _get_replacement_mapping_prompt +from anonymizer.engine.replace.replace_runner import ReplacementResult, ReplacementWorkflow +from anonymizer.engine.replace.strategies import ( + ReplacementEntry, + apply_local_replace_strategy, + apply_replacement_map, + apply_replacements_to_spans, +) +from anonymizer.engine.rewrite.rewrite_workflow import RewriteWorkflow +from anonymizer.engine.schemas import EntitiesSchema +from anonymizer.interface import _protection as protection_module +from anonymizer.interface._protection import _Failed +from anonymizer.interface.anonymizer import Anonymizer +from anonymizer.interface.cli.main import app +from tests.interface.test_private_protection import _AnchoredPhase6Backend, _record +from tests.streaming.structured_trace_prototype import build_synthetic_anonymizer + +_SYNTHETIC_VALUES = { + "Alice": "Avery", + "Bob": "Blake", + "Carol": "Casey", +} + + +def _entity_payload(value: str) -> dict[str, list[dict[str, object]]]: + return { + "entities": [ + { + "id": f"entity-{value}", + "value": value, + "label": "first_name", + "start_position": 0, + "end_position": len(value), + "score": 1.0, + "source": "synthetic-test", + } + ] + } + + +def _detect(dataframe: pd.DataFrame, **_kwargs: object) -> EntityDetectionResult: + detected = dataframe.copy() + values = detected[COL_TEXT].astype(str).tolist() + payloads = [_entity_payload(value) for value in values] + detected[COL_DETECTED_ENTITIES] = payloads + detected[COL_FINAL_ENTITIES] = payloads + detected[COL_ENTITIES_BY_VALUE] = [ + {"entities_by_value": [{"value": value, "labels": ["first_name"]}]} for value in values + ] + detected[COL_TAGGED_TEXT] = [f"{value}" for value in values] + return EntityDetectionResult(dataframe=detected, failed_records=[]) + + +def _detect_no_entities(dataframe: pd.DataFrame, **_kwargs: object) -> EntityDetectionResult: + detected = dataframe.copy() + detected[COL_DETECTED_ENTITIES] = [{"entities": []} for _ in range(len(detected))] + detected[COL_FINAL_ENTITIES] = [{"entities": []} for _ in range(len(detected))] + detected[COL_ENTITIES_BY_VALUE] = [{"entities_by_value": []} for _ in range(len(detected))] + detected[COL_TAGGED_TEXT] = detected[COL_TEXT].astype(str) + return EntityDetectionResult(dataframe=detected, failed_records=[]) + + +def _detect_malformed_anchor(dataframe: pd.DataFrame, **_kwargs: object) -> EntityDetectionResult: + detected = dataframe.copy() + malformed = { + "entities": [ + { + "id": "malformed-alice", + "value": "Alice", + "label": "first_name", + "start_position": 0, + "end_position": 4, + "score": 1.0, + "source": "synthetic-test", + } + ] + } + detected[COL_DETECTED_ENTITIES] = [malformed for _ in range(len(detected))] + detected[COL_FINAL_ENTITIES] = [malformed for _ in range(len(detected))] + detected[COL_ENTITIES_BY_VALUE] = [ + {"entities_by_value": [{"value": "Alice", "labels": ["first_name"]}]} for _ in range(len(detected)) + ] + detected[COL_TAGGED_TEXT] = detected[COL_TEXT].astype(str) + return EntityDetectionResult(dataframe=detected, failed_records=[]) + + +def _replacement_map(value: str) -> dict[str, list[dict[str, str]]]: + return { + "replacements": [ + { + "original": value, + "label": "first_name", + "synthetic": _SYNTHETIC_VALUES[value], + } + ] + } + + +def _replace( + dataframe: pd.DataFrame, + *, + replace_method: ReplaceMethod, + **_kwargs: object, +) -> ReplacementResult: + if isinstance(replace_method, Substitute): + mapped = dataframe.copy() + mapped[COL_REPLACEMENT_MAP] = [_replacement_map(str(value)) for value in mapped[COL_TEXT]] + replaced = apply_replacement_map(mapped) + else: + replaced = apply_local_replace_strategy(dataframe, strategy=replace_method) + return ReplacementResult(dataframe=replaced, failed_records=[]) + + +def _evaluate( + dataframe: pd.DataFrame, + *, + replace_method: ReplaceMethod, + **_kwargs: object, +) -> ReplacementResult: + evaluated = dataframe.copy() + evaluated[COL_ENTITY_COVERAGE] = [1.0] * len(evaluated) + evaluated[COL_MISSED_ENTITIES] = [[] for _ in range(len(evaluated))] + if isinstance(replace_method, Substitute): + evaluated[COL_TYPE_FIDELITY_VALID] = [True] * len(evaluated) + evaluated[COL_TYPE_FIDELITY_INVALID_REPLACEMENTS] = [[] for _ in range(len(evaluated))] + evaluated[COL_RELATIONAL_CONSISTENCY_VALID] = [True] * len(evaluated) + evaluated[COL_RELATIONAL_CONSISTENCY_INVALID_RELATIONS] = [[] for _ in range(len(evaluated))] + evaluated[COL_ATTRIBUTE_FIDELITY_VALID] = [True] * len(evaluated) + evaluated[COL_ATTRIBUTE_FIDELITY_INVALID_ENTITIES] = [[] for _ in range(len(evaluated))] + return ReplacementResult(dataframe=evaluated, failed_records=[]) + + +def _synthetic_anonymizer( + *, + detection_failures: tuple[FailedRecord, ...] = (), + replacement_failures: tuple[FailedRecord, ...] = (), + detect: Callable[..., EntityDetectionResult] = _detect, +) -> Anonymizer: + detection = Mock(spec=EntityDetectionWorkflow) + detection.run.side_effect = lambda dataframe, **kwargs: EntityDetectionResult( + dataframe=detect(dataframe, **kwargs).dataframe, + failed_records=list(detection_failures), + ) + replacement = Mock(spec=ReplacementWorkflow) + replacement.run.side_effect = lambda dataframe, **kwargs: ReplacementResult( + dataframe=_replace(dataframe, **kwargs).dataframe, + failed_records=list(replacement_failures), + ) + replacement.evaluate.side_effect = _evaluate + return Anonymizer( + data_designer=Mock(), + detection_workflow=detection, + replace_runner=replacement, + rewrite_runner=Mock(spec=RewriteWorkflow), + ) + + +def _base_dataframe() -> pd.DataFrame: + return pd.DataFrame( + { + "record_id": ["r-alice-1", "r-bob", "r-alice-2", "r-carol"], + "text": ["Alice", "Bob", "Alice", "Carol"], + }, + index=pd.Index([11, 4, 11, 2]), + ) + + +_FRAME_VARIANTS: dict[str, Callable[[pd.DataFrame], pd.DataFrame]] = { + "duplicate_non_monotonic_index": lambda frame: frame.copy(), + "filtered": lambda frame: frame.iloc[[3, 1]], + "reordered": lambda frame: frame.iloc[[2, 0, 3, 1]], + "concatenated": lambda frame: pd.concat([frame.iloc[2:], frame.iloc[:2]]), + "reset_index": lambda frame: frame.iloc[[1, 3, 0]].reset_index(drop=True), +} + + +def _write_input(frame: pd.DataFrame, path: Path, file_format: str) -> AnonymizerInput: + if file_format == "csv": + frame.to_csv(path, index=False) + else: + frame.to_parquet(path, index=True) + return AnonymizerInput(source=str(path), id_column="record_id") + + +_EXPECTED_REPLACEMENTS = { + "redact": { + "Alice": "[REDACTED_FIRST_NAME]", + "Bob": "[REDACTED_FIRST_NAME]", + "Carol": "[REDACTED_FIRST_NAME]", + }, + "annotate": { + "Alice": "", + "Bob": "", + "Carol": "", + }, + "hash": { + "Alice": "", + "Bob": "", + "Carol": "", + }, + "substitute": { + "Alice": "Avery", + "Bob": "Blake", + "Carol": "Casey", + }, +} + + +@pytest.mark.parametrize( + ("strategy_name", "replace_method"), + [ + pytest.param("redact", Redact(), id="redact"), + pytest.param("annotate", Annotate(), id="annotate"), + pytest.param("hash", Hash(), id="hash"), + pytest.param("substitute", Substitute(), id="substitute"), + ], +) +@pytest.mark.parametrize("variant_name", list(_FRAME_VARIANTS)) +@pytest.mark.parametrize("file_format", ["csv", "parquet"]) +def test_public_run_preserves_supported_dataframe_shapes_and_strategy_outputs( + tmp_path: Path, + strategy_name: str, + replace_method: ReplaceMethod, + variant_name: str, + file_format: str, +) -> None: + source_frame = _FRAME_VARIANTS[variant_name](_base_dataframe()) + input_data = _write_input(source_frame, tmp_path / f"input.{file_format}", file_format) + + result = _synthetic_anonymizer().run( + config=AnonymizerConfig(replace=replace_method, emit_telemetry=False), + data=input_data, + ) + + expected_text = source_frame["text"].tolist() + assert result.dataframe["text"].tolist() == expected_text + assert result.dataframe["text_replaced"].tolist() == [ + _EXPECTED_REPLACEMENTS[strategy_name][value] for value in expected_text + ] + assert result.dataframe["text_with_spans"].tolist() == [ + f"{value}" for value in expected_text + ] + assert result.trace_dataframe["record_id"].tolist() == source_frame["record_id"].tolist() + assert set(result.dataframe.columns) == {"text", "text_replaced", "text_with_spans", COL_FINAL_ENTITIES} + assert COL_DETECTED_ENTITIES in result.trace_dataframe.columns + assert COL_REPLACEMENT_MAP in result.trace_dataframe.columns + assert COL_TARGET_WORK_ID not in result.trace_dataframe.columns + assert result.failed_records == [] + expected_index = list(range(len(source_frame))) if file_format == "csv" else source_frame.index.tolist() + assert result.dataframe.index.tolist() == expected_index + assert result.trace_dataframe.index.tolist() == expected_index + + +@pytest.mark.parametrize( + ("strategy_name", "replace_method"), + [ + pytest.param("redact", Redact(), id="redact"), + pytest.param("annotate", Annotate(), id="annotate"), + pytest.param("hash", Hash(), id="hash"), + pytest.param("substitute", Substitute(), id="substitute"), + ], +) +def test_public_preview_and_evaluate_keep_order_columns_and_metrics( + tmp_path: Path, + strategy_name: str, + replace_method: ReplaceMethod, +) -> None: + source_frame = _FRAME_VARIANTS["reordered"](_base_dataframe()) + input_data = _write_input(source_frame, tmp_path / "input.parquet", "parquet") + anonymizer = _synthetic_anonymizer() + + preview = anonymizer.preview( + config=AnonymizerConfig(replace=replace_method, emit_telemetry=False), + data=input_data, + num_records=2, + ) + evaluated = anonymizer.evaluate(preview) + + expected_text = source_frame["text"].iloc[:2].tolist() + assert preview.preview_num_records == 2 + assert preview.dataframe["text"].tolist() == expected_text + assert preview.dataframe["text_replaced"].tolist() == [ + _EXPECTED_REPLACEMENTS[strategy_name][value] for value in expected_text + ] + assert preview.dataframe.index.tolist() == source_frame.index[:2].tolist() + assert evaluated.dataframe["text"].tolist() == expected_text + assert evaluated.dataframe[COL_ENTITY_COVERAGE].tolist() == [1.0, 1.0] + assert evaluated.dataframe[COL_MISSED_ENTITIES].tolist() == [[], []] + assert evaluated.trace_dataframe.index.tolist() == source_frame.index[:2].tolist() + assert evaluated.failed_records == [] + if isinstance(replace_method, Substitute): + assert evaluated.dataframe[COL_TYPE_FIDELITY_VALID].tolist() == [True, True] + assert evaluated.dataframe[COL_RELATIONAL_CONSISTENCY_VALID].tolist() == [True, True] + assert evaluated.dataframe[COL_ATTRIBUTE_FIDELITY_VALID].tolist() == [True, True] + else: + assert COL_TYPE_FIDELITY_VALID not in evaluated.dataframe.columns + assert COL_RELATIONAL_CONSISTENCY_VALID not in evaluated.dataframe.columns + assert COL_ATTRIBUTE_FIDELITY_VALID not in evaluated.dataframe.columns + + +@pytest.mark.parametrize( + ("replace_method", "expected"), + [ + pytest.param(Redact(), "[REDACTED_FIRST_NAME]", id="redact"), + pytest.param(Substitute(), "Blake", id="substitute"), + ], +) +def test_public_display_renders_the_selected_transformed_record( + tmp_path: Path, + replace_method: ReplaceMethod, + expected: str, +) -> None: + source_frame = _FRAME_VARIANTS["filtered"](_base_dataframe()) + input_data = _write_input(source_frame, tmp_path / "input.parquet", "parquet") + result = _synthetic_anonymizer().run( + config=AnonymizerConfig(replace=replace_method, emit_telemetry=False), + data=input_data, + ) + + display = Mock() + ipython_display = Mock(HTML=lambda html: html, display=display) + with patch.dict("sys.modules", {"IPython": Mock(), "IPython.display": ipython_display}): + result.display_record(index=1) + + rendered = display.call_args.args[0] + assert "Bob" in rendered + assert expected in rendered + assert "Carol" not in rendered + assert result._display_cycle_index == 0 + + +@pytest.mark.parametrize( + "replace_method", + [ + pytest.param(Redact(), id="redact"), + pytest.param(Annotate(), id="annotate"), + pytest.param(Hash(), id="hash"), + pytest.param(Substitute(), id="substitute"), + ], +) +def test_public_validate_config_accepts_each_supported_replacement_strategy(replace_method: ReplaceMethod) -> None: + config = AnonymizerConfig(replace=replace_method, emit_telemetry=False) + + _synthetic_anonymizer().validate_config(config) + + assert type(config.replace).__name__ == type(replace_method).__name__ + + +@pytest.mark.parametrize( + ("strategy", "expected"), + [ + pytest.param("redact", ["[REDACTED_FIRST_NAME]"] * 4, id="redact"), + pytest.param("substitute", ["Avery", "Casey", "Avery", "Blake"], id="substitute"), + ], +) +def test_cli_run_preserves_transformed_row_order_and_literal_output( + tmp_path: Path, + strategy: str, + expected: list[str], +) -> None: + source_frame = _FRAME_VARIANTS["concatenated"](_base_dataframe()) + source = tmp_path / "input.csv" + output = tmp_path / "output.csv" + source_frame.to_csv(source, index=False) + + with patch("anonymizer.interface.cli.main.Anonymizer", return_value=_synthetic_anonymizer()): + with pytest.raises(SystemExit) as exc_info: + app( + [ + "run", + "--source", + str(source), + "--replace", + strategy, + "--no-emit-telemetry", + "--output", + str(output), + ] + ) + + assert exc_info.value.code == 0 + written = pd.read_csv(output) + assert written["text"].tolist() == ["Alice", "Carol", "Alice", "Bob"] + assert written["text_replaced"].tolist() == expected + assert list(written.columns) == ["text", COL_FINAL_ENTITIES, "text_with_spans", "text_replaced"] + + +def test_public_substitute_bypasses_replacement_provider_for_no_entity_rows(tmp_path: Path) -> None: + input_data = _write_input(_base_dataframe(), tmp_path / "input.parquet", "parquet") + adapter = Mock(spec=NddAdapter) + detection = Mock(spec=EntityDetectionWorkflow) + detection.run.side_effect = _detect_no_entities + replacement = ReplacementWorkflow(llm_workflow=LlmReplaceWorkflow(adapter=adapter)) + anonymizer = Anonymizer( + data_designer=Mock(), + detection_workflow=detection, + replace_runner=replacement, + rewrite_runner=Mock(spec=RewriteWorkflow), + ) + + result = anonymizer.run( + config=AnonymizerConfig(replace=Substitute(), emit_telemetry=False), + data=input_data, + ) + + assert result.dataframe["text_replaced"].tolist() == ["Alice", "Bob", "Alice", "Carol"] + assert result.failed_records == [] + adapter.run_workflow.assert_not_called() + + +def test_public_substitute_keeps_custom_instructions_in_the_legacy_prompt() -> None: + instruction = "P8-PROMPT-7f3bd124a9c84d7bb9f05a0b6fb420a1" + + prompt = _get_replacement_mapping_prompt( + entities_column="_legacy_entities", + instructions=instruction, + ) + + assert f"Additional instructions: {instruction}" in prompt + + +def test_public_substitute_keeps_legacy_value_only_fallback_and_non_cascading_application() -> None: + entities = EntitiesSchema.from_raw( + { + "entities": [ + { + "id": "entity-alice", + "value": "Alice", + "label": "first_name", + "start_position": 0, + "end_position": 5, + "score": 1.0, + "source": "compatibility-test", + } + ] + } + ) + + output, application = apply_replacements_to_spans( + "Alice Avery", + entities, + [ReplacementEntry("Alice", "legacy_person", "Avery")], + allow_value_fallback=True, + ) + + assert output == "Avery Avery" + assert application.to_metrics() == { + "targeted_span_count": 1, + "applied_span_count": 1, + "skipped_span_count": 0, + "skipped_span_label_counts": {}, + } + + +def test_malformed_anchor_is_accepted_by_public_legacy_and_fails_closed_in_private_phase6(tmp_path: Path) -> None: + source = pd.DataFrame({"record_id": ["r-alice"], "text": ["Alice"]}) + input_data = _write_input(source, tmp_path / "input.csv", "csv") + + public = _synthetic_anonymizer(detect=_detect_malformed_anchor).run( + config=AnonymizerConfig(replace=Substitute(), emit_telemetry=False), + data=input_data, + ) + + private_anonymizer = build_synthetic_anonymizer({"Alice": "first_name"}) + plan = private_anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + backend = _AnchoredPhase6Backend({"Alice": (_CandidateProposal(0, 4, "Alice", "first_name"),)}) + private = protection_module._ProtectionFlow(private_anonymizer, plan, phase6_backend=backend).protect( + (_record("r-alice", "Alice"),) + ) + + assert public.dataframe["text_replaced"].tolist() == ["Alice"] + assert public.trace_dataframe[COL_REPLACEMENT_APPLICATION].tolist() == [ + { + "targeted_span_count": 1, + "applied_span_count": 0, + "skipped_span_count": 1, + "skipped_span_label_counts": {"first_name": 1}, + } + ] + assert public.failed_records == [] + assert isinstance(private.outcomes[0], _Failed) + assert not hasattr(private.outcomes[0], "output") + + +def test_public_run_keeps_failed_record_shape_and_stage_order(tmp_path: Path) -> None: + detection_failure = FailedRecord(record_id="r-bob", step="entity-detection", reason="detector-timeout") + replacement_failure = FailedRecord(record_id="r-carol", step="replace-map-generation", reason="invalid-map") + source_frame = _FRAME_VARIANTS["concatenated"](_base_dataframe()) + input_data = _write_input(source_frame, tmp_path / "input.parquet", "parquet") + + result = _synthetic_anonymizer( + detection_failures=(detection_failure,), + replacement_failures=(replacement_failure,), + ).run( + config=AnonymizerConfig(replace=Redact(), emit_telemetry=False), + data=input_data, + ) + + assert result.failed_records == [detection_failure, replacement_failure] + assert result.failed_records[0] is detection_failure + assert result.failed_records[1] is replacement_failure + assert [vars(failure) for failure in result.failed_records] == [ + {"record_id": "r-bob", "step": "entity-detection", "reason": "detector-timeout"}, + {"record_id": "r-carol", "step": "replace-map-generation", "reason": "invalid-map"}, + ] + assert list(result.trace_dataframe.columns) == [ + "record_id", + "text", + COL_DETECTED_ENTITIES, + COL_FINAL_ENTITIES, + COL_ENTITIES_BY_VALUE, + "text_with_spans", + COL_REPLACEMENT_MAP, + "text_replaced", + COL_REPLACEMENT_APPLICATION, + ] + assert result.trace_dataframe.index.tolist() == [11, 2, 11, 4] + assert [(failure.record_id, failure.step, failure.reason) for failure in result.failed_records] == [ + ("r-bob", "entity-detection", "detector-timeout"), + ("r-carol", "replace-map-generation", "invalid-map"), + ] diff --git a/tests/interface/test_phase7_privacy_convergence.py b/tests/interface/test_phase7_privacy_convergence.py new file mode 100644 index 00000000..3737c031 --- /dev/null +++ b/tests/interface/test_phase7_privacy_convergence.py @@ -0,0 +1,380 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import hashlib +import logging +import pickle +from collections.abc import Iterator +from dataclasses import dataclass, fields, is_dataclass +from pathlib import Path +from unittest.mock import Mock + +import pandas as pd +import pytest + +from anonymizer.config.anonymizer_config import AnonymizerConfig +from anonymizer.config.replace_strategies import Substitute +from anonymizer.engine.constants import ( + COL_ATTEMPT_ID, + COL_DETECTED_ENTITIES, + COL_ENTITIES_BY_VALUE, + COL_FINAL_ENTITIES, + COL_PHASE7_CANDIDATE_REQUEST, + COL_PHASE7_INVOCATION_ID, + COL_REPLACEMENT_APPLICATION, + COL_REPLACEMENT_MAP, + COL_TAGGED_TEXT, + COL_TARGET_WORK_ID, + COL_TASK_ID, +) +from anonymizer.engine.detection.detection_workflow import EntityDetectionResult, EntityDetectionWorkflow +from anonymizer.engine.execution.phase7_runtime import _Phase7CleanupAttestation +from anonymizer.engine.ndd.adapter import RECORD_ID_COLUMN +from anonymizer.engine.replace.replace_runner import ReplacementResult, ReplacementWorkflow +from anonymizer.engine.replace.strategies import apply_replacement_map +from anonymizer.engine.rewrite.rewrite_workflow import RewriteWorkflow +from anonymizer.interface._protection import _Failed, _ProtectionFlow, _ProtectionPlan, _ProtectionSucceeded +from anonymizer.interface.anonymizer import Anonymizer +from anonymizer.measurement.collector import MeasurementCollector +from anonymizer.measurement.session import measurement_session +from tests.engine.execution.test_phase7_admission import _Proposal +from tests.engine.execution.test_phase7_ndd_backend import ( + _backend, + _dispatch, + _propose, + _request, + _ScriptedAdapter, + _success_response, +) +from tests.engine.execution.test_phase7_validation import _compiled_scope +from tests.interface.test_phase6_public_compatibility import _write_input +from tests.interface.test_phase7_private_wiring import _CandidateBackend, _private_substitute_flow +from tests.interface.test_private_protection import _AnchoredPhase6Backend, _record +from tests.streaming.structured_trace_prototype import build_synthetic_anonymizer + +_ORIGINAL = "P8-ORIGINAL-2f106d9b91b64e5988c4e4e4264cb7f1@example.test" +_SYNTHETIC = "P8-SYNTHETIC-a98765c2c93945d4a4d86a260267451b@example.test" +_PROMPT = "P8-PROMPT-51ed1eb892ca481483281142ccb3e109" +_SOURCE_ID = "P8-SOURCE-ID-7ba6ec61e2aa43c289f28612f895b607" +_DIGEST_MATERIAL = "P8-DIGEST-MATERIAL-2cbabdb08e10433e97cba79b2a5e23fe" +_CONTENT_DIGESTS = tuple( + hashlib.sha256(value.encode("utf-8")).hexdigest() + for value in (_ORIGINAL, _SYNTHETIC, _PROMPT, _SOURCE_ID, _DIGEST_MATERIAL) +) + + +def _leaf_paths(value: object, path: str = "root") -> Iterator[tuple[str, object]]: + if is_dataclass(value) and not isinstance(value, type): + for field in fields(value): + yield from _leaf_paths(getattr(value, field.name), f"{path}.{field.name}") + return + if isinstance(value, dict): + for key, item in value.items(): + yield from _leaf_paths(item, f"{path}[{key!r}]") + return + if isinstance(value, (list, tuple)): + for index, item in enumerate(value): + yield from _leaf_paths(item, f"{path}[{index}]") + return + yield path, value + + +def _frame_leaf_paths(frame: pd.DataFrame, name: str) -> Iterator[tuple[str, object]]: + for row_position in range(len(frame)): + for column in frame.columns: + yield from _leaf_paths(frame.iloc[row_position][column], f"{name}[{row_position}].{column}") + + +def _assert_forbidden_content_absent(value: object) -> None: + rendered = repr(value) + for forbidden in (_ORIGINAL, _SYNTHETIC, _PROMPT, _DIGEST_MATERIAL, *_CONTENT_DIGESTS): + assert forbidden not in rendered + + +def _public_substitute_anonymizer() -> Anonymizer: + detection = Mock(spec=EntityDetectionWorkflow) + + def detect(dataframe: pd.DataFrame, **_kwargs: object) -> EntityDetectionResult: + output = dataframe.copy() + entity = { + "id": "public-entity", + "value": _ORIGINAL, + "label": "email", + "start_position": 0, + "end_position": len(_ORIGINAL), + "score": 1.0, + "source": "privacy-test", + } + output[COL_DETECTED_ENTITIES] = [{"entities": [entity]}] + output[COL_FINAL_ENTITIES] = [{"entities": [entity]}] + output[COL_ENTITIES_BY_VALUE] = [{"entities_by_value": [{"value": _ORIGINAL, "labels": ["email"]}]}] + output[COL_TAGGED_TEXT] = [f"{_ORIGINAL}"] + return EntityDetectionResult(output, []) + + detection.run.side_effect = detect + replacement = Mock(spec=ReplacementWorkflow) + + def replace(dataframe: pd.DataFrame, **_kwargs: object) -> ReplacementResult: + output = dataframe.copy() + output[COL_REPLACEMENT_MAP] = [ + {"replacements": [{"original": _ORIGINAL, "label": "email", "synthetic": _SYNTHETIC}]} + ] + return ReplacementResult(apply_replacement_map(output), []) + + replacement.run.side_effect = replace + return Anonymizer( + data_designer=Mock(), + detection_workflow=detection, + replace_runner=replacement, + rewrite_runner=Mock(spec=RewriteWorkflow), + ) + + +def test_private_release_and_serialization_have_an_exact_content_allowlist( + caplog: pytest.LogCaptureFixture, +) -> None: + flow, backend = _private_substitute_flow(original=_ORIGINAL, synthetic=_SYNTHETIC, label="email") + collector = MeasurementCollector(run_id="phase7-p8-private-release") + + with caplog.at_level(logging.DEBUG), measurement_session(collector): + result = flow.protect((_record(_SOURCE_ID, _ORIGINAL),)) + + outcome = result.outcomes[0] + assert isinstance(outcome, _ProtectionSucceeded) + leaves = list(_leaf_paths(result)) + assert {path for path, value in leaves if value == _SYNTHETIC} == {"root.outcomes[0].output"} + assert {path for path, value in leaves if value == _SOURCE_ID} == {"root.outcomes[0].ref.value"} + assert not {path for path, value in leaves if value == _ORIGINAL} + assert set(field.name for field in fields(outcome.receipt)) == { + "contract_version", + "profile", + "implementation_version", + "terminal_accounting_verified", + "accepted_detections_verified", + } + assert backend.value is None + + restored = pickle.loads(pickle.dumps(result)) + restored_leaves = list(_leaf_paths(restored)) + assert {path for path, value in restored_leaves if value == _SYNTHETIC} == {"root.outcomes[0].output"} + assert {path for path, value in restored_leaves if value == _SOURCE_ID} == {"root.outcomes[0].ref.value"} + diagnostic_surfaces = ( + caplog.text, + collector.records, + outcome.receipt, + repr(result), + ) + for surface in diagnostic_surfaces: + _assert_forbidden_content_absent(surface) + assert _SOURCE_ID not in repr(surface) + allowed_metric_fields = { + "boundary", + "byte_count_bucket", + "cleanup", + "context_count_bucket", + "duration_sec", + "event", + "implementation_profile", + "observation_schema", + "outcome", + "reason", + "reconciliation", + "record_type", + "route", + "run_id", + "run_tags", + "schema_version", + "semantic_profile", + "target_count_bucket", + "timestamp_unix_sec", + } + assert collector.records + assert all(set(record) <= allowed_metric_fields for record in collector.records) + + +@dataclass +class _CleanupFailingBackend(_CandidateBackend): + def discard_values(self) -> None: + super().discard_values() + try: + raise ValueError(_ORIGINAL) + except ValueError as cause: + raise RuntimeError(_PROMPT) from cause + + +def test_cleanup_error_and_withheld_result_expose_no_candidate_or_exception_chain( + caplog: pytest.LogCaptureFixture, +) -> None: + anonymizer = build_synthetic_anonymizer({_ORIGINAL: "email"}) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Substitute(), emit_telemetry=False)) + assert isinstance(plan, _ProtectionPlan) + backend = _CleanupFailingBackend(_SYNTHETIC) + flow = _ProtectionFlow( + anonymizer, + plan, + phase6_backend=_AnchoredPhase6Backend.from_entities({_ORIGINAL: "email"}), + phase7_backend=backend, + ) + collector = MeasurementCollector(run_id="phase7-p8-cleanup-failure") + + with caplog.at_level(logging.DEBUG), measurement_session(collector): + result = flow.protect((_record(_SOURCE_ID, _ORIGINAL),)) + + outcome = result.outcomes[0] + assert isinstance(outcome, _Failed) + assert set(field.name for field in fields(outcome)) == {"ref", "failure"} + assert not hasattr(outcome, "output") + assert not hasattr(outcome, "receipt") + assert backend.value is None + leaves = list(_leaf_paths(result)) + assert {path for path, value in leaves if value == _SOURCE_ID} == {"root.outcomes[0].ref.value"} + assert not any(isinstance(value, BaseException) for _path, value in leaves) + for surface in (caplog.text, collector.records, outcome.failure, repr(result), pickle.dumps(result)): + _assert_forbidden_content_absent(surface) + + +def test_active_workframe_planner_state_and_cleanup_follow_structural_allowlists() -> None: + manifest, handoffs = _compiled_scope( + (_ORIGINAL,), + (("target-0",),), + {"target-0": (_Proposal(_ORIGINAL, "email", "email-cluster"),)}, + ) + adapter = _ScriptedAdapter(_success_response({"email_address": _SYNTHETIC})) + backend = _backend( + adapter, + "task-correlation-33d423175b2a48c1a21fba6f202231fb", + "slot-correlation-6a041a838dc8460a804349ff59e58fcb", + ) + + result = _propose( + backend, + manifest, + handoffs, + _dispatch( + invocation="invocation-correlation-bf56bebc0644465ea109b1daec9633ff", + attempt="attempt-correlation-9a3bf13e16034c0984dcc71a5d971476", + row="row-correlation-85ca65005ce24fdb98eddf264e352bc3", + ), + ) + + frame = adapter.calls[0] + assert set(frame.columns) == { + COL_TARGET_WORK_ID, + COL_PHASE7_INVOCATION_ID, + COL_TASK_ID, + COL_ATTEMPT_ID, + COL_PHASE7_CANDIDATE_REQUEST, + RECORD_ID_COLUMN, + } + request = _request(frame) + assert set(request) == {"schema_version", "slots", "required_distinct_pairs", "relations"} + assert len(request["slots"]) == 1 + assert set(request["slots"][0]) == {"slot_token", "role", "format", "mask", "source_values"} + request_leaves = list(_leaf_paths(request, "request")) + assert {path for path, value in request_leaves if value == _ORIGINAL} == {"request['slots'][0]['source_values'][0]"} + for forbidden in (_SYNTHETIC, _PROMPT, _SOURCE_ID, _DIGEST_MATERIAL, *_CONTENT_DIGESTS): + assert forbidden not in repr(request) + + active = backend._planner.current(manifest.id) + active_leaves = list(_leaf_paths(active, "planner")) + assert {path for path, value in active_leaves if value == _SYNTHETIC} == {"planner.value.assignments[0].value"} + assert not any(value == _ORIGINAL for _path, value in active_leaves) + result_leaves = list(_leaf_paths(result, "candidate")) + assert {path for path, value in result_leaves if value == _SYNTHETIC} == {"candidate.assignments[0].value"} + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(result) + + backend.close() + backend.discard_values() + cleanup_identity = object() + attestation = backend.cleanup_attestation(cleanup_identity) + + assert isinstance(attestation, _Phase7CleanupAttestation) + assert attestation.verified + assert attestation.cleanup_identity is cleanup_identity + assert backend._planner.cleanup_observation() == (0, 0, False) + retired = backend._planner.current(manifest.id) + assert retired is not None + assert retired.value is None + + +def test_public_result_trace_and_serialization_exclude_every_private_identity_and_digest( + tmp_path: Path, +) -> None: + source = pd.DataFrame({"record_id": [_SOURCE_ID], "text": [_ORIGINAL]}) + data = _write_input(source, tmp_path / "input.parquet", "parquet") + anonymizer = _public_substitute_anonymizer() + private_plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Substitute(), emit_telemetry=False)) + assert isinstance(private_plan, _ProtectionPlan) + assert private_plan.phase7_contract is not None + + result = anonymizer.run( + config=AnonymizerConfig(replace=Substitute(instructions=_PROMPT), emit_telemetry=False), + data=data, + ) + + assert {field.name for field in fields(result)} == { + "dataframe", + "trace_dataframe", + "resolved_text_column", + "failed_records", + "replace_method", + "rewrite_config", + "entity_labels", + "data_summary", + "_display_cycle_index", + } + assert list(result.dataframe.columns) == ["text", COL_FINAL_ENTITIES, "text_with_spans", "text_replaced"] + assert list(result.trace_dataframe.columns) == [ + "record_id", + "text", + COL_DETECTED_ENTITIES, + COL_FINAL_ENTITIES, + COL_ENTITIES_BY_VALUE, + "text_with_spans", + COL_REPLACEMENT_MAP, + "text_replaced", + COL_REPLACEMENT_APPLICATION, + ] + leaves = [ + *list(_frame_leaf_paths(result.dataframe, "dataframe")), + *list(_frame_leaf_paths(result.trace_dataframe, "trace")), + ] + synthetic_paths = {path for path, value in leaves if value == _SYNTHETIC} + assert synthetic_paths == { + "dataframe[0].text_replaced", + "trace[0]._replacement_map['replacements'][0]['synthetic']", + "trace[0].text_replaced", + } + source_paths = {path for path, value in leaves if value == _SOURCE_ID} + assert source_paths == {"trace[0].record_id"} + allowed_original_columns = { + "text", + COL_DETECTED_ENTITIES, + COL_FINAL_ENTITIES, + COL_ENTITIES_BY_VALUE, + "text_with_spans", + COL_REPLACEMENT_MAP, + } + assert all( + path.rsplit(".", 1)[-1].split("[", 1)[0] in allowed_original_columns + for path, value in leaves + if value == _ORIGINAL + ) + + serialized = pickle.loads(pickle.dumps(result)) + assert list(serialized.dataframe.columns) == list(result.dataframe.columns) + assert list(serialized.trace_dataframe.columns) == list(result.trace_dataframe.columns) + rendered = (result.dataframe.to_json() or "") + (result.trace_dataframe.to_json() or "") + repr(serialized) + for forbidden in ( + _PROMPT, + _DIGEST_MATERIAL, + private_plan.digest, + private_plan.phase7_contract.digest, + *_CONTENT_DIGESTS, + ): + assert forbidden not in rendered + for column in (*result.dataframe.columns, *result.trace_dataframe.columns): + assert not any(token in column for token in ("phase7", "scope", "slot", "task", "attempt", "bundle", "digest")) diff --git a/tests/interface/test_phase7_private_wiring.py b/tests/interface/test_phase7_private_wiring.py new file mode 100644 index 00000000..b782ab5f --- /dev/null +++ b/tests/interface/test_phase7_private_wiring.py @@ -0,0 +1,200 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path + +import pytest + +import anonymizer.interface._protection as protection_module +from anonymizer.config.anonymizer_config import AnonymizerConfig, Rewrite +from anonymizer.config.replace_strategies import Annotate, Hash, Redact, Substitute +from anonymizer.engine.execution.phase7_admission import _ScopeManifest +from anonymizer.engine.execution.phase7_ndd_backend import _Phase7NddResult, _Phase7NddStatus +from anonymizer.engine.execution.phase7_runtime import _Phase7CleanupAttestation +from anonymizer.engine.execution.phase7_validation import _CandidateAssignment +from anonymizer.interface._protection import ( + _BatchFailureCode, + _Failed, + _NoAcceptedDetections, + _PlanRejected, + _PlanUnsupported, + _ProtectionApplied, + _ProtectionBatchError, + _ProtectionFlow, + _ProtectionPlan, + _ProtectionSucceeded, +) +from anonymizer.interface.anonymizer import Anonymizer +from tests.interface.test_phase6_public_compatibility import _base_dataframe, _synthetic_anonymizer, _write_input +from tests.interface.test_private_protection import _AnchoredPhase6Backend, _record +from tests.streaming.structured_trace_prototype import build_synthetic_anonymizer + + +@dataclass +class _CandidateBackend: + value: str | None + calls: int = 0 + closed: int = 0 + discarded: int = 0 + + def propose_scope( + self, + manifest: object, + handoffs: object, + contract: object, + dispatch: object, + ) -> _Phase7NddResult: + del handoffs, contract + assert isinstance(manifest, _ScopeManifest) + assert dispatch is not None + assert isinstance(self.value, str) + self.calls += 1 + return _Phase7NddResult( + _Phase7NddStatus.CANDIDATE, + tuple(_CandidateAssignment(slot.id, self.value) for slot in manifest.slots), + ) + + def close(self) -> None: + self.closed += 1 + + def discard_values(self) -> None: + self.discarded += 1 + self.value = None + + def cleanup_attestation(self, cleanup_identity: object) -> object: + return _Phase7CleanupAttestation( + "phase7-cleanup-attestation/v1", + True, + 0, + 0, + True, + 0, + False, + cleanup_identity, + ) + + +def _private_substitute_flow( + *, + original: str, + synthetic: str, + label: str = "first_name", +) -> tuple[_ProtectionFlow, _CandidateBackend]: + anonymizer = build_synthetic_anonymizer({original: label}) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Substitute(), emit_telemetry=False)) + assert isinstance(plan, _ProtectionPlan) + backend = _CandidateBackend(synthetic) + flow = protection_module._ProtectionFlow( + anonymizer, + plan, + phase6_backend=_AnchoredPhase6Backend.from_entities({original: label}), + phase7_backend=backend, + ) + return flow, backend + + +def test_private_substitute_plan_selects_phase7_service_without_changing_other_profiles() -> None: + anonymizer = build_synthetic_anonymizer({}) + + substitute = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Substitute(), emit_telemetry=False)) + redact = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + + assert isinstance(substitute, _ProtectionPlan) + assert substitute.profile == "stable-substitute-v1" + assert type(anonymizer._open_protection_flow(substitute)._runtime).__name__ == "_Phase7SubstituteProtectionService" + assert isinstance(redact, _ProtectionPlan) + assert redact.profile == "redact-release-v1" + assert type(anonymizer._open_protection_flow(redact)._runtime).__name__ == "_Phase6RedactProtectionService" + assert isinstance(anonymizer._compile_protection_plan(AnonymizerConfig(replace=Annotate())), _PlanRejected) + assert isinstance(anonymizer._compile_protection_plan(AnonymizerConfig(replace=Hash())), _PlanUnsupported) + assert isinstance(anonymizer._compile_protection_plan(AnonymizerConfig(rewrite=Rewrite())), _PlanUnsupported) + assert isinstance( + anonymizer._compile_protection_plan(AnonymizerConfig(replace=Substitute(instructions="legacy only"))), + _PlanRejected, + ) + + +def test_private_substitute_releases_only_a_qualified_phase7_output() -> None: + flow, backend = _private_substitute_flow(original="Alice", synthetic="Avery") + + result = flow.protect((_record("source-a", "Alice"),)) + + outcome = result.outcomes[0] + assert isinstance(outcome, _ProtectionSucceeded) + assert isinstance(outcome.disposition, _ProtectionApplied) + assert outcome.output == "Avery" + assert outcome.receipt.profile == "stable-substitute-v1" + assert backend.calls == 1 + assert backend.closed == 1 + assert backend.discarded == 1 + assert backend.value is None + + +def test_private_substitute_rejects_an_invalid_bundle_without_output() -> None: + flow, backend = _private_substitute_flow(original="Alice", synthetic="Alice") + + result = flow.protect((_record("source-a", "Alice"),)) + + outcome = result.outcomes[0] + assert isinstance(outcome, _Failed) + assert not hasattr(outcome, "output") + assert backend.calls == 1 + assert backend.closed == 1 + assert backend.discarded == 1 + assert backend.value is None + + +def test_private_substitute_no_entity_scope_bypasses_phase7_adapter() -> None: + flow, backend = _private_substitute_flow(original="absent", synthetic="Avery") + + result = flow.protect((_record("source-a", "ordinary text"),)) + + outcome = result.outcomes[0] + assert isinstance(outcome, _ProtectionSucceeded) + assert isinstance(outcome.disposition, _NoAcceptedDetections) + assert outcome.output == "ordinary text" + assert backend.calls == 0 + assert backend.closed == 1 + assert backend.discarded == 1 + + +def test_private_substitute_rejects_more_than_the_frozen_scope_limit_before_effects() -> None: + flow, backend = _private_substitute_flow(original="Alice", synthetic="Avery") + + with pytest.raises(_ProtectionBatchError) as exc_info: + flow.protect( + ( + _record("source-a", "Alice"), + _record("source-b", "Alice"), + _record("source-c", "Alice"), + ) + ) + + assert exc_info.value.code is _BatchFailureCode.TOO_MANY_RECORDS + assert backend.calls == 0 + assert backend.closed == 0 + assert backend.discarded == 0 + + +def test_public_substitute_never_compiles_or_opens_a_private_flow( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + input_data = _write_input(_base_dataframe(), tmp_path / "input.parquet", "parquet") + anonymizer = _synthetic_anonymizer() + + def private_path_forbidden(*_args: object, **_kwargs: object) -> object: + raise AssertionError("public Substitute entered the private graph profile") + + monkeypatch.setattr(Anonymizer, "_compile_protection_plan", private_path_forbidden) + monkeypatch.setattr(Anonymizer, "_open_protection_flow", private_path_forbidden) + + result = anonymizer.run( + config=AnonymizerConfig(replace=Substitute(), emit_telemetry=False), + data=input_data, + ) + + assert result.dataframe["text_replaced"].tolist() == ["Avery", "Blake", "Avery", "Casey"] diff --git a/tests/interface/test_private_protection.py b/tests/interface/test_private_protection.py new file mode 100644 index 00000000..0f00d240 --- /dev/null +++ b/tests/interface/test_private_protection.py @@ -0,0 +1,556 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import inspect +import threading +from dataclasses import FrozenInstanceError, replace +from pathlib import Path +from typing import Any, cast + +import pandas as pd +import pytest + +import anonymizer.interface._protection as protection_module +from anonymizer.config.anonymizer_config import AnonymizerConfig, AnonymizerInput, Rewrite +from anonymizer.config.replace_strategies import Annotate, Redact +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextBackendCapability, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.mention_admission import _ValidationDecision, _ValidationDecisionKind +from anonymizer.engine.execution.mention_resolution import _SubjectEvidence +from anonymizer.engine.execution.phase6_runtime import ( + _CandidateProposal, + _Phase6AugmentationWork, + _Phase6CandidateWork, + _Phase6ResolverWork, + _Phase6ValidationWork, +) +from anonymizer.engine.execution.protection_service import _Phase6RedactProtectionService +from anonymizer.interface._protection import ( + _BatchFailureCode, + _Failed, + _NoAcceptedDetections, + _PlanRejected, + _PlanUnsupported, + _ProtectionApplied, + _ProtectionBatchError, + _ProtectionReceipt, + _ProtectionRecord, + _ProtectionSucceeded, + _RecordRef, + _Rejected, + _TextSegment, +) +from anonymizer.interface.anonymizer import Anonymizer +from anonymizer.interface.cli import main as cli_main +from anonymizer.measurement import MeasurementCollector, measurement_session +from tests.streaming.structured_trace_prototype import build_synthetic_anonymizer + + +def _record(ref: str, text: str) -> _ProtectionRecord: + return _ProtectionRecord(_RecordRef(ref), (_TextSegment(text),)) + + +def _flow(*, entities: dict[str, str] | None = None): + entity_map = entities or {} + anonymizer = build_synthetic_anonymizer(entity_map) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + return anonymizer, protection_module._ProtectionFlow( + anonymizer, + plan, + phase6_backend=_AnchoredPhase6Backend.from_entities(entity_map), + ) + + +class _AnchoredPhase6Backend: + def __init__( + self, + proposals: dict[str, tuple[_CandidateProposal, ...]], + *, + entities: dict[str, str] | None = None, + fault_stage: str | None = None, + malformed_stage: str | None = None, + ) -> None: + self._proposals = proposals + self._entities = entities or {} + self._fault_stage = fault_stage + self._malformed_stage = malformed_stage + self.closed = False + self.calls: list[str] = [] + + @classmethod + def from_entities(cls, entities: dict[str, str]) -> _AnchoredPhase6Backend: + return cls({}, entities=entities) + + def context_capability(self) -> _ContextBackendCapability: + return _ContextBackendCapability( + _ContextProfile.TARGET_CONTEXT_V1, + _ContextSchemaVersion.V1, + _ContextLimits(128, 1_048_576, 16_384, 2_097_152), + True, + _ContextOrdering.DECLARED, + (_BackendArtifactClass.CONTEXT_REQUEST,), + _RetentionPosture.DISABLED, + ) + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + self._enter("detect") + if self._malformed_stage == "detect": + return cast(Any, (object(),)) + explicit = self._proposals.get(work.target.text) + if explicit is not None: + return explicit + proposals: list[_CandidateProposal] = [] + for value, label in self._entities.items(): + start = 0 + while (found := work.target.text.find(value, start)) >= 0: + proposals.append(_CandidateProposal(found, found + len(value), value, label)) + start = found + len(value) + return tuple(sorted(proposals, key=lambda proposal: (proposal.start, proposal.end))) + + def augment(self, work: _Phase6AugmentationWork) -> tuple[_CandidateProposal, ...]: + self._enter("augment") + return () + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + self._enter("validate") + if self._malformed_stage == "validate": + return cast(Any, ()) + return tuple( + _ValidationDecision(candidate.token, _ValidationDecisionKind.KEEP) for candidate in work.candidates + ) + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SubjectEvidence, ...]: + self._enter("resolve") + if self._malformed_stage == "resolve": + return cast(Any, (object(),)) + return () + + def close_phase6(self) -> bool: + self._enter("close") + self.closed = True + return self._malformed_stage != "close" + + def _enter(self, stage: str) -> None: + self.calls.append(stage) + if self._fault_stage == stage: + raise RuntimeError("BACKEND-SECRET-alice@example.test") + + +def test_private_redact_applies_and_no_detection_success_is_explicit() -> None: + _, flow = _flow(entities={"alice@example.test": "email"}) + result = flow.protect((_record("a", "mail alice@example.test"), _record("b", "ordinary text"))) + + applied, unchanged = result.outcomes + assert isinstance(applied, _ProtectionSucceeded) + assert isinstance(applied.disposition, _ProtectionApplied) + assert applied.output == "mail [REDACTED]" + assert isinstance(unchanged, _ProtectionSucceeded) + assert isinstance(unchanged.disposition, _NoAcceptedDetections) + assert unchanged.output == "ordinary text" + assert result.success_count == 2 + assert result.failure_count == 0 + assert not hasattr(result, "trace_dataframe") + assert "row_token" not in repr(result).lower() + + +def test_private_flow_has_an_engine_private_phase6_backend_seam() -> None: + parameters = inspect.signature(protection_module._ProtectionFlow).parameters + + assert "phase6_backend" in parameters + + +def test_default_private_flow_selects_phase6_redact_service() -> None: + anonymizer = build_synthetic_anonymizer({}) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + + flow = anonymizer._open_protection_flow(plan) + + assert isinstance(flow._runtime, _Phase6RedactProtectionService) + + +def test_private_flow_executes_phase6_anchor_reconstruction() -> None: + text = "Alice and Alice" + anonymizer = build_synthetic_anonymizer({"Alice": "name"}) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + backend = _AnchoredPhase6Backend({text: (_CandidateProposal(0, 5, "Alice", "name"),)}) + flow = protection_module._ProtectionFlow(anonymizer, plan, phase6_backend=backend) + + result = flow.protect((_record("a", text),)) + + outcome = result.outcomes[0] + assert isinstance(outcome, _ProtectionSucceeded) + assert isinstance(outcome.disposition, _ProtectionApplied) + assert outcome.output == "[REDACTED] and Alice" + assert backend.closed + assert "Alice" not in repr(result) + + +def test_phase5_adds_no_public_context_or_graph_parameters() -> None: + for entrypoint in (Anonymizer.run, Anonymizer.preview, Anonymizer.evaluate, cli_main.run, cli_main.preview): + public_parameters = inspect.signature(entrypoint).parameters + assert "context" not in public_parameters + assert "graph" not in public_parameters + + +def test_context_workframe_observations_are_paired_bounded_and_content_free() -> None: + target_canary = "TARGET-CANARY-alice@example.test" + source_canary = "SOURCE-CANARY-7843" + _, flow = _flow(entities={target_canary: "email"}) + collector = MeasurementCollector(run_id="phase5-observation-test") + + with measurement_session(collector): + result = flow.protect((_record(source_canary, target_canary),)) + + assert isinstance(result.outcomes[0], _ProtectionSucceeded) + records = [record for record in collector.records if record["record_type"] == "context_workframe"] + boundaries = {record["boundary"] for record in records} + assert { + "preflight", + "capability_recheck", + "workframe_construction", + "dispatch", + "backend_execution", + "reconciliation", + "cleanup", + "release", + }.issubset(boundaries) + for boundary in boundaries: + paired = [record for record in records if record["boundary"] == boundary] + assert [record["event"] for record in paired] == ["start", "terminal"] + assert paired[-1]["duration_sec"] >= 0 + assert isinstance(paired[-1]["target_count_bucket"], str) + assert isinstance(paired[-1]["context_count_bucket"], str) + rendered = repr(records) + assert target_canary not in rendered + assert source_canary not in rendered + assert "__anonymizer_private_row_correlation__" not in rendered + + +def test_throwing_context_observer_cannot_change_release() -> None: + class ThrowingCollector(MeasurementCollector): + def record(self, record_type: str, **fields: Any) -> None: + if record_type == "context_workframe": + raise RuntimeError("observer unavailable") + super().record(record_type, **fields) + + _, flow = _flow() + with measurement_session(ThrowingCollector()): + result = flow.protect((_record("a", "ordinary text"),)) + + assert isinstance(result.outcomes[0], _ProtectionSucceeded) + + +def test_base_exception_from_context_observer_cannot_change_release() -> None: + class InterruptingCollector(MeasurementCollector): + def record(self, record_type: str, **fields: Any) -> None: + if record_type == "context_workframe": + raise KeyboardInterrupt + super().record(record_type, **fields) + + _, flow = _flow() + with measurement_session(InterruptingCollector()): + result = flow.protect((_record("a", "ordinary text"),)) + + assert isinstance(result.outcomes[0], _ProtectionSucceeded) + + +def test_reentrant_context_observer_is_bounded_and_cannot_change_release() -> None: + _, flow = _flow() + + class ReentrantCollector(MeasurementCollector): + entered = False + + def record(self, record_type: str, **fields: Any) -> None: + if record_type == "context_workframe" and not self.entered: + self.entered = True + nested = flow.protect((_record("nested", "ordinary text"),)) + assert isinstance(nested.outcomes[0], _ProtectionSucceeded) + super().record(record_type, **fields) + + collector = ReentrantCollector() + with measurement_session(collector): + result = flow.protect((_record("outer", "ordinary text"),)) + + assert isinstance(result.outcomes[0], _ProtectionSucceeded) + context_records = [record for record in collector.records if record["record_type"] == "context_workframe"] + assert 0 < len(context_records) <= 16 + assert all( + sum(record["boundary"] == boundary and record["event"] == event for record in context_records) <= 1 + for boundary in {record["boundary"] for record in context_records} + for event in ("start", "terminal") + ) + + +def test_private_phase6_and_public_dataframe_redact_keep_their_distinct_compatibility_outputs(tmp_path: Path) -> None: + secret = "alice@example.test" + anonymizer = build_synthetic_anonymizer({secret: "email"}) + config = AnonymizerConfig(replace=Redact(), emit_telemetry=False) + source = tmp_path / "parity.csv" + pd.DataFrame({"text": [f"mail {secret}", "ordinary text"]}).to_csv(source, index=False) + + public = anonymizer.run(config=config, data=AnonymizerInput(source=str(source), text_column="text")) + plan = anonymizer._compile_protection_plan(config) + private = protection_module._ProtectionFlow( + anonymizer, + plan, + phase6_backend=_AnchoredPhase6Backend.from_entities({secret: "email"}), + ).protect((_record("a", f"mail {secret}"), _record("b", "ordinary text"))) + + assert public.dataframe["text_replaced"].tolist() == ["mail [REDACTED_EMAIL]", "ordinary text"] + assert [cast(_ProtectionSucceeded, outcome).output for outcome in private.outcomes] == [ + "mail [REDACTED]", + "ordinary text", + ] + + +def test_graph_outcomes_are_rejoined_by_private_datum_identity() -> None: + secret = "alice@example.test" + _, flow = _flow(entities={secret: "email"}) + original = flow._runtime.protect + + def reordered(*args: Any, **kwargs: Any): + result = original(*args, **kwargs) + return replace(result, outcomes=tuple(reversed(result.outcomes))) + + flow._runtime.protect = reordered + result = flow.protect((_record("private-a", secret), _record("private-b", "plain"))) + + first, second = result.outcomes + assert isinstance(first, _ProtectionSucceeded) + assert first.ref.value == "private-a" + assert first.output == "[REDACTED]" + assert isinstance(second, _ProtectionSucceeded) + assert second.ref.value == "private-b" + assert second.output == "plain" + + +def test_compilation_is_closed_and_plan_is_content_free_and_immutable() -> None: + anonymizer = build_synthetic_anonymizer({}) + assert isinstance(anonymizer._compile_protection_plan(AnonymizerConfig(replace=Annotate())), _PlanRejected) + assert isinstance(anonymizer._compile_protection_plan(AnonymizerConfig(rewrite=Rewrite())), _PlanUnsupported) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + assert "Redact" not in repr(plan) + with pytest.raises(FrozenInstanceError): + setattr(plan, "profile", "changed") + + +def test_plan_snapshot_detects_nested_tampering_and_digest_covers_models() -> None: + anonymizer = build_synthetic_anonymizer({}) + config = AnonymizerConfig(replace=Redact(), emit_telemetry=False) + first = anonymizer._compile_protection_plan(config) + cast(Redact, config.replace).format_template = "<{label}>" + assert first.invocation.replace_method.format_template == "[REDACTED_{label}]" + + anonymizer._selected_models.detection.entity_detector = "materially-different" + second = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + assert first.digest != second.digest + second.invocation.selected_models.detection.entity_detector = "tampered-after-compile" + outcome = anonymizer._open_protection_flow(second).protect((_record("a", "text"),)).outcomes[0] + assert isinstance(outcome, _Failed) + + +def test_plan_digest_separates_model_config_and_replacement_semantics() -> None: + anonymizer = build_synthetic_anonymizer({}) + baseline = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + anonymizer._model_configs[0].model = "materially-different-model" + different_model = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + assert baseline.digest != different_model.digest + + rejected_profile = anonymizer._compile_protection_plan( + AnonymizerConfig(replace=Redact(format_template="<{label}>", normalize_label=True), emit_telemetry=False) + ) + assert isinstance(rejected_profile, _PlanRejected) + + +def test_receipt_binds_plan_digest_and_fresh_attempt_identity() -> None: + _, flow = _flow() + first = cast(_ProtectionSucceeded, flow.protect((_record("a", "text"),)).outcomes[0]).receipt + second = cast(_ProtectionSucceeded, flow.protect((_record("a", "text"),)).outcomes[0]).receipt + assert isinstance(first, _ProtectionReceipt) + assert isinstance(second, _ProtectionReceipt) + assert first.plan_digest == second.plan_digest + assert first.attempt_id != second.attempt_id + + +@pytest.mark.parametrize( + "records, code", + [ + ((_record("same", "a"), _record("same", "b")), _BatchFailureCode.DUPLICATE_REF), + ((_record("a", "x" * 32_769),), _BatchFailureCode.RECORD_TOO_LARGE), + (tuple(_record(str(index), "x" * 32_768) for index in range(33)), _BatchFailureCode.BATCH_TOO_LARGE), + (tuple(_record(str(index), "x") for index in range(129)), _BatchFailureCode.TOO_MANY_RECORDS), + ((_ProtectionRecord(_RecordRef("a"), ()),), _BatchFailureCode.UNSUPPORTED_CARDINALITY), + ( + (_ProtectionRecord(cast(Any, "secret@example.test"), (_TextSegment("text"),)),), + _BatchFailureCode.MALFORMED_BATCH, + ), + ], +) +def test_outer_batch_is_rejected_before_admission(records: object, code: _BatchFailureCode) -> None: + _, flow = _flow() + with pytest.raises(_ProtectionBatchError) as exc_info: + flow.protect(cast(Any, records)) + assert exc_info.value.code is code + assert repr(exc_info.value) == "" + + +def test_graph_admission_rejects_before_phase6_backend_execution(monkeypatch: pytest.MonkeyPatch) -> None: + anonymizer = build_synthetic_anonymizer({}) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + backend = _AnchoredPhase6Backend.from_entities({}) + flow = protection_module._ProtectionFlow(anonymizer, plan, phase6_backend=backend) + collector = MeasurementCollector() + + monkeypatch.setattr(protection_module, "_trivial_graph", lambda _datums: object()) + + with measurement_session(collector): + result = flow.protect((_record("a", "text"),)) + + assert isinstance(result.outcomes[0], _Failed) + assert backend.calls == [] + observations = [record for record in collector.records if record["record_type"] == "context_workframe"] + assert [(record["boundary"], record["event"]) for record in observations] == [ + ("preflight", "start"), + ("preflight", "terminal"), + ] + + +def test_malformed_nested_segment_is_rejected_before_admission() -> None: + forged = object.__new__(_ProtectionRecord) + object.__setattr__(forged, "ref", _RecordRef("a")) + object.__setattr__(forged, "segments", (object(),)) + + _, flow = _flow() + with pytest.raises(_ProtectionBatchError) as exc_info: + flow.protect((forged,)) + assert exc_info.value.code is _BatchFailureCode.MALFORMED_BATCH + assert str(exc_info.value) == "private protection batch rejected" + assert repr(exc_info.value) == "" + + +def test_missing_nested_record_attributes_are_sanitized_batch_rejections() -> None: + _, flow = _flow() + missing_record = object.__new__(_ProtectionRecord) + missing_ref = object.__new__(_RecordRef) + missing_segment = object.__new__(_TextSegment) + forged_values = ( + missing_record, + _ProtectionRecord(missing_ref, (_TextSegment("text"),)), + _ProtectionRecord(_RecordRef("a"), (missing_segment,)), + ) + for forged in forged_values: + with pytest.raises(_ProtectionBatchError, match="private protection batch rejected"): + flow.protect((forged,)) + + +def test_invalid_ref_is_bounded_and_content_free() -> None: + secret = "secret@example.test" + with pytest.raises(ValueError) as exc_info: + _RecordRef(secret * 30) + assert secret not in str(exc_info.value) + + +def test_cancel_before_admission_and_overlap_are_rejections() -> None: + _, flow = _flow() + cancelled = flow.protect((_record("a", "text"),), cancelled_before_admission=True) + assert isinstance(cancelled.outcomes[0], _Rejected) + assert cancelled.outcomes[0].failure.code.value == "cancelled_before_admission" + + entered = threading.Event() + release = threading.Event() + original = flow._runtime.protect + + def blocked(*args: Any, **kwargs: Any): + entered.set() + release.wait(timeout=5) + return original(*args, **kwargs) + + flow._runtime.protect = blocked + worker = threading.Thread(target=lambda: flow.protect((_record("one", "text"),))) + worker.start() + assert entered.wait(timeout=5) + busy = flow.protect((_record("two", "text"),)) + release.set() + worker.join(timeout=5) + assert isinstance(busy.outcomes[0], _Rejected) + assert busy.outcomes[0].failure.code.value == "busy" + + +@pytest.mark.parametrize("stage", ["detect", "augment", "validate", "resolve"]) +def test_phase6_backend_faults_have_exact_safe_terminal_accounting(stage: str) -> None: + secret = "alice@example.test" + anonymizer = build_synthetic_anonymizer({secret: "email"}) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + backend = _AnchoredPhase6Backend.from_entities({secret: "email"}) + backend._fault_stage = stage + flow = protection_module._ProtectionFlow(anonymizer, plan, phase6_backend=backend) + run = flow.protect((_record("a", secret), _record("b", "plain"))) + + assert len(run.outcomes) == 2 + assert [outcome.ref.value for outcome in run.outcomes] == ["a", "b"] + assert all(isinstance(outcome, _Failed) for outcome in run.outcomes) + assert secret not in repr(run) + + +def test_invocation_failure_suppresses_cause_and_emits_no_output() -> None: + secret = "provider secret alice@example.test" + anonymizer = build_synthetic_anonymizer({}) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + backend = _AnchoredPhase6Backend({}, fault_stage="detect") + flow = protection_module._ProtectionFlow(anonymizer, plan, phase6_backend=backend) + run = flow.protect((_record("private-ref", "raw input"),)) + outcome = run.outcomes[0] + assert isinstance(outcome, _Failed) + assert not hasattr(outcome, "output") + assert secret not in repr(outcome) + assert "private-ref" not in repr(outcome) + + +def test_private_failure_logs_exclude_backend_exception_canary(caplog: pytest.LogCaptureFixture) -> None: + backend_canary = "BACKEND-SECRET-alice@example.test" + anonymizer = build_synthetic_anonymizer({}) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + backend = _AnchoredPhase6Backend({}, fault_stage="detect") + flow = protection_module._ProtectionFlow(anonymizer, plan, phase6_backend=backend) + + run = flow.protect((_record("a", "text"),)) + assert isinstance(run.outcomes[0], _Failed) + rendered = repr(run) + "\n" + "\n".join(record.getMessage() for record in caplog.records) + assert backend_canary not in rendered + + +@pytest.mark.parametrize("stage", ["detect", "validate", "resolve", "close"]) +def test_malformed_phase6_stage_result_fails_closed(stage: str) -> None: + anonymizer = build_synthetic_anonymizer({"text": "label"}) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + backend = _AnchoredPhase6Backend.from_entities({"text": "label"}) + backend._malformed_stage = stage + flow = protection_module._ProtectionFlow(anonymizer, plan, phase6_backend=backend) + + assert isinstance(flow.protect((_record("a", "text"),)).outcomes[0], _Failed) + + +def test_failure_retry_taxonomy_is_unassigned_and_unknown() -> None: + _, flow = _flow() + flow.close() + failure = cast(_Rejected, flow.protect((_record("a", "text"),)).outcomes[0]).failure + assert failure.retry_safety.value == "unknown" + assert failure.retry_owner.value == "unassigned" + + +def test_close_is_idempotent_and_does_not_close_borrowed_anonymizer() -> None: + anonymizer, flow = _flow() + flow.close() + flow.close() + rejected = flow.protect((_record("a", "text"),)) + assert isinstance(rejected.outcomes[0], _Rejected) + assert anonymizer.run is not None diff --git a/tests/streaming/__init__.py b/tests/streaming/__init__.py new file mode 100644 index 00000000..52a7a9da --- /dev/null +++ b/tests/streaming/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/tests/streaming/intake_format_validation.py b/tests/streaming/intake_format_validation.py new file mode 100644 index 00000000..94bf5647 --- /dev/null +++ b/tests/streaming/intake_format_validation.py @@ -0,0 +1,547 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Test-only Intake format validation probes.""" + +from __future__ import annotations + +import json +from collections.abc import Callable, Mapping +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Protocol, cast + +import pandas as pd +from google.protobuf.message import DecodeError +from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest + +from anonymizer.engine.constants import COL_FINAL_ENTITIES, COL_TEXT +from anonymizer.interface._protection import ( + _ProtectionRecord, + _ProtectionRunRecord, + _ProtectionSucceeded, + _RecordRef, + _TextSegment, +) +from tests.streaming.structured_trace_prototype import ( + PROTECTED_TEXT_COLUMN, + SEGMENT_KEY_COLUMN, + CodecBounds, + FieldRole, + ProjectedItem, + SourceFormat, + StructuredItemError, + TraceMapping, + project_complete_item, + reconstruct_complete_item, +) + +Emitter = Callable[[bytes], None] +_JSON_BOUNDS = CodecBounds( + max_bytes=1_048_576, + max_depth=32, + max_targets=128, + max_scalars=512, + max_scalar_bytes=65_536, + max_events=2_048, +) +_ATIF_VERSIONS = {"ATIF-v1.0", "ATIF-v1.7"} +_CHAT_ROLES = {"user", "system", "assistant", "developer", "tool", "function"} +_OTLP_TARGET_ATTRIBUTES = {"input.value", "output.value", "exception.message"} +_OTLP_STRUCTURAL_ATTRIBUTES = { + "openinference.span.kind", + "gen_ai.agent.name", + "gen_ai.conversation.id", + "gen_ai.system", + "gen_ai.request.model", + "gen_ai.usage.input_tokens", + "gen_ai.usage.output_tokens", + "gen_ai.usage.total_tokens", + "nemo.evaluation.name", + "nemo.test_case.id", +} + + +class _PlanAFlow(Protocol): + def protect(self, records: tuple[_ProtectionRecord, ...]) -> _ProtectionRunRecord: ... + + +class IntakeValidationError(RuntimeError): + """Sanitized validation-only complete-item rejection.""" + + +def protect_atif(source: bytes, *, flow: _PlanAFlow, emit: Emitter) -> bytes: + try: + document = _json_object(source) + _validate_atif(document) + mapping = _mapping(document, version=f"intake-{document['schema_version']}", kind="atif") + return _protect_json(source, mapping=mapping, flow=flow, emit=emit) + except IntakeValidationError: + raise + except Exception: + raise IntakeValidationError("ATIF item rejected") from None + + +def protect_chat_completion(source: bytes, *, flow: _PlanAFlow, emit: Emitter) -> bytes: + try: + document = _json_object(source) + _validate_chat_completion(document) + mapping = _mapping(document, version="intake-chat-completion-v1", kind="chat") + return _protect_json(source, mapping=mapping, flow=flow, emit=emit) + except IntakeValidationError: + raise + except Exception: + raise IntakeValidationError("chat completion rejected") from None + + +def _protect_json(source: bytes, *, mapping: TraceMapping, flow: _PlanAFlow, emit: Emitter) -> bytes: + try: + projected = project_complete_item( + source, + source_format=SourceFormat.JSON, + mapping=mapping, + bounds=_JSON_BOUNDS, + ) + protected = _protect_projected(projected, flow=flow) + except IntakeValidationError: + raise + except StructuredItemError: + raise IntakeValidationError("structured item rejected") from None + emit(protected) + return protected + + +def _protect_projected(projected: ProjectedItem, *, flow: _PlanAFlow) -> bytes: + frame = projected.dataframe + records = tuple( + _ProtectionRecord(_RecordRef(str(row[SEGMENT_KEY_COLUMN])), (_TextSegment(str(row[COL_TEXT])),)) + for _, row in frame.iterrows() + ) + run = flow.protect(records) + if len(run.outcomes) != len(records) or not all(isinstance(item, _ProtectionSucceeded) for item in run.outcomes): + raise IntakeValidationError("protection failed") + result = _outcome_dataframe(cast(tuple[_ProtectionSucceeded, ...], run.outcomes)) + try: + return reconstruct_complete_item(projected, result) + except StructuredItemError: + raise IntakeValidationError("protection failed") from None + + +def _outcome_dataframe(outcomes: tuple[_ProtectionSucceeded, ...]) -> pd.DataFrame: + return pd.DataFrame( + { + SEGMENT_KEY_COLUMN: [outcome.ref.value for outcome in outcomes], + PROTECTED_TEXT_COLUMN: [outcome.output for outcome in outcomes], + COL_FINAL_ENTITIES: [{"entities": []} for _ in outcomes], + } + ) + + +def _json_object(source: bytes) -> dict[str, Any]: + value = json.loads(source) + if not isinstance(value, dict): + raise ValueError("object required") + return cast(dict[str, Any], value) + + +def _mapping(document: dict[str, Any], *, version: str, kind: str) -> TraceMapping: + roles: dict[str, FieldRole] = {} + for pointer, value in _scalars(document): + roles[pointer] = _atif_role(pointer, value) if kind == "atif" else _chat_role(pointer, value) + identity = "/session_id" + ordered = ("/trajectory_id",) if kind == "atif" else ("/trace_id",) + return TraceMapping( + version=version, fields=roles, source_identity_pointer=identity, ordered_identity_pointers=ordered + ) + + +def _scalars(value: object, pointer: str = "") -> list[tuple[str, object]]: + if isinstance(value, dict): + return [item for key, child in value.items() for item in _scalars(child, f"{pointer}/{_escape(key)}")] + if isinstance(value, list): + return [item for index, child in enumerate(value) for item in _scalars(child, f"{pointer}/{index}")] + return [(pointer, value)] + + +def _escape(value: str) -> str: + return value.replace("~", "~0").replace("/", "~1") + + +def _atif_role(pointer: str, value: object) -> FieldRole: + tokens = pointer.split("/")[1:] + if pointer in {"/session_id", "/trajectory_id"}: + return FieldRole.STRUCTURAL + if isinstance(value, str) and _atif_target(tokens): + return FieldRole.TARGET + if "extra" in tokens and isinstance(value, str): + raise ValueError("string extension lacks reviewed field policy") + return FieldRole.PRESERVE + + +def _atif_target(tokens: list[str]) -> bool: + if tokens == ["notes"]: + return True + if len(tokens) >= 3 and tokens[0] == "steps" and tokens[2] == "message": + return tokens[-1] in {"message", "text"} + if len(tokens) == 3 and tokens[0] == "steps" and tokens[2] == "reasoning_content": + return True + if "tool_calls" in tokens and "arguments" in tokens: + return True + return "observation" in tokens and "content" in tokens and tokens[-1] in {"content", "text"} + + +def _chat_role(pointer: str, value: object) -> FieldRole: + tokens = pointer.split("/")[1:] + if pointer in {"/session_id", "/trace_id", "/response/id", "/response/created"}: + return FieldRole.STRUCTURAL + if _chat_target(tokens, value): + return FieldRole.TARGET + return FieldRole.PRESERVE + + +def _chat_target(tokens: list[str], value: object) -> bool: + if value is None: + return False + if len(tokens) == 4 and tokens[:2] == ["request", "messages"] and tokens[-1] == "content": + return True + if tokens[:2] == ["request", "messages"] and tokens[-1] == "arguments": + return True + if tokens[:2] == ["response", "choices"] and tokens[-1] == "content": + return True + return tokens[:2] == ["response", "error"] and tokens[-1] == "message" + + +def _validate_atif(document: dict[str, Any]) -> None: + _require_keys(document, {"schema_version", "session_id", "trajectory_id", "agent", "steps"}) + allowed = { + "schema_version", + "session_id", + "trajectory_id", + "agent", + "steps", + "notes", + "final_metrics", + "continued_trajectory_ref", + "extra", + "subagent_trajectories", + "evaluation_context", + } + if set(document) - allowed or document["schema_version"] not in _ATIF_VERSIONS: + raise ValueError("unsupported ATIF shape") + _validate_atif_agent(document["agent"]) + steps = document["steps"] + if not isinstance(steps, list): + raise ValueError("steps must be a list") + for index, step in enumerate(steps, start=1): + _validate_atif_step(step, expected_id=index) + + +def _validate_atif_agent(value: object) -> None: + agent = _as_dict(value) + _require_keys(agent, {"name", "version"}) + if set(agent) - {"name", "version", "model_name", "tool_definitions", "extra"}: + raise ValueError("unknown agent field") + + +def _validate_atif_step(value: object, *, expected_id: int) -> None: + step = _as_dict(value) + _require_keys(step, {"step_id", "source", "message"}) + common = { + "step_id", + "timestamp", + "message", + "is_copied_context", + "extra", + "llm_call_count", + "observation", + "source", + } + agent = {"model_name", "reasoning_effort", "reasoning_content", "tool_calls", "metrics"} + if step["step_id"] != expected_id or step["source"] not in {"system", "user", "agent"}: + raise ValueError("invalid ATIF step identity") + if set(step) - (common | (agent if step["source"] == "agent" else set())): + raise ValueError("unknown step field") + _validate_atif_content(step["message"]) + _validate_atif_calls(step) + + +def _validate_atif_calls(step: dict[str, Any]) -> None: + calls = step.get("tool_calls") or [] + call_ids: set[str] = set() + for value in calls: + call = _as_dict(value) + _require_keys(call, {"tool_call_id", "function_name"}) + if set(call) - {"tool_call_id", "function_name", "arguments", "extra"} or call["tool_call_id"] in call_ids: + raise ValueError("invalid tool call") + call_ids.add(call["tool_call_id"]) + observation = _as_dict(step["observation"]) if step.get("observation") is not None else {"results": []} + if set(observation) - {"results"}: + raise ValueError("invalid observation") + for value in observation.get("results", []): + result = _as_dict(value) + if set(result) - {"source_call_id", "content", "subagent_trajectory_ref", "extra"}: + raise ValueError("invalid observation") + if result.get("source_call_id") is not None and result["source_call_id"] not in call_ids: + raise ValueError("unresolved observation") + if result.get("content") is not None: + _validate_atif_content(result["content"]) + + +def _validate_atif_content(value: object) -> None: + if isinstance(value, str): + return + if not isinstance(value, list): + raise ValueError("invalid ATIF content") + for part_value in value: + part = _as_dict(part_value) + if part.get("type") != "text" or set(part) != {"type", "text"} or not isinstance(part["text"], str): + raise ValueError("unreviewed ATIF content part") + + +def _validate_chat_completion(document: dict[str, Any]) -> None: + _require_keys(document, {"request", "response", "session_id", "trace_id"}) + allowed = { + "request", + "response", + "session_id", + "trace_id", + "provider", + "cost_usd", + "cost_input_usd", + "cost_output_usd", + "cost_details", + "evaluation_context", + } + if set(document) - allowed: + raise ValueError("unknown top-level field") + request = _as_dict(document["request"]) + response = _as_dict(document["response"]) + _validate_chat_request(request) + _validate_chat_response(response) + if ("choices" in response) == ("error" in response): + raise ValueError("response requires exactly one variant") + + +def _validate_chat_request(request: dict[str, Any]) -> None: + _require_keys(request, {"model", "messages"}) + if set(request) - {"model", "messages", "temperature", "provider_extension"}: + raise ValueError("unreviewed request extension") + extension = _as_dict(request["provider_extension"]) if "provider_extension" in request else {} + if set(extension) - {"region", "request_class"}: + raise ValueError("unreviewed provider extension") + for value in request["messages"]: + _validate_chat_message(_as_dict(value)) + + +def _validate_chat_message(message: dict[str, Any]) -> None: + if message.get("role") not in _CHAT_ROLES: + raise ValueError("invalid chat role") + if set(message) - {"role", "content", "tool_calls", "tool_call_id", "name"}: + raise ValueError("unreviewed message extension") + for value in message.get("tool_calls") or []: + call = _as_dict(value) + if set(call) != {"id", "type", "function"}: + raise ValueError("invalid tool call") + function = _as_dict(call["function"]) + if set(function) != {"name", "arguments"} or not isinstance(function["arguments"], str): + raise ValueError("invalid tool function") + + +def _validate_chat_response(response: dict[str, Any]) -> None: + _require_keys(response, {"created"}) + allowed = {"id", "object", "created", "model", "choices", "error", "usage", "provider_response_id"} + if set(response) - allowed: + raise ValueError("unreviewed response extension") + if not _valid_intake_timestamp(response["created"]): + raise ValueError("stable response creation time required") + for value in response.get("choices") or []: + choice = _as_dict(value) + if set(choice) - {"index", "message", "finish_reason"}: + raise ValueError("invalid choice") + message = _as_dict(choice["message"]) + if set(message) - {"role", "content"} or message.get("role") not in _CHAT_ROLES: + raise ValueError("invalid response message") + if "usage" in response and set(_as_dict(response["usage"])) - { + "prompt_tokens", + "completion_tokens", + "total_tokens", + }: + raise ValueError("unreviewed usage field") + + +def _valid_intake_timestamp(value: object) -> bool: + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + return False + try: + parsed = datetime.fromtimestamp(value, tz=timezone.utc) + except (OverflowError, OSError, ValueError): + return False + return parsed <= datetime.now(tz=timezone.utc) + + +def _require_keys(value: Mapping[str, object], required: set[str]) -> None: + if not required.issubset(value): + raise ValueError("required field missing") + + +def _as_dict(value: object) -> dict[str, Any]: + if not isinstance(value, dict): + raise ValueError("object required") + return cast(dict[str, Any], value) + + +def build_local_otlp_request(spec_path: Path) -> bytes: + spec = _json_object(spec_path.read_bytes()) + request = ExportTraceServiceRequest() + resource_spans = request.resource_spans.add() + _add_attributes(resource_spans.resource.attributes, _as_dict(spec["resource_attributes"])) + scope_spans = resource_spans.scope_spans.add() + scope_spans.scope.name = str(spec["scope"]["name"]) + scope_spans.scope.version = str(spec["scope"]["version"]) + for span_spec in cast(list[dict[str, Any]], spec["spans"]): + _add_span(scope_spans.spans.add(), span_spec) + return request.SerializeToString(deterministic=True) + + +def _add_span(span: Any, spec: dict[str, Any]) -> None: + span.trace_id = bytes.fromhex(spec["trace_id"]) + span.span_id = bytes.fromhex(spec["span_id"]) + if spec.get("parent_span_id"): + span.parent_span_id = bytes.fromhex(spec["parent_span_id"]) + span.name = spec["name"] + span.start_time_unix_nano = spec["start_time_unix_nano"] + span.end_time_unix_nano = spec["end_time_unix_nano"] + _add_attributes(span.attributes, spec["attributes"]) + + +def _add_attributes(target: Any, values: Mapping[str, object]) -> None: + for key, value in values.items(): + attribute = target.add() + attribute.key = key + if isinstance(value, bool): + attribute.value.bool_value = value + elif isinstance(value, int): + attribute.value.int_value = value + elif isinstance(value, float): + attribute.value.double_value = value + else: + attribute.value.string_value = str(value) + + +def protect_otlp_request(source: bytes, *, flow: _PlanAFlow, emit: Emitter) -> bytes: + request = _parse_otlp(source) + targets = _otlp_targets(request) + records = tuple(_ProtectionRecord(_RecordRef(ref), (_TextSegment(value),)) for ref, value in targets.items()) + run = flow.protect(records) + if len(run.outcomes) != len(records) or not all(isinstance(item, _ProtectionSucceeded) for item in run.outcomes): + raise IntakeValidationError("protection failed") + replacements = { + cast(_ProtectionSucceeded, item).ref.value: cast(_ProtectionSucceeded, item).output for item in run.outcomes + } + protected_request = ExportTraceServiceRequest() + protected_request.CopyFrom(request) + _apply_otlp_replacements(protected_request, replacements) + protected = protected_request.SerializeToString(deterministic=True) + emit(protected) + return protected + + +def _parse_otlp(source: bytes) -> ExportTraceServiceRequest: + try: + request = ExportTraceServiceRequest.FromString(source) + except DecodeError: + raise IntakeValidationError("OTLP batch rejected") from None + spans = list(_spans(request)) + if not spans or any(not _valid_span_identity(span) for span in spans) or not _valid_otlp_envelope(request): + raise IntakeValidationError("OTLP batch rejected") + return request + + +def _valid_otlp_envelope(request: ExportTraceServiceRequest) -> bool: + for resource_spans in request.resource_spans: + resource_attributes = resource_spans.resource.attributes + if any( + item.key != "service.name" or item.value.WhichOneof("value") != "string_value" + for item in resource_attributes + ): + return False + for scope_spans in resource_spans.scope_spans: + if scope_spans.scope.attributes: + return False + if any(span.events or span.links for span in scope_spans.spans): + return False + return True + + +def _valid_span_identity(span: Any) -> bool: + return ( + len(span.trace_id) == 16 + and any(span.trace_id) + and len(span.span_id) == 8 + and any(span.span_id) + and (not span.parent_span_id or (len(span.parent_span_id) == 8 and any(span.parent_span_id))) + ) + + +def _otlp_targets(request: ExportTraceServiceRequest) -> dict[str, str]: + targets: dict[str, str] = {} + for span in _spans(request): + span_id = span.span_id.hex() + attribute_keys: set[str] = set() + for attribute in span.attributes: + if attribute.key not in _OTLP_TARGET_ATTRIBUTES | _OTLP_STRUCTURAL_ATTRIBUTES: + raise IntakeValidationError("OTLP batch rejected") + if attribute.key in attribute_keys: + raise IntakeValidationError("OTLP batch rejected") + attribute_keys.add(attribute.key) + if attribute.key == "gen_ai.agent.name" and ( + attribute.value.WhichOneof("value") != "string_value" or not attribute.value.string_value + ): + raise IntakeValidationError("OTLP batch rejected") + if attribute.key in _OTLP_TARGET_ATTRIBUTES: + if attribute.value.WhichOneof("value") != "string_value": + raise IntakeValidationError("OTLP batch rejected") + targets[f"{span_id}:{attribute.key}"] = attribute.value.string_value + if span.status.message: + targets[f"{span_id}:status.message"] = span.status.message + return targets + + +def _apply_otlp_replacements(request: ExportTraceServiceRequest, replacements: Mapping[str, str]) -> None: + found: set[str] = set() + for span in _spans(request): + span_id = span.span_id.hex() + for attribute in span.attributes: + ref = f"{span_id}:{attribute.key}" + if ref in replacements: + attribute.value.string_value = replacements[ref] + found.add(ref) + status_ref = f"{span_id}:status.message" + if status_ref in replacements: + span.status.message = replacements[status_ref] + found.add(status_ref) + if found != set(replacements): + raise IntakeValidationError("OTLP batch rejected") + + +def _spans(request: ExportTraceServiceRequest): + for resource_spans in request.resource_spans: + for scope_spans in resource_spans.scope_spans: + yield from scope_spans.spans + + +def otlp_topology(source: bytes) -> tuple[tuple[str, str, str], ...]: + request = _parse_otlp(source) + return tuple((span.trace_id.hex(), span.span_id.hex(), span.parent_span_id.hex()) for span in _spans(request)) + + +def otlp_string_attributes(source: bytes) -> Mapping[str, Mapping[str, str]]: + request = _parse_otlp(source) + return { + span.span_id.hex(): { + attribute.key: attribute.value.string_value + for attribute in span.attributes + if attribute.value.WhichOneof("value") == "string_value" + } + for span in _spans(request) + } diff --git a/tests/streaming/openshell_ocsf_process_adapter.py b/tests/streaming/openshell_ocsf_process_adapter.py new file mode 100644 index 00000000..f14752e8 --- /dev/null +++ b/tests/streaming/openshell_ocsf_process_adapter.py @@ -0,0 +1,407 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Test-only buffered adapter for OpenShell OCSF Process Activity JSONL records.""" + +from __future__ import annotations + +import json +import math +from collections.abc import Callable, Mapping +from pathlib import Path +from typing import cast + +from anonymizer.interface.anonymizer import Anonymizer +from tests.streaming.structured_trace_prototype import ( + CodecBounds, + FailureCode, + FieldRole, + ProjectedItem, + SourceFormat, + StructuredItemError, + TraceMapping, + project_complete_item, + protect_and_emit, +) + +MAPPING_VERSION = "openshell-ocsf-process-activity-1007/v1" +FIDELITY_CLASS = "semantic-jsonl-item-v1" + +_ACTIVITY_LABELS = {1: "Launch", 2: "Terminate"} +_SEVERITY_LABELS = { + 0: "Unknown", + 1: "Informational", + 2: "Low", + 3: "Medium", + 4: "High", + 5: "Critical", + 6: "Fatal", + 99: "Other", +} +_STATUS_LABELS = {0: "Unknown", 1: "Success", 2: "Failure", 99: "Other"} +_LAUNCH_LABELS = {0: "Unknown", 1: "Spawn", 2: "Fork", 3: "Exec", 99: "Other"} +_ACTION_LABELS = {0: "Unknown", 1: "Allowed", 2: "Denied", 3: "Observed", 4: "Modified", 99: "Other"} +_DISPOSITION_LABELS = { + 0: "Unknown", + 1: "Allowed", + 2: "Blocked", + 3: "Quarantined", + 4: "Isolated", + 5: "Deleted", + 6: "Dropped", + 7: "Custom Action", + 8: "Approved", + 9: "Restored", + 10: "Exonerated", + 11: "Corrected", + 12: "Partially Corrected", + 13: "Uncorrected", + 14: "Delayed", + 15: "Detected", + 16: "No Action", + 17: "Logged", + 18: "Tagged", + 19: "Alert", + 20: "Count", + 21: "Reset", + 22: "Captcha", + 23: "Challenge", + 24: "Access Revoked", + 25: "Rejected", + 26: "Unauthorized", + 27: "Error", + 99: "Other", +} +_REQUIRED_EVENT_FIELDS = frozenset( + { + "class_uid", + "class_name", + "category_uid", + "category_name", + "activity_id", + "activity_name", + "type_uid", + "type_name", + "time", + "severity_id", + "severity", + "metadata", + "device", + "container", + "process", + } +) +_OPTIONAL_EVENT_FIELDS = frozenset( + { + "status_id", + "status", + "message", + "actor", + "launch_type_id", + "launch_type", + "exit_code", + "action_id", + "action", + "disposition_id", + "disposition", + } +) + + +def project_process_activity_item(source: bytes, *, bounds: CodecBounds) -> ProjectedItem: + """Validate and project one complete OpenShell Process Activity JSONL record.""" + document = _decode_and_validate(source, bounds=bounds) + mapping = _mapping_for(document) + projected = project_complete_item( + source, + source_format=SourceFormat.JSONL, + mapping=mapping, + bounds=bounds, + ) + if projected.manifest.fidelity != FIDELITY_CLASS: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + return projected + + +def protect_process_activity_item( + source: bytes, + *, + bounds: CodecBounds, + anonymizer: Anonymizer, + source_ref: Path, + emit: Callable[[bytes], None], +) -> bytes: + """Protect and schema-check one complete item before external emission.""" + document = _decode_and_validate(source, bounds=bounds) + staged: list[bytes] = [] + protected = protect_and_emit( + source, + source_format=SourceFormat.JSONL, + mapping=_mapping_for(document), + bounds=bounds, + anonymizer=anonymizer, + source_ref=source_ref, + emit=staged.append, + ) + if staged != [protected]: + raise StructuredItemError(FailureCode.SEGMENT_PROCESSING_FAILED) + _decode_and_validate(protected, bounds=bounds) + emit(protected) + return protected + + +def replay_process_activity_corpus( + corpus: bytes, + *, + max_corpus_bytes: int, + max_records: int, + bounds: CodecBounds, + anonymizer: Anonymizer, + source_ref: Path, + emit: Callable[[bytes], None], +) -> tuple[bytes, ...]: + """Replay bounded JSONL records without making OpenShell a dispatcher.""" + if max_corpus_bytes <= 0 or max_records <= 0: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + if not corpus or len(corpus) > max_corpus_bytes: + raise StructuredItemError(FailureCode.ITEM_TOO_LARGE) + lines = corpus.splitlines(keepends=True) + if len(lines) > max_records: + raise StructuredItemError(FailureCode.ITEM_TOO_LARGE) + if any(not line.endswith(b"\n") or not line[:-1].strip() for line in lines): + raise StructuredItemError(FailureCode.INVALID_SOURCE) + + protected_items: list[bytes] = [] + for line in lines: + protected_items.append( + protect_process_activity_item( + line, + bounds=bounds, + anonymizer=anonymizer, + source_ref=source_ref, + emit=emit, + ) + ) + return tuple(protected_items) + + +def _decode_and_validate(source: bytes, *, bounds: CodecBounds) -> dict[str, object]: + if len(source) > bounds.max_bytes: + raise StructuredItemError(FailureCode.ITEM_TOO_LARGE) + try: + lines = source.splitlines() + if len(lines) != 1 or not source.endswith(b"\n") or not lines[0].strip(): + raise StructuredItemError(FailureCode.INVALID_SOURCE) + document = json.loads( + lines[0].decode("utf-8"), + object_pairs_hook=_reject_duplicate_keys, + parse_constant=_reject_nonfinite_number, + parse_float=_parse_finite_float, + ) + if not isinstance(document, dict): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + typed_document = cast(dict[str, object], document) + _validate_process_activity(typed_document) + return typed_document + except StructuredItemError: + raise + except RecursionError: + raise StructuredItemError(FailureCode.STRUCTURE_TOO_DEEP) from None + except (UnicodeDecodeError, json.JSONDecodeError, TypeError, ValueError): + raise StructuredItemError(FailureCode.INVALID_SOURCE) from None + except Exception: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) from None + + +def _validate_process_activity(event: dict[str, object]) -> None: + _require_keys(event, required=_REQUIRED_EVENT_FIELDS, optional=_OPTIONAL_EVENT_FIELDS) + _validate_base_event(event) + _validate_optional_event_fields(event) + source_identity = _validate_metadata(_require_object(event["metadata"])) + _validate_device(_require_object(event["device"])) + _validate_container(_require_object(event["container"]), source_identity=source_identity) + _validate_process(_require_object(event["process"])) + if "actor" in event: + actor = _require_object(event["actor"]) + _require_keys(actor, required={"process"}) + _validate_process(_require_object(actor["process"])) + + +def _validate_base_event(event: Mapping[str, object]) -> None: + if _require_int(event["class_uid"]) != 1007 or _require_string(event["class_name"]) != "Process Activity": + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + if _require_int(event["category_uid"]) != 1 or _require_string(event["category_name"]) != "System Activity": + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + activity_id = _require_int(event["activity_id"]) + activity_name = _require_string(event["activity_name"]) + if _ACTIVITY_LABELS.get(activity_id) != activity_name: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + if _require_int(event["type_uid"]) != 100_700 + activity_id: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + if _require_string(event["type_name"]) != f"Process Activity: {activity_name}": + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + _require_int(event["time"]) + _validate_pair(event, "severity_id", "severity", _SEVERITY_LABELS, required=True) + + +def _validate_optional_event_fields(event: Mapping[str, object]) -> None: + _validate_pair(event, "status_id", "status", _STATUS_LABELS) + _validate_pair(event, "launch_type_id", "launch_type", _LAUNCH_LABELS) + _validate_pair(event, "action_id", "action", _ACTION_LABELS) + _validate_pair(event, "disposition_id", "disposition", _DISPOSITION_LABELS) + if "message" in event: + _require_string(event["message"]) + if "exit_code" in event: + _require_int(event["exit_code"]) + + +def _validate_metadata(metadata: dict[str, object]) -> str: + _require_keys(metadata, required={"version", "product", "profiles", "uid"}) + source_identity = _require_string(metadata["uid"]) + if _require_string(metadata["version"]) != "1.7.0" or not source_identity: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + if metadata["profiles"] != ["security_control", "container", "host"]: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + product = _require_object(metadata["product"]) + _require_keys(product, required={"name", "vendor_name", "version"}) + if ( + _require_string(product["name"]) != "OpenShell Sandbox Supervisor" + or _require_string(product["vendor_name"]) != "OpenShell" + or not _require_string(product["version"]) + ): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + return source_identity + + +def _validate_device(device: dict[str, object]) -> None: + _require_keys(device, required={"hostname", "os"}) + _require_string(device["hostname"]) + os_info = _require_object(device["os"]) + _require_keys(os_info, required={"name"}) + if _require_string(os_info["name"]) != "Linux": + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + + +def _validate_container(container: dict[str, object], *, source_identity: str) -> None: + _require_keys(container, required={"name", "uid", "image"}) + _require_string(container["name"]) + if _require_string(container["uid"]) != source_identity: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + image = _require_object(container["image"]) + _require_keys(image, required={"name"}) + _require_string(image["name"]) + + +def _validate_process(process: dict[str, object]) -> None: + _require_keys(process, required={"name", "pid"}, optional={"cmd_line", "parent_process"}) + _require_string(process["name"]) + _require_int(process["pid"]) + if "cmd_line" in process: + _require_string(process["cmd_line"]) + if "parent_process" in process: + _validate_process(_require_object(process["parent_process"])) + + +def _mapping_for(document: Mapping[str, object]) -> TraceMapping: + fields: dict[str, FieldRole] = {} + for pointer, _ in _iter_scalars(document): + fields[pointer] = FieldRole.TARGET if _is_target(pointer) else FieldRole.STRUCTURAL + return TraceMapping( + version=MAPPING_VERSION, + fields=fields, + source_identity_pointer="/metadata/uid", + ordered_identity_pointers=("/type_name",), + ) + + +def _is_target(pointer: str) -> bool: + tokens = pointer.removeprefix("/").split("/") + if pointer == "/message" or pointer == "/device/hostname" or pointer == "/container/name": + return True + if pointer == "/container/image/name": + return True + return tokens[-1] in {"name", "cmd_line"} and tokens[0] in {"process", "actor"} + + +def _iter_scalars(value: object, *, pointer: str = "") -> list[tuple[str, object]]: + if isinstance(value, dict): + result: list[tuple[str, object]] = [] + for key, item in cast(dict[str, object], value).items(): + escaped = key.replace("~", "~0").replace("/", "~1") + result.extend(_iter_scalars(item, pointer=f"{pointer}/{escaped}")) + return result + if isinstance(value, list): + result = [] + for index, item in enumerate(value): + result.extend(_iter_scalars(item, pointer=f"{pointer}/{index}")) + return result + return [(pointer, value)] + + +def _validate_pair( + value: Mapping[str, object], + id_key: str, + label_key: str, + labels: Mapping[int, str], + *, + required: bool = False, +) -> None: + present = id_key in value or label_key in value + if required and not present: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + if not present: + return + if id_key not in value or label_key not in value: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + if labels.get(_require_int(value[id_key])) != _require_string(value[label_key]): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + + +def _require_keys( + value: Mapping[str, object], + *, + required: set[str] | frozenset[str], + optional: set[str] | frozenset[str] | None = None, +) -> None: + keys = set(value) + allowed = required | (optional or set()) + if not required.issubset(keys) or not keys.issubset(allowed): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + + +def _require_object(value: object) -> dict[str, object]: + if not isinstance(value, dict) or any(not isinstance(key, str) for key in value): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + return cast(dict[str, object], value) + + +def _require_string(value: object) -> str: + if not isinstance(value, str): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + return value + + +def _require_int(value: object) -> int: + if not isinstance(value, int) or isinstance(value, bool): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + return value + + +def _reject_duplicate_keys(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + if key in result: + raise ValueError("duplicate JSON object key") + result[key] = value + return result + + +def _reject_nonfinite_number(_: str) -> float: + raise ValueError("non-finite JSON number") + + +def _parse_finite_float(value: str) -> float: + parsed = float(value) + if not math.isfinite(parsed): + raise ValueError("non-finite JSON number") + return parsed diff --git a/tests/streaming/run_internal_characterization.py b/tests/streaming/run_internal_characterization.py new file mode 100644 index 00000000..c79d6b57 --- /dev/null +++ b/tests/streaming/run_internal_characterization.py @@ -0,0 +1,139 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Synthetic-only, aggregate characterization of private row verification.""" + +from __future__ import annotations + +import json +import sys +import time +import tracemalloc +from pathlib import Path +from types import FrameType + +import pandas as pd + +from anonymizer.config.replace_strategies import Redact +from anonymizer.engine.constants import COL_REPLACED_TEXT, COL_TEXT +from anonymizer.engine.private_row_verification import _InvocationRowVerifier +from anonymizer.engine.replace.strategies import apply_local_replace_strategy + + +def _synthetic_entities(text: str) -> list[dict[str, object]]: + value = "synthetic-secret" + positions: list[int] = [] + start = 0 + while (found := text.find(value, start)) >= 0: + positions.append(found) + start = found + len(value) + return [ + { + "value": value, + "label": "synthetic_identifier", + "start_position": position, + "end_position": position + len(value), + } + for position in positions + ] + + +def _run_arm(name: str, rows: list[str], *, suitable: bool = True) -> dict[str, object]: + if not suitable: + return { + "arm": name, + "status": "blocked", + "availability": "governed_unavailable", + "reason_code": "source_specific_manifest_not_owned", + "input_bytes": None, + "output_bytes": None, + "rows": None, + "targets": None, + "provider_calls": None, + "elapsed_ms": None, + "peak_memory_bytes": None, + "raw_copy_count": None, + "artifact_delta_bytes": None, + "structural_validity": None, + "privacy_check": None, + "reconstruction_failures": None, + } + frame = pd.DataFrame( + { + COL_TEXT: rows, + "final_entities": [{"entities": _synthetic_entities(text)} for text in rows], + } + ) + verifier = _InvocationRowVerifier(frame) + bound = verifier.bind(frame) + verifier.freeze_accepted_detections(bound) + # Count actual live references to the synthetic source objects through the + # private working frames. This is an aggregate copy-pressure proxy, not a + # claim about allocator-level copies. + source_ids = {id(text) for text in rows} + provider_calls = 0 + + def profile_provider_calls(frame: FrameType, event: str, _arg: object) -> None: + nonlocal provider_calls + module_name = getattr(frame, "f_globals", {}).get("__name__", "") + if event == "call" and module_name.startswith(("data_designer", "openai")): + provider_calls += 1 + + previous_profiler = sys.getprofile() + tracemalloc.start() + started = time.perf_counter() + sys.setprofile(profile_provider_calls) + try: + protected = apply_local_replace_strategy(bound, strategy=Redact()) + finally: + sys.setprofile(previous_profiler) + elapsed_ms = round((time.perf_counter() - started) * 1000, 3) + _, peak = tracemalloc.get_traced_memory() + tracemalloc.stop() + verified = verifier.finish(protected) + outputs = verified[COL_REPLACED_TEXT].astype(str).tolist() + targets = sum(text.count("synthetic-secret") for text in rows) + input_bytes = sum(len(text.encode()) for text in rows) + output_bytes = sum(len(text.encode()) for text in outputs) + raw_copy_count = sum( + id(value) in source_ids + for dataframe in (frame, bound, protected, verified) + for value in dataframe[COL_TEXT].tolist() + ) + structural_validity = len(verified) == len(rows) and len(outputs) == len(rows) + privacy_check = all("synthetic-secret" not in text for text in outputs) + return { + "arm": name, + "status": "completed", + "availability": "available", + "reason_code": None, + "input_bytes": input_bytes, + "output_bytes": output_bytes, + "rows": len(rows), + "targets": targets, + "provider_calls": provider_calls, + "elapsed_ms": elapsed_ms, + "peak_memory_bytes": peak, + "raw_copy_count": raw_copy_count, + "artifact_delta_bytes": output_bytes - input_bytes, + "structural_validity": structural_validity, + "privacy_check": privacy_check, + "reconstruction_failures": int(not structural_validity), + } + + +def main() -> None: + report = { + "fixture_policy": str(Path("tests/fixtures/streaming/POLICY.md")), + "synthetic_only": True, + "arms": [ + _run_arm("field_per_row", ["synthetic-secret alpha", "synthetic-secret beta"]), + _run_arm("whole_synthetic_blob", ["synthetic-secret alpha\nsynthetic-secret beta"]), + _run_arm("generic_manifest", [], suitable=False), + ], + } + print(json.dumps(report, sort_keys=True, separators=(",", ":"))) + + +if __name__ == "__main__": + main() diff --git a/tests/streaming/sandbox_session_export.py b/tests/streaming/sandbox_session_export.py new file mode 100644 index 00000000..1a4d706f --- /dev/null +++ b/tests/streaming/sandbox_session_export.py @@ -0,0 +1,242 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Test-only export of a completed Sandbox Codex session to ATIF v1.0.""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from pathlib import Path +from typing import Any, cast + +_ITEM_TYPES = {"agent_message", "command_execution", "file_change"} +_EVENT_TYPES = {"thread.started", "turn.started", "item.started", "item.completed", "turn.completed"} +_MAX_ARTIFACT_BYTES = 16 * 1024 * 1024 +_MAX_EVENTS = 10_000 + + +class SandboxSessionExportError(ValueError): + """The Sandbox artifact set is incomplete or outside the reviewed shape.""" + + +def export_codex_session_to_atif(run_dir: Path, *, session_id: str) -> bytes: + """Map one successful Sandbox Codex run to closed, test-only ATIF bytes.""" + if not session_id: + raise SandboxSessionExportError("ATIF session identity is required") + manifest = _read_object(run_dir / "manifest.json") + status = _read_object(run_dir / "status.json") + _require_exact_keys(status, required={"state", "exit_code"}, context="status", allow_extra=True) + exit_code = status["exit_code"] + if ( + status["state"] != "completed" + or not isinstance(exit_code, int) + or isinstance(exit_code, bool) + or exit_code != 0 + ): + raise SandboxSessionExportError("Sandbox run is not successfully completed") + + provenance = _object_at(manifest, "artifacts", "provenance") + dispatch = _object_at(provenance, "dispatch") + if dispatch.get("agent") != "codex": + raise SandboxSessionExportError("Sandbox run is not a Codex session") + + prompt_metadata = _object_at(provenance, "prompt") + prompt_name = Path(_string_at(prompt_metadata, "path")).name + if not prompt_name or prompt_metadata.get("exists") is not True or prompt_metadata.get("type") != "file": + raise SandboxSessionExportError("Sandbox prompt artifact is unavailable") + prompt = _read_text(run_dir / prompt_name) + + events = _read_jsonl(run_dir / "agent-output.jsonl") + for event in events: + event_type = event.get("type") + if event_type not in _EVENT_TYPES: + raise SandboxSessionExportError("Sandbox event type is outside the reviewed shape") + if event_type in {"item.started", "item.completed"}: + item = _object_at(event, "item") + _string_at(item, "id") + if item.get("type") not in _ITEM_TYPES: + raise SandboxSessionExportError("Sandbox item type is outside the reviewed shape") + thread_ids = [event.get("thread_id") for event in events if event.get("type") == "thread.started"] + started_turns = [event for event in events if event.get("type") == "turn.started"] + completed_turns = [event for event in events if event.get("type") == "turn.completed"] + if ( + len(thread_ids) != 1 + or not isinstance(thread_ids[0], str) + or not thread_ids[0] + or len(started_turns) != 1 + or len(completed_turns) != 1 + ): + raise SandboxSessionExportError("Sandbox event lifecycle is outside the reviewed shape") + usage = _object_at(completed_turns[0], "usage") + prompt_tokens = _nonnegative_int_at(usage, "input_tokens") + completion_tokens = _nonnegative_int_at(usage, "output_tokens") + + item_steps: list[dict[str, Any]] = [] + seen_ids: set[str] = set() + for event in events: + if event.get("type") != "item.completed": + continue + item = _object_at(event, "item") + item_id = _string_at(item, "id") + if item_id in seen_ids: + raise SandboxSessionExportError("Sandbox item identity is duplicated") + seen_ids.add(item_id) + item_steps.append(_item_step(item, item_id=item_id)) + if not item_steps: + raise SandboxSessionExportError("Sandbox session has no completed items") + + model = _string_at(manifest, "model") + document = { + "schema_version": "ATIF-v1.0", + "session_id": session_id, + "trajectory_id": thread_ids[0], + "agent": { + "name": "codex", + "version": _string_at(manifest, "runtime_version"), + "model_name": model, + }, + "steps": [ + { + "step_id": 1, + "timestamp": _string_at(manifest, "started_at"), + "source": "user", + "message": prompt, + }, + *( + dict(step, step_id=index, timestamp=_string_at(manifest, "finished_at"), model_name=model) + for index, step in enumerate(item_steps, start=2) + ), + ], + "final_metrics": { + "total_prompt_tokens": prompt_tokens, + "total_completion_tokens": completion_tokens, + "total_steps": len(item_steps) + 1, + }, + } + return json.dumps(document, separators=(",", ":")).encode() + + +def _item_step(item: Mapping[str, Any], *, item_id: str) -> dict[str, Any]: + item_type = item.get("type") + if item_type == "agent_message": + return {"source": "agent", "message": _string_at(item, "text")} + if item_type == "command_execution": + if item.get("status") != "completed" or not isinstance(item.get("exit_code"), int): + raise SandboxSessionExportError("Sandbox command is not successfully completed") + return _tool_step( + item_id=item_id, + function_name="shell", + arguments={"command": _string_at(item, "command")}, + content=_text_at(item, "aggregated_output"), + ) + if item_type == "file_change": + if item.get("status") != "completed" or not isinstance(item.get("changes"), list): + raise SandboxSessionExportError("Sandbox file change is not successfully completed") + changes = [] + for value in item["changes"]: + change = _as_object(value, context="file change") + _require_exact_keys(change, required={"path", "kind"}, context="file change") + changes.append({"path": _string_at(change, "path"), "kind": _string_at(change, "kind")}) + return _tool_step( + item_id=item_id, + function_name="file_change", + arguments={"changes": changes}, + content="completed", + ) + raise SandboxSessionExportError("Sandbox item type is outside the reviewed shape") + + +def _tool_step( + *, + item_id: str, + function_name: str, + arguments: Mapping[str, Any], + content: str, +) -> dict[str, Any]: + return { + "source": "agent", + "message": "", + "tool_calls": [ + { + "tool_call_id": item_id, + "function_name": function_name, + "arguments": dict(arguments), + } + ], + "observation": {"results": [{"source_call_id": item_id, "content": content}]}, + } + + +def _read_object(path: Path) -> dict[str, Any]: + try: + return _as_object(json.loads(_read_text(path)), context=path.name) + except json.JSONDecodeError: + raise SandboxSessionExportError(f"{path.name} is not valid JSON") from None + + +def _read_jsonl(path: Path) -> list[dict[str, Any]]: + try: + lines = _read_text(path).splitlines() + if len(lines) > _MAX_EVENTS: + raise SandboxSessionExportError("Sandbox event count exceeds the test-only bound") + return [_as_object(json.loads(line), context=path.name) for line in lines] + except json.JSONDecodeError: + raise SandboxSessionExportError(f"{path.name} is not valid JSONL") from None + + +def _read_text(path: Path) -> str: + try: + if path.is_symlink() or path.stat().st_size > _MAX_ARTIFACT_BYTES: + raise SandboxSessionExportError(f"Sandbox artifact is outside the test-only bound: {path.name}") + return path.read_text() + except (OSError, UnicodeError): + raise SandboxSessionExportError(f"Sandbox artifact is unavailable: {path.name}") from None + + +def _object_at(value: Mapping[str, Any], *keys: str) -> dict[str, Any]: + current: object = value + for key in keys: + current = _as_object(current, context=key) + if key not in current: + raise SandboxSessionExportError(f"Sandbox artifact is missing {key}") + current = current[key] + return _as_object(current, context=keys[-1]) + + +def _as_object(value: object, *, context: str) -> dict[str, Any]: + if not isinstance(value, dict) or not all(isinstance(key, str) for key in value): + raise SandboxSessionExportError(f"Sandbox {context} must be an object") + return cast(dict[str, Any], value) + + +def _string_at(value: Mapping[str, Any], key: str) -> str: + result = _text_at(value, key) + if not result: + raise SandboxSessionExportError(f"Sandbox artifact requires nonempty string {key}") + return result + + +def _text_at(value: Mapping[str, Any], key: str) -> str: + result = value.get(key) + if not isinstance(result, str): + raise SandboxSessionExportError(f"Sandbox artifact requires string {key}") + return result + + +def _nonnegative_int_at(value: Mapping[str, Any], key: str) -> int: + result = value.get(key) + if not isinstance(result, int) or isinstance(result, bool) or result < 0: + raise SandboxSessionExportError(f"Sandbox artifact requires nonnegative integer {key}") + return result + + +def _require_exact_keys( + value: Mapping[str, Any], + *, + required: set[str], + context: str, + allow_extra: bool = False, +) -> None: + if not required <= value.keys() or (not allow_extra and set(value) != required): + raise SandboxSessionExportError(f"Sandbox {context} fields are outside the reviewed shape") diff --git a/tests/streaming/structured_trace_prototype.py b/tests/streaming/structured_trace_prototype.py new file mode 100644 index 00000000..29942e3b --- /dev/null +++ b/tests/streaming/structured_trace_prototype.py @@ -0,0 +1,761 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Test-only adopter-owned structured trace prototype.""" + +from __future__ import annotations + +import hashlib +import json +import math +from collections.abc import Callable, Mapping +from copy import deepcopy +from dataclasses import dataclass +from enum import Enum +from pathlib import Path +from types import MappingProxyType +from typing import cast +from unittest.mock import Mock + +import pandas as pd +from data_designer.interface.data_designer import DataDesigner + +from anonymizer.config.anonymizer_config import AnonymizerConfig, AnonymizerInput +from anonymizer.config.replace_strategies import Redact +from anonymizer.engine.constants import COL_DETECTED_ENTITIES, COL_FINAL_ENTITIES, COL_TEXT +from anonymizer.engine.detection.detection_workflow import EntityDetectionResult, EntityDetectionWorkflow +from anonymizer.engine.execution.context_contract import ( + _BackendArtifactClass, + _ContextBackendCapability, + _ContextLimits, + _ContextOrdering, + _ContextProfile, + _ContextSchemaVersion, + _RetentionPosture, +) +from anonymizer.engine.execution.mention_admission import _ValidationDecision, _ValidationDecisionKind +from anonymizer.engine.execution.mention_resolution import _SubjectEvidence +from anonymizer.engine.execution.phase6_runtime import ( + _CandidateProposal, + _Phase6AugmentationWork, + _Phase6CandidateWork, + _Phase6ResolverWork, + _Phase6ValidationWork, +) +from anonymizer.engine.ndd.adapter import FailedRecord +from anonymizer.engine.replace.replace_runner import ReplacementWorkflow +from anonymizer.engine.resolved_input import ResolvedInput +from anonymizer.interface._protection import _ProtectionFlow +from anonymizer.interface.anonymizer import Anonymizer +from anonymizer.interface.results import AnonymizerResult + +SEGMENT_KEY_COLUMN = "caller_segment_key" +PUBLIC_TEXT_COLUMN = "segment_text" +PROTECTED_TEXT_COLUMN = f"{PUBLIC_TEXT_COLUMN}_replaced" + + +class SourceFormat(str, Enum): + JSON = "json" + JSONL = "jsonl" + + +class FieldRole(str, Enum): + TARGET = "target" + PRESERVE = "preserve" + STRUCTURAL = "structural" + + +class FailureCode(str, Enum): + INVALID_SOURCE = "invalid_source" + ITEM_TOO_LARGE = "item_too_large" + STRUCTURE_TOO_DEEP = "structure_too_deep" + TOO_MANY_TARGETS = "too_many_targets" + UNKNOWN_FIELD = "unknown_field" + MAPPING_MISMATCH = "mapping_mismatch" + SEGMENT_PROCESSING_FAILED = "segment_processing_failed" + MISSING_SEGMENT = "missing_segment" + DUPLICATE_SEGMENT = "duplicate_segment" + UNKNOWN_SEGMENT = "unknown_segment" + UNPROTECTED_TARGET = "unprotected_target" + + +class StructuredItemError(RuntimeError): + """Sanitized test-visible complete-item failure.""" + + def __init__(self, code: FailureCode) -> None: + self.code = code + super().__init__(f"structured item rejected ({code.value})") + + +@dataclass(frozen=True) +class CodecBounds: + max_bytes: int + max_depth: int + max_targets: int + max_scalars: int = 64 + max_scalar_bytes: int = 1_024 + max_events: int = 256 + + +@dataclass(frozen=True) +class TraceMapping: + version: str + fields: Mapping[str, FieldRole] + source_identity_pointer: str + ordered_identity_pointers: tuple[str, ...] + + +@dataclass(frozen=True) +class SegmentManifest: + segment_key: str + pointer: str + occurrence_index: int + source_order: int + input_sha256: str + + +@dataclass(frozen=True) +class ReconstructionManifest: + mapping_version: str + source_format: SourceFormat + fidelity: str + source_identity: str + source_order: tuple[str, ...] + source_identity_pointer: str + ordered_identity_pointers: tuple[str, ...] + segments: tuple[SegmentManifest, ...] + preserved_sha256: Mapping[str, str] + structural_sha256: Mapping[str, str] + _template_json: bytes + + @property + def template(self) -> object: + """Return a detached view of the private reconstruction template.""" + return json.loads(self._template_json) + + +@dataclass(frozen=True) +class ProjectedItem: + _dataframe: pd.DataFrame + manifest: ReconstructionManifest + + @property + def dataframe(self) -> pd.DataFrame: + """Return a detached view of the private segment projection.""" + return self._dataframe.copy(deep=True) + + +ResultTransform = Callable[[pd.DataFrame], pd.DataFrame] +Emitter = Callable[[bytes], None] + + +def project_complete_item( + source: bytes, + *, + source_format: SourceFormat, + mapping: TraceMapping, + bounds: CodecBounds, +) -> ProjectedItem: + """Parse and project one complete source item under a closed field policy.""" + document, scalars = _validate_source_and_mapping( + source, + source_format=source_format, + mapping=mapping, + bounds=bounds, + ) + return _build_projection(document, scalars=scalars, source_format=source_format, mapping=mapping) + + +def _validate_source_and_mapping( + source: bytes, + *, + source_format: SourceFormat, + mapping: TraceMapping, + bounds: CodecBounds, +) -> tuple[object, dict[str, object]]: + _validate_configuration(source_format=source_format, mapping=mapping, bounds=bounds) + if len(source) > bounds.max_bytes: + raise StructuredItemError(FailureCode.ITEM_TOO_LARGE) + document = _parse_source(source, source_format=source_format, bounds=bounds) + scalars = _collect_bounded_scalars(document, bounds=bounds) + + for pointer in mapping.fields: + _resolve_pointer(document, pointer) + + declared_pointers = set(mapping.fields) + actual_pointers = set(scalars) + if actual_pointers - declared_pointers: + raise StructuredItemError(FailureCode.UNKNOWN_FIELD) + if declared_pointers - actual_pointers: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + + target_pointers = [pointer for pointer in scalars if mapping.fields[pointer] is FieldRole.TARGET] + if len(target_pointers) > bounds.max_targets: + raise StructuredItemError(FailureCode.TOO_MANY_TARGETS) + return document, scalars + + +def _validate_configuration( + *, + source_format: SourceFormat, + mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + if not isinstance(source_format, SourceFormat): + raise StructuredItemError(FailureCode.INVALID_SOURCE) + if ( + not isinstance(bounds, CodecBounds) + or bounds.max_bytes <= 0 + or bounds.max_depth <= 0 + or bounds.max_targets < 0 + or bounds.max_scalars <= 0 + or bounds.max_scalar_bytes <= 0 + or bounds.max_events <= 0 + ): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + if not isinstance(mapping, TraceMapping) or not mapping.version or not isinstance(mapping.fields, Mapping): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + + identity_pointers = (mapping.source_identity_pointer, *mapping.ordered_identity_pointers) + if len(identity_pointers) != len(set(identity_pointers)): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + for pointer, role in mapping.fields.items(): + if not isinstance(pointer, str) or not isinstance(role, FieldRole): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + _pointer_tokens(pointer) + if any(mapping.fields.get(pointer) is not FieldRole.STRUCTURAL for pointer in identity_pointers): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + + +def _build_projection( + document: object, + *, + scalars: Mapping[str, object], + source_format: SourceFormat, + mapping: TraceMapping, +) -> ProjectedItem: + source_identity = _require_string(_resolve_pointer(document, mapping.source_identity_pointer)) + source_order = tuple( + _require_string(_resolve_pointer(document, pointer)) for pointer in mapping.ordered_identity_pointers + ) + + template = deepcopy(document) + rows: list[dict[str, str]] = [] + segments: list[SegmentManifest] = [] + preserved_sha256: dict[str, str] = {} + structural_sha256: dict[str, str] = {} + for pointer, value in scalars.items(): + role = mapping.fields[pointer] + if role is FieldRole.TARGET: + text = _require_string(value) + occurrence_index = 0 + segment_key = f"{mapping.version}:{pointer}#{occurrence_index}" + segments.append( + SegmentManifest( + segment_key=segment_key, + pointer=pointer, + occurrence_index=occurrence_index, + source_order=len(segments), + input_sha256=_sha256_value(text), + ) + ) + rows.append({COL_TEXT: text, SEGMENT_KEY_COLUMN: segment_key}) + _set_pointer(template, pointer, {"__protected_segment_key__": segment_key}) + elif role is FieldRole.PRESERVE: + preserved_sha256[pointer] = _sha256_value(value) + else: + structural_sha256[pointer] = _sha256_value(value) + + fidelity = "semantic-json-v1" if source_format is SourceFormat.JSON else "semantic-jsonl-item-v1" + manifest = ReconstructionManifest( + mapping_version=mapping.version, + source_format=source_format, + fidelity=fidelity, + source_identity=source_identity, + source_order=source_order, + source_identity_pointer=mapping.source_identity_pointer, + ordered_identity_pointers=mapping.ordered_identity_pointers, + segments=tuple(segments), + preserved_sha256=MappingProxyType(preserved_sha256), + structural_sha256=MappingProxyType(structural_sha256), + _template_json=json.dumps(template, ensure_ascii=False, separators=(",", ":")).encode(), + ) + dataframe = pd.DataFrame( + { + COL_TEXT: [row[COL_TEXT] for row in rows], + SEGMENT_KEY_COLUMN: [row[SEGMENT_KEY_COLUMN] for row in rows], + } + ) + return ProjectedItem(_dataframe=dataframe.copy(deep=True), manifest=manifest) + + +class _SyntheticDetectionWorkflow: + """Deterministic detector used only to exercise the facade and local Redact.""" + + def __init__( + self, + sensitive_entities: Mapping[str, str], + *, + failed_segment_key: str | None, + ) -> None: + self._sensitive_entities = sensitive_entities + self._failed_segment_key = failed_segment_key + + def run(self, dataframe: pd.DataFrame, **_: object) -> EntityDetectionResult: + output = dataframe.copy() + failed_records: list[FailedRecord] = [] + if self._failed_segment_key is not None: + failed = output[SEGMENT_KEY_COLUMN] == self._failed_segment_key + if failed.any(): + raw_text = str(output.loc[failed, COL_TEXT].iloc[0]) + failed_records.append( + FailedRecord( + record_id="engine-row-private-8675309", + step="synthetic-detection", + reason=f"synthetic detector failed while processing {raw_text}", + ) + ) + output = output.loc[~failed].copy() + + entity_rows = [self._find_entities(str(text)) for text in output[COL_TEXT]] + output[COL_DETECTED_ENTITIES] = entity_rows + output[COL_FINAL_ENTITIES] = entity_rows + return EntityDetectionResult(dataframe=output, failed_records=failed_records) + + def _find_entities(self, text: str) -> dict[str, list[dict[str, str | int]]]: + entities: list[dict[str, str | int]] = [] + for value, label in self._sensitive_entities.items(): + start = 0 + while True: + found = text.find(value, start) + if found < 0: + break + entities.append( + { + "value": value, + "label": label, + "start_position": found, + "end_position": found + len(value), + } + ) + start = found + len(value) + entities.sort(key=lambda entity: cast(int, entity["start_position"])) + return {"entities": entities} + + +def build_synthetic_anonymizer( + sensitive_entities: Mapping[str, str], + *, + failed_segment_key: str | None = None, +) -> Anonymizer: + detector = _SyntheticDetectionWorkflow(sensitive_entities, failed_segment_key=failed_segment_key) + data_designer = cast(DataDesigner, Mock(spec=DataDesigner)) + return Anonymizer( + data_designer=data_designer, + detection_workflow=cast(EntityDetectionWorkflow, detector), + replace_runner=ReplacementWorkflow(), + ) + + +class _SyntheticPhase6Backend: + """Exact-span Phase 6 backend for test-only private-flow adopters.""" + + def __init__(self, sensitive_entities: Mapping[str, str]) -> None: + self._sensitive_entities = sensitive_entities + + def context_capability(self) -> _ContextBackendCapability: + return _ContextBackendCapability( + _ContextProfile.TARGET_CONTEXT_V1, + _ContextSchemaVersion.V1, + _ContextLimits(128, 1_048_576, 16_384, 2_097_152), + True, + _ContextOrdering.DECLARED, + (_BackendArtifactClass.CONTEXT_REQUEST,), + _RetentionPosture.DISABLED, + ) + + def detect(self, work: _Phase6CandidateWork) -> tuple[_CandidateProposal, ...]: + proposals: list[_CandidateProposal] = [] + for value, label in self._sensitive_entities.items(): + start = 0 + while (found := work.target.text.find(value, start)) >= 0: + proposals.append(_CandidateProposal(found, found + len(value), value, label)) + start = found + len(value) + return tuple(sorted(proposals, key=lambda proposal: (proposal.start, proposal.end))) + + def augment(self, work: _Phase6AugmentationWork) -> tuple[_CandidateProposal, ...]: + return () + + def validate(self, work: _Phase6ValidationWork) -> tuple[_ValidationDecision, ...]: + return tuple( + _ValidationDecision(candidate.token, _ValidationDecisionKind.KEEP) for candidate in work.candidates + ) + + def resolve(self, work: _Phase6ResolverWork) -> tuple[_SubjectEvidence, ...]: + return () + + def close_phase6(self) -> bool: + return True + + +def build_synthetic_protection_flow( + sensitive_entities: Mapping[str, str], +) -> _ProtectionFlow: + """Build a private flow without making provider calls from format tests.""" + anonymizer = build_synthetic_anonymizer(sensitive_entities) + plan = anonymizer._compile_protection_plan(AnonymizerConfig(replace=Redact(), emit_telemetry=False)) + return _ProtectionFlow(anonymizer, plan, phase6_backend=_SyntheticPhase6Backend(sensitive_entities)) + + +def run_projected_segments( + anonymizer: Anonymizer, + projected: ProjectedItem, + *, + source_ref: Path, +) -> AnonymizerResult: + data = AnonymizerInput( + source=str(source_ref), + text_column=PUBLIC_TEXT_COLUMN, + data_summary="Synthetic structured trace target fields.", + ) + config = AnonymizerConfig(replace=Redact(), emit_telemetry=False) + context = ResolvedInput( + dataframe=projected.dataframe, + requested_text_column=PUBLIC_TEXT_COLUMN, + resolved_text_column=PUBLIC_TEXT_COLUMN, + ) + anonymizer.validate_config(config) + return anonymizer._run_internal( + config=config, + data=data, + context=context, + preview_num_records=None, + ) + + +def reconstruct_complete_item( + projected: ProjectedItem, + result_dataframe: pd.DataFrame, + *, + failed_records: list[FailedRecord] | None = None, + detected_values_by_key: Mapping[str, tuple[str, ...]] | None = None, +) -> bytes: + """Patch exactly one protected result per declared segment into the private template.""" + try: + if not isinstance(projected.manifest.source_format, SourceFormat): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + protected_by_key = _collect_protected_results( + projected, + result_dataframe, + failed_records=failed_records, + detected_values_by_key=detected_values_by_key, + ) + reconstructed = projected.manifest.template + for segment in projected.manifest.segments: + _set_pointer(reconstructed, segment.pointer, protected_by_key[segment.segment_key]) + _verify_unchanged_fields(reconstructed, projected.manifest.preserved_sha256) + _verify_unchanged_fields(reconstructed, projected.manifest.structural_sha256) + _verify_identity(reconstructed, projected.manifest) + + protected = json.dumps(reconstructed, ensure_ascii=False, separators=(",", ":")).encode() + if projected.manifest.source_format is SourceFormat.JSONL: + protected += b"\n" + return protected + except StructuredItemError: + raise + except Exception: + raise StructuredItemError(FailureCode.SEGMENT_PROCESSING_FAILED) from None + + +def _collect_protected_results( + projected: ProjectedItem, + result_dataframe: pd.DataFrame, + *, + failed_records: list[FailedRecord] | None, + detected_values_by_key: Mapping[str, tuple[str, ...]] | None, +) -> dict[str, str]: + if failed_records: + raise StructuredItemError(FailureCode.SEGMENT_PROCESSING_FAILED) + if not isinstance(result_dataframe, pd.DataFrame): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + required_columns = (SEGMENT_KEY_COLUMN, PROTECTED_TEXT_COLUMN, COL_FINAL_ENTITIES) + labels = result_dataframe.columns.tolist() + if any(labels.count(column) != 1 for column in required_columns): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + + keys = result_dataframe[SEGMENT_KEY_COLUMN] + if keys.duplicated().any(): + raise StructuredItemError(FailureCode.DUPLICATE_SEGMENT) + if not keys.map(lambda value: isinstance(value, str)).all(): + raise StructuredItemError(FailureCode.UNKNOWN_SEGMENT) + + expected = {segment.segment_key: segment for segment in projected.manifest.segments} + actual_keys = set(cast(list[str], keys.tolist())) + if actual_keys - set(expected): + raise StructuredItemError(FailureCode.UNKNOWN_SEGMENT) + if set(expected) - actual_keys: + raise StructuredItemError(FailureCode.MISSING_SEGMENT) + + protected_by_key: dict[str, str] = {} + for _, row in result_dataframe.iterrows(): + segment_key = cast(str, row[SEGMENT_KEY_COLUMN]) + protected_text = row[PROTECTED_TEXT_COLUMN] + if not isinstance(protected_text, str): + raise StructuredItemError(FailureCode.SEGMENT_PROCESSING_FAILED) + row_detected_values = _detected_entity_values(row[COL_FINAL_ENTITIES]) + detected_values = ( + row_detected_values if detected_values_by_key is None else detected_values_by_key.get(segment_key) + ) + if detected_values is None or (detected_values_by_key is not None and row_detected_values != detected_values): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + if detected_values and any(value in protected_text for value in detected_values): + raise StructuredItemError(FailureCode.UNPROTECTED_TARGET) + protected_by_key[segment_key] = protected_text + return protected_by_key + + +def protect_and_emit( + source: bytes, + *, + source_format: SourceFormat, + mapping: TraceMapping, + bounds: CodecBounds, + anonymizer: Anonymizer, + source_ref: Path, + emit: Emitter, + result_transform: ResultTransform | None = None, +) -> bytes: + projected = project_complete_item( + source, + source_format=source_format, + mapping=mapping, + bounds=bounds, + ) + try: + result = run_projected_segments(anonymizer, projected, source_ref=source_ref) + detected_values_by_key = _snapshot_detected_values(result.trace_dataframe) + result_dataframe = result.trace_dataframe + if result_transform is not None: + result_dataframe = result_transform(result_dataframe.copy()) + protected = reconstruct_complete_item( + projected, + result_dataframe, + failed_records=result.failed_records, + detected_values_by_key=detected_values_by_key, + ) + except StructuredItemError: + raise + except Exception: + raise StructuredItemError(FailureCode.SEGMENT_PROCESSING_FAILED) from None + emit(protected) + return protected + + +def _parse_source(source: bytes, *, source_format: SourceFormat, bounds: CodecBounds) -> object: + try: + if source_format is SourceFormat.JSON: + encoded_item = source + elif source_format is SourceFormat.JSONL: + lines = source.splitlines() + if len(lines) != 1 or not lines[0].strip(): + raise StructuredItemError(FailureCode.INVALID_SOURCE) + encoded_item = lines[0] + else: + raise StructuredItemError(FailureCode.INVALID_SOURCE) + return json.loads( + encoded_item.decode("utf-8"), + object_pairs_hook=_reject_duplicate_keys, + parse_constant=_reject_nonfinite_number, + parse_int=lambda value: _parse_bounded_int(value, bounds.max_scalar_bytes), + parse_float=lambda value: _parse_bounded_float(value, bounds.max_scalar_bytes), + ) + except StructuredItemError: + raise + except RecursionError: + raise StructuredItemError(FailureCode.STRUCTURE_TOO_DEEP) from None + except (UnicodeDecodeError, json.JSONDecodeError, ValueError): + raise StructuredItemError(FailureCode.INVALID_SOURCE) from None + + +def _collect_bounded_scalars(value: object, *, bounds: CodecBounds) -> dict[str, object]: + scalars: dict[str, object] = {} + pending: list[tuple[object, str, int]] = [(value, "", 1)] + events = 0 + while pending: + events += 1 + if events > bounds.max_events: + raise StructuredItemError(FailureCode.ITEM_TOO_LARGE) + item, pointer, depth = pending.pop() + if depth > bounds.max_depth: + raise StructuredItemError(FailureCode.STRUCTURE_TOO_DEEP) + if isinstance(item, dict): + for key, child in reversed(cast(dict[str, object], item).items()): + if len(key.encode("utf-8")) > bounds.max_scalar_bytes: + raise StructuredItemError(FailureCode.ITEM_TOO_LARGE) + pending.append((child, f"{pointer}/{_escape_pointer_token(key)}", depth + 1)) + elif isinstance(item, list): + for index in range(len(item) - 1, -1, -1): + pending.append((item[index], f"{pointer}/{index}", depth + 1)) + else: + if len(json.dumps(item, ensure_ascii=False).encode()) > bounds.max_scalar_bytes: + raise StructuredItemError(FailureCode.ITEM_TOO_LARGE) + scalars[pointer] = item + if len(scalars) > bounds.max_scalars: + raise StructuredItemError(FailureCode.ITEM_TOO_LARGE) + return scalars + + +def _escape_pointer_token(token: str) -> str: + return token.replace("~", "~0").replace("/", "~1") + + +def _pointer_tokens(pointer: str) -> tuple[str, ...]: + if not pointer.startswith("/"): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + decoded: list[str] = [] + for token in pointer[1:].split("/"): + index = 0 + while index < len(token): + if token[index] == "~": + if index + 1 >= len(token) or token[index + 1] not in {"0", "1"}: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + index += 2 + else: + index += 1 + decoded.append(token.replace("~1", "/").replace("~0", "~")) + return tuple(decoded) + + +def _resolve_pointer(document: object, pointer: str) -> object: + current = document + try: + for token in _pointer_tokens(pointer): + if isinstance(current, list): + current = cast(list[object], current)[_array_index(token, len(current))] + elif isinstance(current, dict): + current = cast(dict[str, object], current)[token] + else: + raise KeyError(token) + except (KeyError, IndexError, ValueError): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) from None + return current + + +def _set_pointer(document: object, pointer: str, value: object) -> None: + tokens = _pointer_tokens(pointer) + if not tokens: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + parent = document + try: + for token in tokens[:-1]: + if isinstance(parent, list): + parent = cast(list[object], parent)[_array_index(token, len(parent))] + elif isinstance(parent, dict): + parent = cast(dict[str, object], parent)[token] + else: + raise KeyError(token) + final = tokens[-1] + if isinstance(parent, list): + cast(list[object], parent)[_array_index(final, len(parent))] = value + elif isinstance(parent, dict): + cast(dict[str, object], parent)[final] = value + else: + raise KeyError(final) + except (KeyError, IndexError, ValueError): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) from None + + +def _require_string(value: object) -> str: + if not isinstance(value, str): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + return value + + +def _array_index(token: str, length: int) -> int: + if token != "0" and (not token.isascii() or not token.isdigit() or token.startswith("0")): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + index = int(token) + if index >= length: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + return index + + +def _reject_duplicate_keys(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + if key in result: + raise ValueError("duplicate JSON object key") + result[key] = value + return result + + +def _reject_nonfinite_number(_: str) -> float: + raise ValueError("non-finite JSON number") + + +def _parse_bounded_int(value: str, max_scalar_bytes: int) -> int: + if len(value.encode()) > max_scalar_bytes: + raise StructuredItemError(FailureCode.ITEM_TOO_LARGE) + return int(value) + + +def _parse_bounded_float(value: str, max_scalar_bytes: int) -> float: + if len(value.encode()) > max_scalar_bytes: + raise StructuredItemError(FailureCode.ITEM_TOO_LARGE) + parsed = float(value) + if not math.isfinite(parsed): + raise StructuredItemError(FailureCode.INVALID_SOURCE) + return parsed + + +def _detected_entity_values(inventory: object) -> tuple[str, ...]: + if not isinstance(inventory, dict) or set(inventory) != {"entities"}: + raise StructuredItemError(FailureCode.SEGMENT_PROCESSING_FAILED) + entities = cast(dict[str, object], inventory)["entities"] + if not isinstance(entities, list): + raise StructuredItemError(FailureCode.SEGMENT_PROCESSING_FAILED) + values: list[str] = [] + for entity in cast(list[object], entities): + if not isinstance(entity, dict): + raise StructuredItemError(FailureCode.SEGMENT_PROCESSING_FAILED) + entity_value = cast(dict[str, object], entity).get("value") + if not isinstance(entity_value, str) or not entity_value: + raise StructuredItemError(FailureCode.SEGMENT_PROCESSING_FAILED) + values.append(entity_value) + return tuple(values) + + +def _snapshot_detected_values(result_dataframe: pd.DataFrame) -> Mapping[str, tuple[str, ...]]: + if not isinstance(result_dataframe, pd.DataFrame): + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + labels = result_dataframe.columns.tolist() + if labels.count(SEGMENT_KEY_COLUMN) != 1 or labels.count(COL_FINAL_ENTITIES) != 1: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + snapshot: dict[str, tuple[str, ...]] = {} + for _, row in result_dataframe.iterrows(): + segment_key = row[SEGMENT_KEY_COLUMN] + if not isinstance(segment_key, str) or segment_key in snapshot: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + snapshot[segment_key] = _detected_entity_values(row[COL_FINAL_ENTITIES]) + return MappingProxyType(snapshot) + + +def _sha256_value(value: object) -> str: + canonical = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode() + return hashlib.sha256(canonical).hexdigest() + + +def _verify_unchanged_fields(document: object, expected_digests: Mapping[str, str]) -> None: + for pointer, expected_digest in expected_digests.items(): + if _sha256_value(_resolve_pointer(document, pointer)) != expected_digest: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + + +def _verify_identity(document: object, manifest: ReconstructionManifest) -> None: + if _require_string(_resolve_pointer(document, manifest.source_identity_pointer)) != manifest.source_identity: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) + source_order = tuple( + _require_string(_resolve_pointer(document, pointer)) for pointer in manifest.ordered_identity_pointers + ) + if source_order != manifest.source_order: + raise StructuredItemError(FailureCode.MAPPING_MISMATCH) diff --git a/tests/streaming/test_intake_dogfood.py b/tests/streaming/test_intake_dogfood.py new file mode 100644 index 00000000..fc68232d --- /dev/null +++ b/tests/streaming/test_intake_dogfood.py @@ -0,0 +1,632 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Opt-in dogfood against an operator-owned NeMo Platform Intake service. + +Set ``ANONYMIZER_INTAKE_DOGFOOD_BASE_URL`` to the service origin, for example +``http://127.0.0.1:8080``. The default profile protects each request and checks +its declared synthetic PII before crossing the HTTP boundary. Set +``ANONYMIZER_INTAKE_DOGFOOD_ALLOW_RAW=1`` only for isolated characterization +that intentionally sends raw synthetic fixtures. These tests never start, +configure, stop, or clean up Intake or its storage. +""" + +from __future__ import annotations + +import json +import os +import time +from collections.abc import Mapping +from pathlib import Path +from urllib.error import HTTPError +from urllib.parse import urlencode +from urllib.request import Request, urlopen +from uuid import uuid4 + +import pytest +from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest + +from tests.streaming.intake_format_validation import ( + IntakeValidationError, + build_local_otlp_request, + protect_atif, + protect_chat_completion, + protect_otlp_request, +) +from tests.streaming.sandbox_session_export import export_codex_session_to_atif +from tests.streaming.structured_trace_prototype import build_synthetic_protection_flow + +FIXTURES = Path(__file__).parents[1] / "fixtures" / "streaming" +_BASE_URL_ENV = "ANONYMIZER_INTAKE_DOGFOOD_BASE_URL" +_ALLOW_RAW_ENV = "ANONYMIZER_INTAKE_DOGFOOD_ALLOW_RAW" +_SANDBOX_RUN_DIR_ENV = "ANONYMIZER_SANDBOX_DOGFOOD_RUN_DIR" + +pytestmark = pytest.mark.integration + + +@pytest.fixture(scope="module") +def intake_base_url() -> str: + base_url = os.environ.get(_BASE_URL_ENV) + if base_url is None: + pytest.skip(f"set {_BASE_URL_ENV} to run Intake dogfood") + origin = base_url.rstrip("/") + status, body = _request(origin, "/health/ready") + if status != 200 or json.loads(body) != {"status": "ready"}: + pytest.fail("configured Intake service is not ready") + return f"{origin}/apis/intake/v2/workspaces/default" + + +@pytest.fixture(scope="module") +def dogfood_run_id() -> str: + return uuid4().hex[:12] + + +def _flow(entities: Mapping[str, str]): + return build_synthetic_protection_flow(entities) + + +def _dogfood_variants() -> tuple[str, ...]: + if os.environ.get(_ALLOW_RAW_ENV) == "1": + return ("raw", "protected") + return ("protected",) + + +def _assert_safe_outbound(body: bytes, sensitive: tuple[str, ...]) -> None: + assert all(value.encode() not in body for value in sensitive) + assert b"[REDACTED]" in body + + +def _request( + base_url: str, + path: str, + *, + body: bytes | None = None, + content_type: str | None = None, +) -> tuple[int, bytes]: + headers = {"Content-Type": content_type} if content_type else {} + method = "POST" if body is not None else "GET" + request = Request(f"{base_url}{path}", data=body, headers=headers, method=method) + try: + with urlopen(request, timeout=20) as response: + return response.status, response.read() + except HTTPError as error: + pytest.fail(f"Intake {method} {path} returned HTTP {error.code}") + + +class _IntakeEmitter: + def __init__(self, base_url: str, path: str, content_type: str) -> None: + self._base_url = base_url + self._path = path + self._content_type = content_type + self.calls = 0 + + def __call__(self, body: bytes) -> None: + self.calls += 1 + status, _ = _request( + self._base_url, + self._path, + body=body, + content_type=self._content_type, + ) + assert status == 200 + + +class _DogfoodDeliveryError(RuntimeError): + """Sanitized test-only protected-payload delivery failure.""" + + +def _deliver( + base_url: str, + path: str, + *, + body: bytes, + content_type: str, +) -> tuple[int, bytes]: + result: tuple[int, bytes] | None = None + failed = False + try: + result = _request(base_url, path, body=body, content_type=content_type) + except OSError: + failed = True + if failed: + raise _DogfoodDeliveryError("Protected payload delivery failed") from None + if result is None: # pragma: no cover - defensive typing guard + raise RuntimeError("Protected payload delivery returned no result") + return result + + +def _deliver_and_forget( + base_url: str, + path: str, + *, + body: bytes, + content_type: str, +) -> None: + _deliver(base_url, path, body=body, content_type=content_type) + + +def _query_spans(base_url: str, filters: Mapping[str, str]) -> list[dict[str, object]]: + query_values = {f"filter[{key}]": value for key, value in filters.items()} + query_values.update({"page": "1", "page_size": "100"}) + query = urlencode(query_values) + status, body = _request(base_url, f"/spans?{query}") + if status != 200: + pytest.fail("Intake span query did not return HTTP 200") + return json.loads(body)["data"] + + +def _stored_spans(base_url: str, filters: Mapping[str, str]) -> list[dict[str, object]]: + for _ in range(30): + data = _query_spans(base_url, filters) + if data: + return data + time.sleep(0.2) + pytest.fail("Intake did not expose the ingested synthetic spans") + + +def _stable_spans( + base_url: str, + filters: Mapping[str, str], + *, + expected_count: int, +) -> list[dict[str, object]]: + previous = "" + stable_reads = 0 + for _ in range(30): + data = _query_spans(base_url, filters) + if len(data) > expected_count: + pytest.fail("Intake exposed duplicate synthetic spans") + snapshot = json.dumps(data, sort_keys=True) + if len(data) == expected_count and snapshot == previous: + stable_reads += 1 + else: + stable_reads = 1 if len(data) == expected_count else 0 + if stable_reads == 3: + return data + previous = snapshot + time.sleep(0.2) + pytest.fail("Intake read model did not reach the expected stable state") + + +def _assert_visibility( + base_url: str, + session_id: str, + sensitive: tuple[str, ...], + *, + expect_sensitive: bool, +) -> list[dict[str, object]]: + stored = _stored_spans(base_url, {"session_id": session_id}) + rendered = json.dumps(stored) + present = [value for value in sensitive if value in rendered] + assert bool(present) is expect_sensitive + if not expect_sensitive: + assert "[REDACTED]" in rendered + return stored + + +@pytest.mark.parametrize( + ("fixture_name", "version", "entities"), + [ + ( + "intake_atif_v10.json", + "v10", + {"Alice": "person", "alice@example.test": "email", "Acme": "organization"}, + ), + ( + "intake_atif_v17.json", + "v17", + {"Bob": "person", "bob@example.test": "email", "Acme": "organization"}, + ), + ], +) +def test_atif_round_trips_through_intake( + intake_base_url: str, + dogfood_run_id: str, + fixture_name: str, + version: str, + entities: dict[str, str], +) -> None: + original = json.loads((FIXTURES / fixture_name).read_bytes()) + for variant in _dogfood_variants(): + document = json.loads(json.dumps(original)) + session_id = f"sdk-dogfood-{dogfood_run_id}-atif-{version}-{variant}" + document["session_id"] = session_id + document["trajectory_id"] = f"trajectory-{dogfood_run_id}-{version}-{variant}" + body = json.dumps(document, separators=(",", ":")).encode() + if variant == "protected": + emitted: list[bytes] = [] + body = protect_atif(body, flow=_flow(entities), emit=emitted.append) + assert emitted == [body] + _assert_safe_outbound(body, tuple(entities)) + + status, _ = _request( + intake_base_url, + "/ingest/atif", + body=body, + content_type="application/json", + ) + + assert status == 201 + stored = _assert_visibility( + intake_base_url, + session_id, + tuple(entities), + expect_sensitive=variant == "raw", + ) + assert len({span["trace_id"] for span in stored}) == 1 + assert any(span.get("parent_span_id") is not None for span in stored) + + +def test_chat_completion_round_trips_through_intake( + intake_base_url: str, + dogfood_run_id: str, +) -> None: + original = json.loads((FIXTURES / "intake_chat_completion.json").read_bytes()) + entities = {"Carol": "person", "carol@example.test": "email", "Acme": "organization"} + for variant in _dogfood_variants(): + document = json.loads(json.dumps(original)) + session_id = f"sdk-dogfood-{dogfood_run_id}-chat-{variant}" + document["session_id"] = session_id + document["trace_id"] = f"sdk-dogfood-{dogfood_run_id}-chat-{variant}" + document["response"]["created"] = int(time.time()) + body = json.dumps(document, separators=(",", ":")).encode() + if variant == "protected": + emitted: list[bytes] = [] + body = protect_chat_completion(body, flow=_flow(entities), emit=emitted.append) + assert emitted == [body] + _assert_safe_outbound(body, tuple(entities)) + + status, _ = _request( + intake_base_url, + "/ingest/chat-completions", + body=body, + content_type="application/json", + ) + + assert status == 201 + stored = _assert_visibility( + intake_base_url, + session_id, + tuple(entities), + expect_sensitive=variant == "raw", + ) + assert len(stored) == 1 + + +def _otlp_variant( + source: bytes, + *, + variant: int, + session_id: str, + agent_name: str, +) -> bytes: + message = ExportTraceServiceRequest.FromString(source) + spans = message.resource_spans[0].scope_spans[0].spans + trace_id = variant.to_bytes(16, "big") + new_ids = {span.span_id: (variant * 16 + index).to_bytes(8, "big") for index, span in enumerate(spans, start=1)} + for span in spans: + parent = bytes(span.parent_span_id) + old_id = bytes(span.span_id) + span.trace_id = trace_id + span.span_id = new_ids[old_id] + if parent: + span.parent_span_id = new_ids[parent] + for attribute in span.attributes: + if attribute.key == "gen_ai.conversation.id": + attribute.value.string_value = session_id + elif attribute.key == "gen_ai.agent.name": + attribute.value.string_value = agent_name + return message.SerializeToString(deterministic=True) + + +def test_otlp_round_trip_and_agent_name_filter( + intake_base_url: str, + dogfood_run_id: str, +) -> None: + source = build_local_otlp_request(FIXTURES / "intake_local_otlp_trace.json") + entities = {"Dave": "person", "dave@example.test": "email", "Acme": "organization"} + protected_session = "" + protected_agent = "" + for index, variant in enumerate(_dogfood_variants(), start=1): + session_id = f"sdk-dogfood-{dogfood_run_id}-otlp-{variant}" + agent_name = f"sdk-validation-agent-{dogfood_run_id}-{variant}" + body = _otlp_variant( + source, + variant=int(dogfood_run_id[:8], 16) + index, + session_id=session_id, + agent_name=agent_name, + ) + if variant == "protected": + emitted: list[bytes] = [] + body = protect_otlp_request(body, flow=_flow(entities), emit=emitted.append) + assert emitted == [body] + _assert_safe_outbound(body, tuple(entities)) + protected_session = session_id + protected_agent = agent_name + + status, response_body = _request( + intake_base_url, + "/ingest/otlp/v1/traces", + body=body, + content_type="application/x-protobuf", + ) + + assert status == 200 + assert json.loads(response_body)["errors"] == [] + stored = _assert_visibility( + intake_base_url, + session_id, + tuple(entities), + expect_sensitive=variant == "raw", + ) + assert len(stored) == 2 + assert sum(span.get("parent_span_id") is not None for span in stored) == 1 + + filtered = _stored_spans(intake_base_url, {"agent_name": protected_agent}) + assert len(filtered) == 1 + assert filtered[0]["agent_name"] == protected_agent + assert filtered[0]["session_id"] == protected_session + + +def test_invalid_otlp_exposes_adapter_intake_atomicity_mismatch( + intake_base_url: str, + dogfood_run_id: str, +) -> None: + if os.environ.get(_ALLOW_RAW_ENV) != "1": + pytest.skip(f"set {_ALLOW_RAW_ENV}=1 only for isolated raw characterization") + source = build_local_otlp_request(FIXTURES / "intake_local_otlp_trace.json") + session_id = f"sdk-dogfood-{dogfood_run_id}-otlp-partial" + variant = int(dogfood_run_id[:8], 16) + 3 + request = ExportTraceServiceRequest.FromString( + _otlp_variant( + source, + variant=variant, + session_id=session_id, + agent_name=f"sdk-validation-agent-{dogfood_run_id}-partial", + ) + ) + invalid = request.resource_spans[0].scope_spans[0].spans.add() + invalid.span_id = (variant * 16 + 3).to_bytes(8, "big") + invalid.name = "missing-trace-id" + body = request.SerializeToString(deterministic=True) + emitted: list[bytes] = [] + + with pytest.raises(IntakeValidationError, match="OTLP batch rejected"): + protect_otlp_request(body, flow=_flow({}), emit=emitted.append) + + assert emitted == [] + status, response_body = _request( + intake_base_url, + "/ingest/otlp/v1/traces", + body=body, + content_type="application/x-protobuf", + ) + stored = _stored_spans(intake_base_url, {"session_id": session_id}) + + assert status == 200 + assert len(json.loads(response_body)["errors"]) == 1 + assert len(stored) == 2 + + +def test_invalid_otlp_does_not_cross_intake_boundary( + intake_base_url: str, + dogfood_run_id: str, +) -> None: + source = build_local_otlp_request(FIXTURES / "intake_local_otlp_trace.json") + session_id = f"sdk-dogfood-{dogfood_run_id}-otlp-withheld" + variant = int(dogfood_run_id[:8], 16) + 4 + request = ExportTraceServiceRequest.FromString( + _otlp_variant( + source, + variant=variant, + session_id=session_id, + agent_name=f"sdk-validation-agent-{dogfood_run_id}-withheld", + ) + ) + invalid = request.resource_spans[0].scope_spans[0].spans.add() + invalid.span_id = (variant * 16 + 3).to_bytes(8, "big") + invalid.name = "missing-trace-id" + emitter = _IntakeEmitter( + intake_base_url, + "/ingest/otlp/v1/traces", + "application/x-protobuf", + ) + + with pytest.raises(IntakeValidationError, match="OTLP batch rejected"): + protect_otlp_request( + request.SerializeToString(deterministic=True), + flow=_flow({}), + emit=emitter, + ) + + assert emitter.calls == 0 + assert _query_spans(intake_base_url, {"session_id": session_id}) == [] + + +def test_protected_delivery_failure_is_sanitized_and_retryable( + intake_base_url: str, + dogfood_run_id: str, +) -> None: + document = json.loads((FIXTURES / "intake_chat_completion.json").read_bytes()) + entities = {"Carol": "person", "carol@example.test": "email", "Acme": "organization"} + session_id = f"sdk-dogfood-{dogfood_run_id}-delivery-unavailable" + document["session_id"] = session_id + document["trace_id"] = f"sdk-dogfood-{dogfood_run_id}-delivery-unavailable" + document["response"]["created"] = int(time.time()) + emitted: list[bytes] = [] + protected = protect_chat_completion( + json.dumps(document, separators=(",", ":")).encode(), + flow=_flow(entities), + emit=emitted.append, + ) + _assert_safe_outbound(protected, tuple(entities)) + + with pytest.raises(_DogfoodDeliveryError, match="Protected payload delivery failed") as exc_info: + _deliver( + "http://127.0.0.1:1/apis/intake/v2/workspaces/default", + "/ingest/chat-completions", + body=protected, + content_type="application/json", + ) + + assert emitted == [protected] + assert exc_info.value.__cause__ is None + assert exc_info.value.__context__ is None + assert _query_spans(intake_base_url, {"session_id": session_id}) == [] + + status, _ = _deliver( + intake_base_url, + "/ingest/chat-completions", + body=protected, + content_type="application/json", + ) + assert status == 201 + stored = _stable_spans(intake_base_url, {"session_id": session_id}, expected_count=1) + assert "[REDACTED]" in json.dumps(stored) + + +def _assert_exact_retry_deduplicates( + intake_base_url: str, + *, + session_id: str, + path: str, + content_type: str, + body: bytes, + expected_status: int, + expected_count: int, + expected_parent_count: int, +) -> None: + _deliver_and_forget(intake_base_url, path, body=body, content_type=content_type) + after_commit = _stable_spans( + intake_base_url, + {"session_id": session_id}, + expected_count=expected_count, + ) + status, _ = _deliver(intake_base_url, path, body=body, content_type=content_type) + assert status == expected_status + after_retry = _stable_spans( + intake_base_url, + {"session_id": session_id}, + expected_count=expected_count, + ) + + assert len(after_retry) == len(after_commit) == expected_count + assert {span["span_id"] for span in after_retry} == {span["span_id"] for span in after_commit} + assert sum(span.get("parent_span_id") is not None for span in after_retry) == expected_parent_count + + +def test_atif_exact_retry_deduplicates( + intake_base_url: str, + dogfood_run_id: str, +) -> None: + document = json.loads((FIXTURES / "intake_atif_v10.json").read_bytes()) + entities = {"Alice": "person", "alice@example.test": "email", "Acme": "organization"} + session_id = f"sdk-dogfood-{dogfood_run_id}-retry-atif" + document["session_id"] = session_id + document["trajectory_id"] = f"trajectory-{dogfood_run_id}-retry-atif" + body = protect_atif( + json.dumps(document, separators=(",", ":")).encode(), + flow=_flow(entities), + emit=lambda _: None, + ) + _assert_safe_outbound(body, tuple(entities)) + _assert_exact_retry_deduplicates( + intake_base_url, + session_id=session_id, + path="/ingest/atif", + content_type="application/json", + body=body, + expected_status=201, + expected_count=4, + expected_parent_count=3, + ) + + +def test_completed_sandbox_session_is_protected_before_intake( + intake_base_url: str, + dogfood_run_id: str, +) -> None: + run_dir_value = os.environ.get(_SANDBOX_RUN_DIR_ENV) + if run_dir_value is None: + pytest.skip(f"set {_SANDBOX_RUN_DIR_ENV} to export a completed Sandbox Codex run") + entities = { + "Mira Testperson": "person", + "mira.sandbox@example.test": "email", + "555-0109": "phone_number", + "Acme Validation Lab": "organization", + } + session_id = f"sdk-dogfood-{dogfood_run_id}-sandbox" + source = export_codex_session_to_atif(Path(run_dir_value), session_id=session_id) + assert all(value.encode() in source for value in entities) + body = protect_atif(source, flow=_flow(entities), emit=lambda _: None) + _assert_safe_outbound(body, tuple(entities)) + + status, _ = _deliver( + intake_base_url, + "/ingest/atif", + body=body, + content_type="application/json", + ) + + assert status == 201 + stored = _assert_visibility(intake_base_url, session_id, tuple(entities), expect_sensitive=False) + assert len(stored) >= 2 + assert any(span.get("parent_span_id") is not None for span in stored) + + +def test_chat_exact_retry_deduplicates( + intake_base_url: str, + dogfood_run_id: str, +) -> None: + document = json.loads((FIXTURES / "intake_chat_completion.json").read_bytes()) + entities = {"Carol": "person", "carol@example.test": "email", "Acme": "organization"} + session_id = f"sdk-dogfood-{dogfood_run_id}-retry-chat" + document["session_id"] = session_id + document["trace_id"] = f"sdk-dogfood-{dogfood_run_id}-retry-chat" + document["response"]["created"] = int(time.time()) + body = protect_chat_completion( + json.dumps(document, separators=(",", ":")).encode(), + flow=_flow(entities), + emit=lambda _: None, + ) + _assert_safe_outbound(body, tuple(entities)) + _assert_exact_retry_deduplicates( + intake_base_url, + session_id=session_id, + path="/ingest/chat-completions", + content_type="application/json", + body=body, + expected_status=201, + expected_count=1, + expected_parent_count=0, + ) + + +def test_otlp_exact_retry_deduplicates( + intake_base_url: str, + dogfood_run_id: str, +) -> None: + source = build_local_otlp_request(FIXTURES / "intake_local_otlp_trace.json") + entities = {"Dave": "person", "dave@example.test": "email", "Acme": "organization"} + session_id = f"sdk-dogfood-{dogfood_run_id}-retry-otlp" + body = protect_otlp_request( + _otlp_variant( + source, + variant=int(dogfood_run_id[:8], 16) + 5, + session_id=session_id, + agent_name=f"sdk-validation-agent-{dogfood_run_id}-retry", + ), + flow=_flow(entities), + emit=lambda _: None, + ) + _assert_safe_outbound(body, tuple(entities)) + _assert_exact_retry_deduplicates( + intake_base_url, + session_id=session_id, + path="/ingest/otlp/v1/traces", + content_type="application/x-protobuf", + body=body, + expected_status=200, + expected_count=2, + expected_parent_count=1, + ) diff --git a/tests/streaming/test_intake_format_validation.py b/tests/streaming/test_intake_format_validation.py new file mode 100644 index 00000000..db7a9588 --- /dev/null +++ b/tests/streaming/test_intake_format_validation.py @@ -0,0 +1,242 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any + +import pytest +from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest + +from tests.streaming.intake_format_validation import ( + IntakeValidationError, + build_local_otlp_request, + otlp_string_attributes, + otlp_topology, + protect_atif, + protect_chat_completion, + protect_otlp_request, +) +from tests.streaming.structured_trace_prototype import build_synthetic_protection_flow + +FIXTURES = Path(__file__).parents[1] / "fixtures" / "streaming" + + +def _flow(sensitive_entities: dict[str, str]): + return build_synthetic_protection_flow(sensitive_entities) + + +@pytest.mark.parametrize( + ("fixture_name", "sensitive_entities", "expected_version"), + [ + ( + "intake_atif_v10.json", + {"Alice": "person", "alice@example.test": "email", "Acme": "organization"}, + "ATIF-v1.0", + ), + ( + "intake_atif_v17.json", + {"Bob": "person", "bob@example.test": "email", "Acme": "organization"}, + "ATIF-v1.7", + ), + ], +) +def test_atif_boundary_versions_round_trip_through_plan_a( + fixture_name: str, + sensitive_entities: dict[str, str], + expected_version: str, +) -> None: + source = (FIXTURES / fixture_name).read_bytes() + original = json.loads(source) + emitted: list[bytes] = [] + + protected = protect_atif(source, flow=_flow(sensitive_entities), emit=emitted.append) + + assert protected + assert emitted == [protected] + result = json.loads(protected) + assert result["schema_version"] == expected_version + assert result["session_id"] == original["session_id"] + assert result["trajectory_id"] == original["trajectory_id"] + assert result["agent"] == original["agent"] + assert [step["step_id"] for step in result["steps"]] == [1, 2] + assert [step["source"] for step in result["steps"]] == ["user", "agent"] + if expected_version == "ATIF-v1.7": + assert result["steps"][1]["tool_calls"][0]["arguments"]["limit"] == 5 + rendered = json.dumps(result) + assert all(value not in rendered for value in sensitive_entities) + assert "[REDACTED]" in rendered + + +def test_chat_completion_preserves_extensions_and_protects_declared_content() -> None: + source = (FIXTURES / "intake_chat_completion.json").read_bytes() + original = json.loads(source) + emitted: list[bytes] = [] + entities = {"Carol": "person", "carol@example.test": "email", "Acme": "organization"} + + protected = protect_chat_completion(source, flow=_flow(entities), emit=emitted.append) + + assert emitted == [protected] + result = json.loads(protected) + assert result["request"]["model"] == original["request"]["model"] + assert result["request"]["provider_extension"] == original["request"]["provider_extension"] + assert result["response"]["provider_response_id"] == original["response"]["provider_response_id"] + assert result["response"]["created"] == original["response"]["created"] + assert result["response"]["usage"] == original["response"]["usage"] + assert result["session_id"] == original["session_id"] + rendered = json.dumps(result) + assert all(value not in rendered for value in entities) + + +def test_local_chain_llm_otlp_protobuf_round_trips_through_plan_a() -> None: + source = build_local_otlp_request(FIXTURES / "intake_local_otlp_trace.json") + original_topology = otlp_topology(source) + original_attributes = otlp_string_attributes(source) + emitted: list[bytes] = [] + entities = {"Dave": "person", "dave@example.test": "email", "Acme": "organization"} + + protected = protect_otlp_request(source, flow=_flow(entities), emit=emitted.append) + + assert emitted == [protected] + assert otlp_topology(protected) == original_topology + result_attributes = otlp_string_attributes(protected) + assert result_attributes["0000000000000001"] == original_attributes["0000000000000001"] + assert result_attributes["0000000000000002"]["gen_ai.agent.name"] == "sdk-validation-agent" + assert result_attributes["0000000000000002"]["gen_ai.request.model"] == "gpt-validation" + rendered = json.dumps(result_attributes) + assert all(value not in rendered for value in entities) + assert "[REDACTED]" in rendered + + +def test_invalid_otlp_span_withholds_complete_batch() -> None: + source = build_local_otlp_request(FIXTURES / "intake_local_otlp_trace.json") + request = ExportTraceServiceRequest.FromString(source) + invalid = request.resource_spans[0].scope_spans[0].spans.add() + invalid.span_id = bytes.fromhex("0000000000000003") + invalid.name = "missing-trace-id" + emitted: list[bytes] = [] + + with pytest.raises(IntakeValidationError, match="OTLP batch rejected"): + protect_otlp_request(request.SerializeToString(), flow=_flow({}), emit=emitted.append) + + assert emitted == [] + + +def test_non_success_plan_a_outcome_withholds_json_item() -> None: + source = (FIXTURES / "intake_atif_v10.json").read_bytes() + flow = _flow({"Alice": "person"}) + flow.close() + emitted: list[bytes] = [] + + with pytest.raises(IntakeValidationError, match="protection failed"): + protect_atif(source, flow=flow, emit=emitted.append) + + assert emitted == [] + + +def test_atif_unknown_top_level_field_is_rejected() -> None: + document: dict[str, Any] = json.loads((FIXTURES / "intake_atif_v10.json").read_bytes()) + document["undeclared_content"] = "Alice" + + with pytest.raises(IntakeValidationError, match="ATIF item rejected"): + protect_atif(json.dumps(document).encode(), flow=_flow({"Alice": "person"}), emit=lambda _: None) + + +def test_chat_completion_requires_exactly_one_response_variant() -> None: + document: dict[str, Any] = json.loads((FIXTURES / "intake_chat_completion.json").read_bytes()) + document["response"]["error"] = {"message": "Carol failed"} + + with pytest.raises(IntakeValidationError, match="chat completion rejected"): + protect_chat_completion(json.dumps(document).encode(), flow=_flow({"Carol": "person"}), emit=lambda _: None) + + +def test_chat_completion_rejects_unreviewed_nested_provider_content() -> None: + document: dict[str, Any] = json.loads((FIXTURES / "intake_chat_completion.json").read_bytes()) + document["response"]["provider_secret"] = "eve@example.test" + + with pytest.raises(IntakeValidationError, match="chat completion rejected"): + protect_chat_completion(json.dumps(document).encode(), flow=_flow({}), emit=lambda _: None) + + +def test_chat_completion_requires_stable_creation_time() -> None: + document: dict[str, Any] = json.loads((FIXTURES / "intake_chat_completion.json").read_bytes()) + del document["response"]["created"] + + with pytest.raises(IntakeValidationError, match="chat completion rejected"): + protect_chat_completion(json.dumps(document).encode(), flow=_flow({}), emit=lambda _: None) + + +@pytest.mark.parametrize("created", [True, 0, 10**100]) +def test_chat_completion_rejects_invalid_creation_time(created: object) -> None: + document: dict[str, Any] = json.loads((FIXTURES / "intake_chat_completion.json").read_bytes()) + document["response"]["created"] = created + + with pytest.raises(IntakeValidationError, match="chat completion rejected"): + protect_chat_completion(json.dumps(document).encode(), flow=_flow({}), emit=lambda _: None) + + +def test_chat_completion_rejects_future_creation_time() -> None: + document: dict[str, Any] = json.loads((FIXTURES / "intake_chat_completion.json").read_bytes()) + document["response"]["created"] = int(time.time()) + 3_600 + + with pytest.raises(IntakeValidationError, match="chat completion rejected"): + protect_chat_completion(json.dumps(document).encode(), flow=_flow({}), emit=lambda _: None) + + +def test_atif_rejects_unreviewed_image_content_part() -> None: + document: dict[str, Any] = json.loads((FIXTURES / "intake_atif_v17.json").read_bytes()) + document["steps"][0]["message"].append( + {"type": "image", "source": {"media_type": "image/png", "path": "/private/alice.png"}} + ) + + with pytest.raises(IntakeValidationError, match="ATIF item rejected"): + protect_atif(json.dumps(document).encode(), flow=_flow({}), emit=lambda _: None) + + +def test_otlp_rejects_unreviewed_resource_and_event_content() -> None: + source = build_local_otlp_request(FIXTURES / "intake_local_otlp_trace.json") + request = ExportTraceServiceRequest.FromString(source) + resource_attribute = request.resource_spans[0].resource.attributes.add() + resource_attribute.key = "deployment.secret" + resource_attribute.value.string_value = "alice@example.test" + + with pytest.raises(IntakeValidationError, match="OTLP batch rejected"): + protect_otlp_request(request.SerializeToString(), flow=_flow({}), emit=lambda _: None) + + request = ExportTraceServiceRequest.FromString(source) + event = request.resource_spans[0].scope_spans[0].spans[1].events.add() + event.name = "exception" + event_attribute = event.attributes.add() + event_attribute.key = "exception.message" + event_attribute.value.string_value = "alice@example.test" + + with pytest.raises(IntakeValidationError, match="OTLP batch rejected"): + protect_otlp_request(request.SerializeToString(), flow=_flow({}), emit=lambda _: None) + + +@pytest.mark.parametrize("invalid_value", ["", False]) +def test_otlp_rejects_invalid_agent_name(invalid_value: str | bool) -> None: + request = ExportTraceServiceRequest.FromString(build_local_otlp_request(FIXTURES / "intake_local_otlp_trace.json")) + attributes = request.resource_spans[0].scope_spans[0].spans[1].attributes + agent_name = next(attribute for attribute in attributes if attribute.key == "gen_ai.agent.name") + if isinstance(invalid_value, str): + agent_name.value.string_value = invalid_value + else: + agent_name.value.bool_value = invalid_value + + with pytest.raises(IntakeValidationError, match="OTLP batch rejected"): + protect_otlp_request(request.SerializeToString(), flow=_flow({}), emit=lambda _: None) + + +def test_otlp_rejects_duplicate_attribute_keys() -> None: + request = ExportTraceServiceRequest.FromString(build_local_otlp_request(FIXTURES / "intake_local_otlp_trace.json")) + attributes = request.resource_spans[0].scope_spans[0].spans[1].attributes + duplicate = attributes.add() + duplicate.key = "input.value" + duplicate.value.string_value = "duplicate" + + with pytest.raises(IntakeValidationError, match="OTLP batch rejected"): + protect_otlp_request(request.SerializeToString(), flow=_flow({}), emit=lambda _: None) diff --git a/tests/streaming/test_openshell_ocsf_process_adapter.py b/tests/streaming/test_openshell_ocsf_process_adapter.py new file mode 100644 index 00000000..adea2d51 --- /dev/null +++ b/tests/streaming/test_openshell_ocsf_process_adapter.py @@ -0,0 +1,163 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Contract tests for the test-only OpenShell OCSF JSONL replay adapter.""" + +from __future__ import annotations + +import json +from collections.abc import Callable +from pathlib import Path +from typing import cast + +import pytest + +from anonymizer.engine.constants import COL_TEXT +from tests.streaming.openshell_ocsf_process_adapter import ( + FIDELITY_CLASS, + MAPPING_VERSION, + project_process_activity_item, + protect_process_activity_item, + replay_process_activity_corpus, +) +from tests.streaming.structured_trace_prototype import ( + SEGMENT_KEY_COLUMN, + CodecBounds, + FailureCode, + StructuredItemError, + build_synthetic_anonymizer, +) + +CORPUS = Path(__file__).parents[1] / "fixtures" / "streaming" / "openshell_process_activity.jsonl" +SENSITIVE_ENTITIES = { + "alice@example.test": "email", + "alice-host.example.test": "hostname", + "alice-workspace": "workspace", + "alice-agent": "process_name", + "registry.example.test/alice/agent:latest": "container_image", +} + + +@pytest.fixture +def bounds() -> CodecBounds: + return CodecBounds( + max_bytes=8_192, + max_depth=12, + max_targets=16, + max_scalars=64, + max_scalar_bytes=1_024, + max_events=256, + ) + + +def test_openshell_process_activity_replay_is_keyed_and_complete_item_buffered(bounds: CodecBounds) -> None: + source = CORPUS.read_bytes() + emitted: list[bytes] = [] + + protected = replay_process_activity_corpus( + source, + max_corpus_bytes=32_768, + max_records=8, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=CORPUS, + emit=emitted.append, + ) + + assert tuple(emitted) == protected + assert len(protected) == 2 + assert all(item.endswith(b"\n") and item.count(b"\n") == 1 for item in protected) + assert all(value.encode() not in b"".join(protected) for value in SENSITIVE_ENTITIES) + + source_items = [json.loads(line) for line in source.splitlines()] + protected_items = [json.loads(line) for line in protected] + source_order = [(item["metadata"]["uid"], item["time"], item["type_uid"]) for item in source_items] + protected_order = [(item["metadata"]["uid"], item["time"], item["type_uid"]) for item in protected_items] + assert protected_order == source_order + assert [item["activity_name"] for item in protected_items] == ["Launch", "Terminate"] + + +def test_mapping_and_manifest_are_deterministic_and_source_specific(bounds: CodecBounds) -> None: + line = CORPUS.read_bytes().splitlines(keepends=True)[0] + first = project_process_activity_item(line, bounds=bounds) + second = project_process_activity_item(line, bounds=bounds) + + assert first.manifest.mapping_version == MAPPING_VERSION + assert first.manifest.fidelity == FIDELITY_CLASS + assert first.manifest.source_identity == "sandbox-alpha" + assert first.manifest.source_order == ("Process Activity: Launch",) + assert [segment.segment_key for segment in first.manifest.segments] == [ + segment.segment_key for segment in second.manifest.segments + ] + assert first.dataframe[SEGMENT_KEY_COLUMN].is_unique + target_text = "\n".join(first.dataframe[COL_TEXT]) + assert all(value in target_text for value in SENSITIVE_ENTITIES) + assert all(segment.segment_key.startswith(f"{MAPPING_VERSION}:/") for segment in first.manifest.segments) + + +@pytest.mark.parametrize( + ("mutation", "expected_code"), + [ + (lambda item: item.__setitem__("unmapped", {"private": "alice@example.test"}), FailureCode.MAPPING_MISMATCH), + (lambda item: item.__setitem__("type_uid", 100799), FailureCode.MAPPING_MISMATCH), + ( + lambda item: cast(dict[str, object], item["process"]).__setitem__( + "environment", "TOKEN=alice@example.test" + ), + FailureCode.MAPPING_MISMATCH, + ), + ], +) +def test_unknown_or_schema_invalid_fields_fail_closed_before_emission( + mutation: Callable[[dict[str, object]], None], + expected_code: FailureCode, + bounds: CodecBounds, +) -> None: + item = cast(dict[str, object], json.loads(CORPUS.read_bytes().splitlines()[0])) + mutation(item) + source = json.dumps(item, separators=(",", ":")).encode() + b"\n" + emitted: list[bytes] = [] + + with pytest.raises(StructuredItemError) as exc_info: + protect_process_activity_item( + source, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=CORPUS, + emit=emitted.append, + ) + + assert exc_info.value.code is expected_code + assert emitted == [] + assert "alice@example.test" not in str(exc_info.value) + + +def test_corpus_limits_and_incomplete_final_record_fail_before_emission(bounds: CodecBounds) -> None: + corpus = CORPUS.read_bytes() + emitted: list[bytes] = [] + + with pytest.raises(StructuredItemError) as count_error: + replay_process_activity_corpus( + corpus, + max_corpus_bytes=32_768, + max_records=1, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=CORPUS, + emit=emitted.append, + ) + assert count_error.value.code is FailureCode.ITEM_TOO_LARGE + assert emitted == [] + + with pytest.raises(StructuredItemError) as partial_error: + replay_process_activity_corpus( + corpus.rstrip(b"\n"), + max_corpus_bytes=32_768, + max_records=8, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=CORPUS, + emit=emitted.append, + ) + assert partial_error.value.code is FailureCode.INVALID_SOURCE + assert emitted == [] diff --git a/tests/streaming/test_private_row_verification.py b/tests/streaming/test_private_row_verification.py new file mode 100644 index 00000000..f73ad17c --- /dev/null +++ b/tests/streaming/test_private_row_verification.py @@ -0,0 +1,367 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Contract tests for Anonymizer's invocation-private row verification seam.""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path +from typing import Any, cast + +import pandas as pd +import pytest + +from anonymizer.config.anonymizer_config import AnonymizerConfig, AnonymizerInput +from anonymizer.config.replace_strategies import Redact +from anonymizer.engine.constants import COL_FINAL_ENTITIES, COL_TEXT +from anonymizer.engine.detection.detection_workflow import EntityDetectionResult +from anonymizer.engine.private_row_verification import ( + PRIVATE_CORRELATION_COLUMN, + PrivateRowVerificationError, + _InvocationRowVerifier, +) +from anonymizer.engine.replace.replace_runner import ReplacementResult +from anonymizer.engine.resolved_input import ResolvedInput +from tests.streaming.structured_trace_prototype import build_synthetic_anonymizer + + +def test_private_tracking_column_collision_is_rejected_without_exposing_text() -> None: + secret = "synthetic-secret@example.test" + frame = pd.DataFrame( + { + COL_TEXT: [secret], + "__anonymizer_private_row_correlation__": ["caller-value"], + } + ) + anonymizer = build_synthetic_anonymizer({secret: "email"}) + context = ResolvedInput(frame, requested_text_column="text", resolved_text_column="text") + + with pytest.raises(PrivateRowVerificationError) as exc_info: + anonymizer._run_internal( + config=AnonymizerConfig(replace=Redact(), emit_telemetry=False), + data=AnonymizerInput(source=str(Path(__file__)), text_column="text"), + context=context, + preview_num_records=None, + ) + + assert "private_column_collision" in str(exc_info.value) + assert secret not in str(exc_info.value) + + +def test_real_local_redact_seam_strips_private_correlation_from_public_result() -> None: + secret = "synthetic-secret@example.test" + frame = pd.DataFrame({COL_TEXT: [secret]}) + anonymizer = build_synthetic_anonymizer({secret: "email"}) + context = ResolvedInput(frame, requested_text_column="text", resolved_text_column="text") + + result = anonymizer._run_internal( + config=AnonymizerConfig(replace=Redact(), emit_telemetry=False), + data=AnonymizerInput(source=str(Path(__file__)), text_column="text"), + context=context, + preview_num_records=None, + ) + + assert PRIVATE_CORRELATION_COLUMN not in result.dataframe + assert PRIVATE_CORRELATION_COLUMN not in result.trace_dataframe + + +def test_real_engine_seam_sanitizes_pipeline_exception() -> None: + secret = "synthetic-secret@example.test" + frame = pd.DataFrame({COL_TEXT: [secret]}) + anonymizer = build_synthetic_anonymizer({secret: "email"}) + cast(Any, anonymizer._detection_workflow).run = lambda *_args, **_kwargs: (_ for _ in ()).throw( + RuntimeError(secret) + ) + context = ResolvedInput(frame, requested_text_column="text", resolved_text_column="text") + + with pytest.raises(PrivateRowVerificationError) as exc_info: + anonymizer._run_internal( + config=AnonymizerConfig(replace=Redact(), emit_telemetry=False), + data=AnonymizerInput(source=str(Path(__file__)), text_column="text"), + context=context, + preview_num_records=None, + ) + + assert "invocation_failed" in str(exc_info.value) + assert secret not in str(exc_info.value) + assert exc_info.value.__cause__ is None + assert exc_info.value.__context__ is None + + +@pytest.mark.parametrize( + ("transform", "raises", "expected_rows"), + [ + (lambda frame: frame.iloc[::-1].reset_index(drop=True), False, 2), + (lambda frame: frame.iloc[:1].copy(), False, 1), + (lambda frame: pd.concat([frame, frame.iloc[[0]]], ignore_index=True), True, None), + ( + lambda frame: frame.assign( + **{PRIVATE_CORRELATION_COLUMN: ["unknown", frame.iloc[1][PRIVATE_CORRELATION_COLUMN]]} + ), + True, + None, + ), + ], + ids=["reorder", "drop", "duplicate", "unknown"], +) +def test_real_local_engine_seam_accounts_for_correlation_transformations( + transform: Any, raises: bool, expected_rows: int | None +) -> None: + secret = "synthetic-secret@example.test" + frame = pd.DataFrame({COL_TEXT: [secret, "synthetic second"]}) + anonymizer = build_synthetic_anonymizer({secret: "email"}) + original_run = cast(Any, anonymizer._detection_workflow.run) + + def transformed_run(*args: Any, **kwargs: Any) -> EntityDetectionResult: + result = original_run(*args, **kwargs) + return EntityDetectionResult(dataframe=transform(result.dataframe), failed_records=result.failed_records) + + cast(Any, anonymizer._detection_workflow).run = transformed_run + context = ResolvedInput(frame, requested_text_column="text", resolved_text_column="text") + call = lambda: anonymizer._run_internal( + config=AnonymizerConfig(replace=Redact(), emit_telemetry=False), + data=AnonymizerInput(source=str(Path(__file__)), text_column="text"), + context=context, + preview_num_records=None, + ) + + if raises: + with pytest.raises(PrivateRowVerificationError) as exc_info: + call() + assert secret not in str(exc_info.value) + else: + assert len(call().dataframe) == expected_rows + + +def test_real_engine_seam_rejects_duplicate_reordered_legacy_detection_output() -> None: + secret = "synthetic-secret@example.test" + frame = pd.DataFrame({COL_TEXT: [secret, secret]}) + anonymizer = build_synthetic_anonymizer({secret: "email"}) + original_run = cast(Any, anonymizer._detection_workflow.run) + + def reordered_legacy_run(*args: Any, **kwargs: Any) -> EntityDetectionResult: + result = original_run(*args, **kwargs) + legacy = result.dataframe.iloc[::-1].reset_index(drop=True) + return EntityDetectionResult( + dataframe=legacy.drop(columns=[PRIVATE_CORRELATION_COLUMN]), + failed_records=result.failed_records, + ) + + cast(Any, anonymizer._detection_workflow).run = reordered_legacy_run + context = ResolvedInput(frame, requested_text_column="text", resolved_text_column="text") + + with pytest.raises(PrivateRowVerificationError) as exc_info: + anonymizer._run_internal( + config=AnonymizerConfig(replace=Redact(), emit_telemetry=False), + data=AnonymizerInput(source=str(Path(__file__)), text_column="text"), + context=context, + preview_num_records=None, + ) + + assert "invocation_failed" in str(exc_info.value) + assert secret not in str(exc_info.value) + + +def test_real_engine_seam_rejects_final_accepted_detection_removal() -> None: + secret = "synthetic-secret@example.test" + frame = pd.DataFrame({COL_TEXT: [secret]}) + anonymizer = build_synthetic_anonymizer({secret: "email"}) + original_run = cast(Any, anonymizer._replace_runner.run) + + def remove_accepted_detection(*args: Any, **kwargs: Any) -> ReplacementResult: + result = original_run(*args, **kwargs) + return ReplacementResult( + dataframe=result.dataframe.drop(columns=[COL_FINAL_ENTITIES]), + failed_records=result.failed_records, + ) + + cast(Any, anonymizer._replace_runner).run = remove_accepted_detection + context = ResolvedInput(frame, requested_text_column="text", resolved_text_column="text") + + with pytest.raises(PrivateRowVerificationError) as exc_info: + anonymizer._run_internal( + config=AnonymizerConfig(replace=Redact(), emit_telemetry=False), + data=AnonymizerInput(source=str(Path(__file__)), text_column="text"), + context=context, + preview_num_records=None, + ) + + assert "invocation_failed" in str(exc_info.value) + assert secret not in str(exc_info.value) + + +def _detected_frame() -> pd.DataFrame: + return pd.DataFrame( + { + COL_TEXT: ["synthetic alpha", "synthetic beta"], + "final_entities": [{"entities": [{"value": "alpha"}]}, {"entities": []}], + } + ) + + +def test_verifier_rejects_reordered_drop_duplicate_and_unknown_rows() -> None: + base = _detected_frame() + for transform, expected_code in ( + (lambda frame: frame.iloc[::-1].reset_index(drop=True), None), + (lambda frame: frame.iloc[:1].copy(), None), + (lambda frame: pd.concat([frame, frame.iloc[[0]]], ignore_index=True), "correlation_duplicate"), + ( + lambda frame: frame.assign( + **{PRIVATE_CORRELATION_COLUMN: ["unknown", frame.iloc[1][PRIVATE_CORRELATION_COLUMN]]} + ), + "correlation_unknown", + ), + ): + verifier = _InvocationRowVerifier(base) + bound = verifier.bind(base) + verifier.freeze_accepted_detections(bound) + candidate = transform(bound.copy()) + if expected_code is None: + verified = verifier.finish(candidate) + assert PRIVATE_CORRELATION_COLUMN not in verified + if len(candidate) == 1: + assert len(verified) == 1 + outcomes = [outcome.value for outcome in verifier._outcomes.values()] + assert outcomes.count("failed") == 1 + assert outcomes.count("success") == 1 + else: + with pytest.raises(PrivateRowVerificationError, match=expected_code): + verifier.finish(candidate) + + +def test_verifier_rejects_a_missing_correlation_instead_of_recovering_from_text() -> None: + base = _detected_frame() + verifier = _InvocationRowVerifier(base) + bound = verifier.bind(base) + verifier.freeze_accepted_detections(bound) + + with pytest.raises(PrivateRowVerificationError, match="correlation_missing"): + verifier.finish(bound.drop(columns=[PRIVATE_CORRELATION_COLUMN])) + + assert {outcome.value for outcome in verifier._outcomes.values()} == {"failed"} + + +def test_stage_output_never_receives_positional_correlation_rebinding() -> None: + base = _detected_frame() + verifier = _InvocationRowVerifier(base) + bound = verifier.bind(base) + + with pytest.raises(PrivateRowVerificationError, match="correlation_missing"): + verifier.bind_complete_stage_output(bound.iloc[::-1].drop(columns=[PRIVATE_CORRELATION_COLUMN])) + + +def test_row_failure_precedes_invocation_cancellation() -> None: + base = _detected_frame() + verifier = _InvocationRowVerifier(base) + bound = verifier.bind(base) + verifier.freeze_accepted_detections(bound.iloc[:1].copy()) + verifier.abort(cancelled=True) + + outcomes = [outcome.value for outcome in verifier._outcomes.values()] + assert outcomes.count("failed") == 1 + assert outcomes.count("cancelled") == 1 + + +def test_abort_sanitizes_underlying_failure_text() -> None: + verifier = _InvocationRowVerifier(_detected_frame()) + secret = "provider replied with synthetic-secret and engine-id-8675309" + + error = verifier.abort_with_failure(stage="replace", cause=RuntimeError(secret)) + + with pytest.raises(PrivateRowVerificationError) as exc_info: + raise error + + assert "invocation_failed" in str(exc_info.value) + assert secret not in str(exc_info.value) + assert exc_info.value.__cause__ is None + assert exc_info.value.__context__ is None + + +def test_verifier_rejects_accepted_detection_tampering_and_closes_without_raw_values() -> None: + base = _detected_frame() + verifier = _InvocationRowVerifier(base) + bound = verifier.bind(base) + verifier.freeze_accepted_detections(bound) + tampered = bound.copy() + tampered.at[0, "final_entities"] = {"entities": [{"value": "different-secret"}]} + + with pytest.raises(PrivateRowVerificationError) as exc_info: + verifier.finish(tampered) + + assert "accepted_detection_tampered" in str(exc_info.value) + assert "different-secret" not in str(exc_info.value) + with pytest.raises(PrivateRowVerificationError, match="invocation_closed"): + verifier.finish(bound) + + +def test_verifier_exception_never_exposes_private_correlation() -> None: + token = "private-row-token-canary" + base = _detected_frame() + verifier = _InvocationRowVerifier(base, correlations=(token, "private-row-token-peer")) + bound = verifier.bind(base) + verifier.freeze_accepted_detections(bound) + tampered = bound.copy() + tampered.at[0, COL_FINAL_ENTITIES] = {"entities": [{"value": "different"}]} + + with pytest.raises(PrivateRowVerificationError) as exc_info: + verifier.finish(tampered) + + assert token not in str(exc_info.value) + assert token not in repr(exc_info.value) + assert token not in repr(exc_info.value.failure) + + +def test_verifier_requires_frozen_accepted_detection_evidence_at_finish() -> None: + base = _detected_frame() + verifier = _InvocationRowVerifier(base) + bound = verifier.bind(base) + verifier.freeze_accepted_detections(bound) + + with pytest.raises(PrivateRowVerificationError, match="accepted_detection_missing"): + verifier.finish(bound.drop(columns=[COL_FINAL_ENTITIES])) + + +def test_verifier_is_nonserializable_and_cancellation_is_terminal() -> None: + import pickle + + verifier = _InvocationRowVerifier(_detected_frame()) + with pytest.raises(TypeError, match="not serializable"): + pickle.dumps(verifier) + verifier.abort(cancelled=True) + with pytest.raises(PrivateRowVerificationError, match="invocation_closed"): + verifier.bind(_detected_frame()) + + +def test_characterization_reports_governed_unavailability_without_zero_measurements() -> None: + completed = subprocess.run( + [sys.executable, "tests/streaming/run_internal_characterization.py"], + check=True, + capture_output=True, + text=True, + ) + report = json.loads(completed.stdout) + arms = {arm["arm"]: arm for arm in report["arms"]} + blocked = arms["generic_manifest"] + + assert set(blocked) == set(arms["field_per_row"]) + assert blocked["status"] == "blocked" + assert blocked["availability"] == "governed_unavailable" + assert blocked["reason_code"] == "source_specific_manifest_not_owned" + for metric in ( + "input_bytes", + "output_bytes", + "rows", + "targets", + "provider_calls", + "elapsed_ms", + "peak_memory_bytes", + "raw_copy_count", + "artifact_delta_bytes", + "structural_validity", + "privacy_check", + "reconstruction_failures", + ): + assert blocked[metric] is None diff --git a/tests/streaming/test_sandbox_session_export.py b/tests/streaming/test_sandbox_session_export.py new file mode 100644 index 00000000..a1590f01 --- /dev/null +++ b/tests/streaming/test_sandbox_session_export.py @@ -0,0 +1,97 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from tests.streaming.sandbox_session_export import SandboxSessionExportError, export_codex_session_to_atif + + +def _write_run(run_dir: Path, *, item_type: str = "command_execution") -> None: + manifest = { + "runtime_version": "0.18.12", + "model": "test-model", + "started_at": "2026-08-19T00:00:00Z", + "finished_at": "2026-08-19T00:00:01Z", + "artifacts": { + "provenance": { + "dispatch": {"agent": "codex"}, + "prompt": {"path": "/tmp/prompt.md", "exists": True, "type": "file"}, + } + }, + } + status = {"state": "completed", "exit_code": 0} + item = { + "id": "item-1", + "type": item_type, + "command": "printf 'Alice'", + "aggregated_output": "Alice", + "exit_code": 0, + "status": "completed", + } + events = [ + {"type": "thread.started", "thread_id": "thread-1"}, + {"type": "turn.started"}, + {"type": "item.completed", "item": {"id": "item-0", "type": "agent_message", "text": "Alice"}}, + {"type": "item.completed", "item": item}, + { + "type": "item.completed", + "item": { + "id": "item-2", + "type": "file_change", + "changes": [{"path": "/workspace/Alice.txt", "kind": "add"}], + "status": "completed", + }, + }, + {"type": "turn.completed", "usage": {"input_tokens": 12, "output_tokens": 3}}, + ] + run_dir.mkdir() + (run_dir / "manifest.json").write_text(json.dumps(manifest)) + (run_dir / "status.json").write_text(json.dumps(status)) + (run_dir / "prompt.md").write_text("Contact Alice") + (run_dir / "agent-output.jsonl").write_text("\n".join(json.dumps(event) for event in events)) + + +def test_exports_completed_codex_session_to_atif(tmp_path: Path) -> None: + run_dir = tmp_path / "run" + _write_run(run_dir) + + document = json.loads(export_codex_session_to_atif(run_dir, session_id="session-1")) + + assert document["schema_version"] == "ATIF-v1.0" + assert document["session_id"] == "session-1" + assert document["trajectory_id"] == "thread-1" + assert [step["step_id"] for step in document["steps"]] == [1, 2, 3, 4] + assert document["steps"][0]["message"] == "Contact Alice" + assert document["steps"][1]["message"] == "Alice" + assert document["steps"][2]["tool_calls"][0]["arguments"] == {"command": "printf 'Alice'"} + assert document["steps"][2]["observation"]["results"][0]["content"] == "Alice" + assert document["steps"][3]["tool_calls"][0]["arguments"] == { + "changes": [{"path": "/workspace/Alice.txt", "kind": "add"}] + } + assert document["final_metrics"] == { + "total_prompt_tokens": 12, + "total_completion_tokens": 3, + "total_steps": 4, + } + + +def test_rejects_unreviewed_completed_item(tmp_path: Path) -> None: + run_dir = tmp_path / "run" + _write_run(run_dir, item_type="web_search") + + with pytest.raises(SandboxSessionExportError, match="item type"): + export_codex_session_to_atif(run_dir, session_id="session-1") + + +def test_rejects_incomplete_run(tmp_path: Path) -> None: + run_dir = tmp_path / "run" + _write_run(run_dir) + (run_dir / "status.json").write_text(json.dumps({"state": "running", "exit_code": 0})) + + with pytest.raises(SandboxSessionExportError, match="not successfully completed"): + export_codex_session_to_atif(run_dir, session_id="session-1") diff --git a/tests/streaming/test_structured_trace_prototype.py b/tests/streaming/test_structured_trace_prototype.py new file mode 100644 index 00000000..e0856b2f --- /dev/null +++ b/tests/streaming/test_structured_trace_prototype.py @@ -0,0 +1,824 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import json +from dataclasses import replace +from pathlib import Path +from typing import cast + +import pandas as pd +import pytest + +from anonymizer.engine.constants import COL_FINAL_ENTITIES, COL_TEXT +from tests.streaming.structured_trace_prototype import ( + PROTECTED_TEXT_COLUMN, + SEGMENT_KEY_COLUMN, + CodecBounds, + FailureCode, + FieldRole, + ProjectedItem, + SourceFormat, + StructuredItemError, + TraceMapping, + build_synthetic_anonymizer, + project_complete_item, + protect_and_emit, + reconstruct_complete_item, + run_projected_segments, +) + +FIXTURE_DIR = Path(__file__).parents[1] / "fixtures" / "streaming" +JSON_FIXTURE = FIXTURE_DIR / "complete_trace.json" +JSONL_FIXTURE = FIXTURE_DIR / "complete_trace.jsonl" + +SENSITIVE_ENTITIES = { + "alice@example.test": "email", + "Alice Example": "full_name", + "+1-555-0100": "phone_number", +} + +TARGET_POINTERS = ( + "/messages/0/content", + "/messages/1/tool_calls/0/arguments/customer_email", + "/messages/1/tool_calls/0/arguments/case_note", + "/messages/2/tool_result/content", + "/messages/2/tool_result/account_email", +) + + +@pytest.fixture +def bounds() -> CodecBounds: + return CodecBounds(max_bytes=8_192, max_depth=12, max_targets=8) + + +@pytest.fixture +def trace_mapping() -> TraceMapping: + target = FieldRole.TARGET + preserve = FieldRole.PRESERVE + structural = FieldRole.STRUCTURAL + fields = { + "/schema_version": structural, + "/trace_id": structural, + "/metadata/environment": preserve, + "/metadata/retention_class": preserve, + "/metadata/sequence": structural, + "/messages/0/id": structural, + "/messages/0/parent_id": structural, + "/messages/0/role": structural, + "/messages/0/sequence": structural, + "/messages/0/content": target, + "/messages/1/id": structural, + "/messages/1/parent_id": structural, + "/messages/1/role": structural, + "/messages/1/sequence": structural, + "/messages/1/content": preserve, + "/messages/1/tool_calls/0/id": structural, + "/messages/1/tool_calls/0/type": structural, + "/messages/1/tool_calls/0/name": structural, + "/messages/1/tool_calls/0/arguments/customer_email": target, + "/messages/1/tool_calls/0/arguments/case_note": target, + "/messages/2/id": structural, + "/messages/2/parent_id": structural, + "/messages/2/role": structural, + "/messages/2/sequence": structural, + "/messages/2/tool_call_id": structural, + "/messages/2/name": structural, + "/messages/2/tool_result/status": structural, + "/messages/2/tool_result/content": target, + "/messages/2/tool_result/account_email": target, + } + return TraceMapping( + version="synthetic-trace/v1", + fields=fields, + source_identity_pointer="/trace_id", + ordered_identity_pointers=("/messages/0/id", "/messages/1/id", "/messages/2/id"), + ) + + +def test_complete_json_trace_round_trips_through_redact_after_result_reordering( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + source = JSON_FIXTURE.read_bytes() + original = json.loads(source) + + projected = project_complete_item( + source, + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + ) + + assert isinstance(projected, ProjectedItem) + assert tuple(segment.pointer for segment in projected.manifest.segments) == TARGET_POINTERS + assert tuple(projected.dataframe[SEGMENT_KEY_COLUMN]) == tuple( + f"synthetic-trace/v1:{pointer}#0" for pointer in TARGET_POINTERS + ) + assert all(segment.occurrence_index == 0 for segment in projected.manifest.segments) + assert projected.manifest.source_identity == "trace-2026-0001" + assert projected.manifest.source_order == ("msg-001", "msg-002", "msg-003") + assert projected.manifest.fidelity == "semantic-json-v1" + assert "alice@example.test" not in json.dumps(projected.manifest.template) + + anonymizer = build_synthetic_anonymizer(SENSITIVE_ENTITIES) + result = run_projected_segments(anonymizer, projected, source_ref=JSON_FIXTURE) + reordered = result.trace_dataframe.iloc[::-1].reset_index(drop=True) + protected = reconstruct_complete_item(projected, reordered, failed_records=result.failed_records) + reconstructed = json.loads(protected) + + assert reconstructed["trace_id"] == original["trace_id"] + assert reconstructed["schema_version"] == original["schema_version"] + assert reconstructed["metadata"] == original["metadata"] + assert [message["id"] for message in reconstructed["messages"]] == [ + message["id"] for message in original["messages"] + ] + assert [message["parent_id"] for message in reconstructed["messages"]] == [ + message["parent_id"] for message in original["messages"] + ] + assert [message["sequence"] for message in reconstructed["messages"]] == [0, 1, 2] + assert reconstructed["messages"][1]["content"] == "I will query the synthetic account." + assert reconstructed["messages"][1]["tool_calls"][0]["id"] == "call-001" + assert reconstructed["messages"][2]["tool_call_id"] == "call-001" + + for pointer in TARGET_POINTERS: + protected_value = _resolve_pointer(reconstructed, pointer) + assert isinstance(protected_value, str) + assert "[REDACTED_" in protected_value + assert all(value not in protected_value for value in SENSITIVE_ENTITIES) + assert set(reordered[SEGMENT_KEY_COLUMN]) == {segment.segment_key for segment in projected.manifest.segments} + assert len(reordered) == len(projected.manifest.segments) + assert not reordered[SEGMENT_KEY_COLUMN].duplicated().any() + + +def test_single_item_jsonl_variant_reconstructs_with_line_boundary( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + source = JSONL_FIXTURE.read_bytes() + projected = project_complete_item( + source, + source_format=SourceFormat.JSONL, + mapping=trace_mapping, + bounds=bounds, + ) + assert isinstance(projected, ProjectedItem) + + anonymizer = build_synthetic_anonymizer(SENSITIVE_ENTITIES) + result = run_projected_segments(anonymizer, projected, source_ref=JSONL_FIXTURE) + protected = reconstruct_complete_item(projected, result.trace_dataframe, failed_records=result.failed_records) + + assert protected.endswith(b"\n") + assert len(protected.splitlines()) == 1 + assert json.loads(protected)["trace_id"] == "trace-2026-0001" + + +def test_protect_and_emit_emits_complete_item_exactly_once( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + emitted: list[bytes] = [] + + protected = protect_and_emit( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=JSON_FIXTURE, + emit=emitted.append, + ) + + assert emitted == [protected] + assert all(value.encode() not in protected for value in SENSITIVE_ENTITIES) + + +def test_json_and_jsonl_have_semantically_equivalent_protected_output( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + outputs = [] + for source_format, fixture in ((SourceFormat.JSON, JSON_FIXTURE), (SourceFormat.JSONL, JSONL_FIXTURE)): + projected = project_complete_item( + fixture.read_bytes(), + source_format=source_format, + mapping=trace_mapping, + bounds=bounds, + ) + result = run_projected_segments(build_synthetic_anonymizer(SENSITIVE_ENTITIES), projected, source_ref=fixture) + outputs.append( + json.loads( + reconstruct_complete_item(projected, result.trace_dataframe, failed_records=result.failed_records) + ) + ) + + assert outputs[0] == outputs[1] + + +@pytest.mark.parametrize("outcome", ["missing", "duplicate", "unknown"]) +def test_segment_cardinality_failures_prevent_buffered_emission( + outcome: str, + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + emitted: list[bytes] = [] + expected_code = { + "missing": FailureCode.MISSING_SEGMENT, + "duplicate": FailureCode.DUPLICATE_SEGMENT, + "unknown": FailureCode.UNKNOWN_SEGMENT, + }[outcome] + transform = { + "missing": _drop_first_result, + "duplicate": _duplicate_first_result, + "unknown": _add_unknown_result, + }[outcome] + + with pytest.raises(StructuredItemError) as exc_info: + protect_and_emit( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=JSON_FIXTURE, + emit=emitted.append, + result_transform=transform, + ) + + assert exc_info.value.code is expected_code + assert emitted == [] + assert all(value not in str(exc_info.value) for value in SENSITIVE_ENTITIES) + + +def test_dropped_failed_row_prevents_emission_and_sanitizes_failure( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + projected = project_complete_item( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + ) + assert isinstance(projected, ProjectedItem) + failed_key = projected.manifest.segments[0].segment_key + emitted: list[bytes] = [] + + with pytest.raises(StructuredItemError) as exc_info: + protect_and_emit( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES, failed_segment_key=failed_key), + source_ref=JSON_FIXTURE, + emit=emitted.append, + ) + + assert exc_info.value.code is FailureCode.SEGMENT_PROCESSING_FAILED + assert emitted == [] + assert "engine-row-private-8675309" not in str(exc_info.value) + assert all(value not in str(exc_info.value) for value in SENSITIVE_ENTITIES) + + +def test_raw_target_passthrough_is_not_used_as_fallback_output( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + emitted: list[bytes] = [] + + with pytest.raises(StructuredItemError) as exc_info: + protect_and_emit( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=JSON_FIXTURE, + emit=emitted.append, + result_transform=_restore_raw_target, + ) + + assert exc_info.value.code is FailureCode.UNPROTECTED_TARGET + assert emitted == [] + assert all(value not in str(exc_info.value) for value in SENSITIVE_ENTITIES) + + +def test_changed_but_still_leaky_target_prevents_emission( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + emitted: list[bytes] = [] + + with pytest.raises(StructuredItemError) as exc_info: + protect_and_emit( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=JSON_FIXTURE, + emit=emitted.append, + result_transform=_restore_changed_leaky_target, + ) + + assert exc_info.value.code is FailureCode.UNPROTECTED_TARGET + assert emitted == [] + assert all(value not in str(exc_info.value) for value in SENSITIVE_ENTITIES) + + +def test_unchanged_target_is_allowed_when_detection_found_no_entity( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + parsed = json.loads(JSON_FIXTURE.read_bytes()) + safe_text = "No customer details are present." + parsed["messages"][0]["content"] = safe_text + source = json.dumps(parsed).encode() + emitted: list[bytes] = [] + + protected = protect_and_emit( + source, + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=JSON_FIXTURE, + emit=emitted.append, + ) + + assert emitted == [protected] + assert json.loads(protected)["messages"][0]["content"] == safe_text + + +def test_unknown_content_field_is_rejected_until_mapping_classifies_it( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + parsed = json.loads(JSON_FIXTURE.read_bytes()) + parsed["messages"][0]["debug_note"] = "synthetic diagnostic" + source = json.dumps(parsed).encode() + + with pytest.raises(StructuredItemError) as exc_info: + project_complete_item( + source, + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + ) + assert exc_info.value.code is FailureCode.UNKNOWN_FIELD + assert "synthetic diagnostic" not in str(exc_info.value) + + classified_mapping = replace( + trace_mapping, + fields={**trace_mapping.fields, "/messages/0/debug_note": FieldRole.PRESERVE}, + ) + projected = project_complete_item( + source, + source_format=SourceFormat.JSON, + mapping=classified_mapping, + bounds=bounds, + ) + assert isinstance(projected, ProjectedItem) + assert "/messages/0/debug_note" in projected.manifest.preserved_sha256 + + +def test_malformed_source_failure_does_not_expose_raw_input( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + raw_fragment = "alice@example.test" + malformed = f'{{"content":"{raw_fragment}"'.encode() + + with pytest.raises(StructuredItemError) as exc_info: + project_complete_item( + malformed, + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + ) + + assert exc_info.value.code is FailureCode.INVALID_SOURCE + assert raw_fragment not in str(exc_info.value) + assert exc_info.value.__cause__ is None + + +@pytest.mark.parametrize( + ("bounds_override", "expected_code"), + [ + ({"max_bytes": 32}, FailureCode.ITEM_TOO_LARGE), + ({"max_depth": 3}, FailureCode.STRUCTURE_TOO_DEEP), + ({"max_targets": 2}, FailureCode.TOO_MANY_TARGETS), + ], +) +def test_codec_enforces_explicit_byte_depth_and_target_bounds( + bounds_override: dict[str, int], + expected_code: FailureCode, + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + constrained = replace(bounds, **bounds_override) + + with pytest.raises(StructuredItemError) as exc_info: + project_complete_item( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=constrained, + ) + + assert exc_info.value.code is expected_code + assert all(value not in str(exc_info.value) for value in SENSITIVE_ENTITIES) + + +def test_deep_json_parser_failure_is_bounded_and_sanitized(bounds: CodecBounds) -> None: + source = ('{"id":"source","nested":' + "[" * 1_100 + "0" + "]" * 1_100 + "}").encode() + nested_pointer = "/nested" + "/0" * 1_100 + mapping = TraceMapping( + "v1", + {"/id": FieldRole.STRUCTURAL, nested_pointer: FieldRole.PRESERVE}, + "/id", + (), + ) + + with pytest.raises(StructuredItemError) as exc_info: + project_complete_item( + source, + source_format=SourceFormat.JSON, + mapping=mapping, + bounds=bounds, + ) + + assert exc_info.value.code is FailureCode.STRUCTURE_TOO_DEEP + assert exc_info.value.__cause__ is None + + +def test_oversized_numeric_scalar_failure_is_bounded_and_sanitized(bounds: CodecBounds) -> None: + source = ("{" + '"number":' + "9" * 5_000 + "}").encode() + mapping = TraceMapping("v1", {"/number": FieldRole.STRUCTURAL}, "/number", ()) + + with pytest.raises(StructuredItemError) as exc_info: + project_complete_item(source, source_format=SourceFormat.JSON, mapping=mapping, bounds=bounds) + + assert exc_info.value.code is FailureCode.ITEM_TOO_LARGE + assert exc_info.value.__cause__ is None + + +def test_scalar_count_bound_fails_closed() -> None: + source = b'{"id":"one","extra":"two"}' + mapping = TraceMapping( + "v1", + {"/id": FieldRole.STRUCTURAL, "/extra": FieldRole.PRESERVE}, + "/id", + (), + ) + constrained = CodecBounds(128, 4, 1, 1, 32) + + with pytest.raises(StructuredItemError) as exc_info: + project_complete_item(source, source_format=SourceFormat.JSON, mapping=mapping, bounds=constrained) + + assert exc_info.value.code is FailureCode.ITEM_TOO_LARGE + + +def test_parser_event_count_bound_fails_closed() -> None: + source = b'{"id":"source","items":[0,1]}' + mapping = TraceMapping( + "v1", + { + "/id": FieldRole.STRUCTURAL, + "/items/0": FieldRole.PRESERVE, + "/items/1": FieldRole.PRESERVE, + }, + "/id", + (), + ) + constrained = CodecBounds(128, 4, 1, 4, 32, 4) + + with pytest.raises(StructuredItemError) as exc_info: + project_complete_item(source, source_format=SourceFormat.JSON, mapping=mapping, bounds=constrained) + + assert exc_info.value.code is FailureCode.ITEM_TOO_LARGE + + +@pytest.mark.parametrize( + "source", + [ + b'{"value":NaN}', + b'{"value":Infinity}', + b'{"value":-Infinity}', + b'{"value":1e999}', + b'{"value":1,"value":2}', + ], +) +def test_strict_json_rejects_nonfinite_numbers_and_duplicate_keys( + source: bytes, + bounds: CodecBounds, +) -> None: + mapping = TraceMapping("v1", {"/value": FieldRole.STRUCTURAL}, "/value", ()) + + with pytest.raises(StructuredItemError) as exc_info: + project_complete_item(source, source_format=SourceFormat.JSON, mapping=mapping, bounds=bounds) + + assert exc_info.value.code is FailureCode.INVALID_SOURCE + + +@pytest.mark.parametrize( + "mapping", + [ + TraceMapping("v1", {"/id": FieldRole.PRESERVE}, "/id", ()), + TraceMapping( + "v1", + {"/id": FieldRole.STRUCTURAL, "/ordered": FieldRole.PRESERVE}, + "/id", + ("/ordered",), + ), + ], +) +def test_identity_pointers_must_be_declared_structural( + mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + source = b'{"id":"source","ordered":"first"}' + + with pytest.raises(StructuredItemError) as exc_info: + project_complete_item(source, source_format=SourceFormat.JSON, mapping=mapping, bounds=bounds) + + assert exc_info.value.code is FailureCode.MAPPING_MISMATCH + + +@pytest.mark.parametrize("pointer", ["id", "/items/~2", "/items/-1", "/items/01", "/items/+1"]) +def test_invalid_json_pointer_forms_are_rejected(pointer: str, bounds: CodecBounds) -> None: + mapping = TraceMapping( + "v1", + {"/id": FieldRole.STRUCTURAL, pointer: FieldRole.PRESERVE}, + "/id", + (), + ) + + with pytest.raises(StructuredItemError) as exc_info: + project_complete_item( + b'{"id":"source","items":["zero","one"]}', + source_format=SourceFormat.JSON, + mapping=mapping, + bounds=bounds, + ) + + assert exc_info.value.code is FailureCode.MAPPING_MISMATCH + + +def test_invalid_bounds_and_source_format_fail_closed( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + with pytest.raises(StructuredItemError) as bounds_error: + project_complete_item( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=replace(bounds, max_depth=-1), + ) + assert bounds_error.value.code is FailureCode.MAPPING_MISMATCH + + with pytest.raises(StructuredItemError) as format_error: + project_complete_item( + JSON_FIXTURE.read_bytes(), + source_format=cast(SourceFormat, "yaml"), + mapping=trace_mapping, + bounds=bounds, + ) + assert format_error.value.code is FailureCode.INVALID_SOURCE + + projected = project_complete_item( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + ) + invalid_manifest = replace(projected.manifest, source_format=cast(SourceFormat, "yaml")) + with pytest.raises(StructuredItemError) as manifest_error: + reconstruct_complete_item( + replace(projected, manifest=invalid_manifest), + pd.DataFrame(), + ) + assert manifest_error.value.code is FailureCode.MAPPING_MISMATCH + + +def test_reconstruction_revalidates_source_identity_and_order( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + projected = project_complete_item( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + ) + result = run_projected_segments(build_synthetic_anonymizer(SENSITIVE_ENTITIES), projected, source_ref=JSON_FIXTURE) + mismatched_manifests = ( + replace(projected.manifest, source_identity="other-source"), + replace(projected.manifest, source_order=("msg-003", "msg-002", "msg-001")), + ) + + for mismatched_manifest in mismatched_manifests: + with pytest.raises(StructuredItemError) as exc_info: + reconstruct_complete_item( + replace(projected, manifest=mismatched_manifest), + result.trace_dataframe, + failed_records=result.failed_records, + ) + assert exc_info.value.code is FailureCode.MAPPING_MISMATCH + + +def test_manifest_state_cannot_be_mutated_after_projection( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + projected = project_complete_item( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + ) + template = cast(dict[str, object], projected.manifest.template) + cast(dict[str, object], template["metadata"])["environment"] = "tampered" + detached_dataframe = projected.dataframe + detached_dataframe.loc[detached_dataframe.index[0], COL_TEXT] = "tampered projection" + + result = run_projected_segments(build_synthetic_anonymizer(SENSITIVE_ENTITIES), projected, source_ref=JSON_FIXTURE) + protected = reconstruct_complete_item(projected, result.trace_dataframe, failed_records=result.failed_records) + + assert json.loads(protected)["metadata"]["environment"] == "test" + assert "tampered projection" not in projected.dataframe[COL_TEXT].tolist() + with pytest.raises(TypeError): + cast(dict[str, str], projected.manifest.preserved_sha256)["/new"] = "digest" + + +@pytest.mark.parametrize("column", [SEGMENT_KEY_COLUMN, PROTECTED_TEXT_COLUMN]) +def test_duplicate_required_dataframe_column_is_sanitized( + column: str, + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + projected = project_complete_item( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + ) + result = run_projected_segments(build_synthetic_anonymizer(SENSITIVE_ENTITIES), projected, source_ref=JSON_FIXTURE) + malformed = pd.concat([result.trace_dataframe, result.trace_dataframe[[column]]], axis=1) + + with pytest.raises(StructuredItemError) as exc_info: + reconstruct_complete_item(projected, malformed, failed_records=result.failed_records) + + assert exc_info.value.code is FailureCode.MAPPING_MISMATCH + assert exc_info.value.__cause__ is None + + +@pytest.mark.parametrize("column", [SEGMENT_KEY_COLUMN, PROTECTED_TEXT_COLUMN]) +def test_missing_required_dataframe_column_is_sanitized( + column: str, + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + projected = project_complete_item( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + ) + result = run_projected_segments(build_synthetic_anonymizer(SENSITIVE_ENTITIES), projected, source_ref=JSON_FIXTURE) + + with pytest.raises(StructuredItemError) as exc_info: + reconstruct_complete_item( + projected, + result.trace_dataframe.drop(columns=column), + failed_records=result.failed_records, + ) + + assert exc_info.value.code is FailureCode.MAPPING_MISMATCH + assert exc_info.value.__cause__ is None + + +def test_result_transform_cannot_erase_grounded_detection_inventory( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + emitted: list[bytes] = [] + + with pytest.raises(StructuredItemError) as exc_info: + protect_and_emit( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=JSON_FIXTURE, + emit=emitted.append, + result_transform=_erase_inventory_and_restore_leaky_target, + ) + + assert exc_info.value.code is FailureCode.MAPPING_MISMATCH + assert emitted == [] + + +def test_result_transform_failure_is_sanitized_and_does_not_emit( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + emitted: list[bytes] = [] + + with pytest.raises(StructuredItemError) as exc_info: + protect_and_emit( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + anonymizer=build_synthetic_anonymizer(SENSITIVE_ENTITIES), + source_ref=JSON_FIXTURE, + emit=emitted.append, + result_transform=_raise_private_transform_error, + ) + + assert exc_info.value.code is FailureCode.SEGMENT_PROCESSING_FAILED + assert exc_info.value.__cause__ is None + assert emitted == [] + assert "engine-row-private-8675309" not in str(exc_info.value) + assert all(value not in str(exc_info.value) for value in SENSITIVE_ENTITIES) + + +def test_generic_caller_columns_remain_outside_target_protection( + trace_mapping: TraceMapping, + bounds: CodecBounds, +) -> None: + projected = project_complete_item( + JSON_FIXTURE.read_bytes(), + source_format=SourceFormat.JSON, + mapping=trace_mapping, + bounds=bounds, + ) + assert isinstance(projected, ProjectedItem) + caller_dataframe = projected.dataframe + caller_dataframe["caller_owned_raw_copy"] = caller_dataframe[COL_TEXT] + projected = replace(projected, _dataframe=caller_dataframe) + + result = run_projected_segments( + build_synthetic_anonymizer(SENSITIVE_ENTITIES), + projected, + source_ref=JSON_FIXTURE, + ) + + assert result.trace_dataframe["caller_owned_raw_copy"].tolist() == projected.dataframe[COL_TEXT].tolist() + assert "alice@example.test" in " ".join(result.trace_dataframe["caller_owned_raw_copy"]) + assert "alice@example.test" not in " ".join(result.trace_dataframe[PROTECTED_TEXT_COLUMN]) + + +def _drop_first_result(dataframe: pd.DataFrame) -> pd.DataFrame: + return dataframe.iloc[1:].copy() + + +def _duplicate_first_result(dataframe: pd.DataFrame) -> pd.DataFrame: + return pd.concat([dataframe, dataframe.iloc[[0]]], ignore_index=True) + + +def _add_unknown_result(dataframe: pd.DataFrame) -> pd.DataFrame: + unknown = dataframe.iloc[[0]].copy() + unknown[SEGMENT_KEY_COLUMN] = "synthetic-trace/v1:/unknown#0" + return pd.concat([dataframe, unknown], ignore_index=True) + + +def _restore_raw_target(dataframe: pd.DataFrame) -> pd.DataFrame: + first_index = dataframe.index[0] + dataframe.loc[first_index, PROTECTED_TEXT_COLUMN] = dataframe.loc[first_index, "segment_text"] + return dataframe + + +def _restore_changed_leaky_target(dataframe: pd.DataFrame) -> pd.DataFrame: + first_index = dataframe.index[0] + dataframe.loc[first_index, PROTECTED_TEXT_COLUMN] = "changed alice@example.test" + return dataframe + + +def _raise_private_transform_error(dataframe: pd.DataFrame) -> pd.DataFrame: + raw_text = dataframe.loc[dataframe.index[0], "segment_text"] + raise RuntimeError(f"engine-row-private-8675309 failed for {raw_text}") + + +def _erase_inventory_and_restore_leaky_target(dataframe: pd.DataFrame) -> pd.DataFrame: + first_index = dataframe.index[0] + dataframe.at[first_index, PROTECTED_TEXT_COLUMN] = "changed alice@example.test" + dataframe.at[first_index, COL_FINAL_ENTITIES] = {"entities": []} + return dataframe + + +def _resolve_pointer(document: object, pointer: str) -> object: + current = document + for token in pointer.removeprefix("/").split("/"): + token = token.replace("~1", "/").replace("~0", "~") + if isinstance(current, list): + current = cast(list[object], current)[int(token)] + else: + assert isinstance(current, dict) + current = cast(dict[str, object], current)[token] + return current diff --git a/tests/test_debug_logging.py b/tests/test_debug_logging.py index a7f78767..8c021eb6 100644 --- a/tests/test_debug_logging.py +++ b/tests/test_debug_logging.py @@ -49,6 +49,7 @@ def _stub_anonymizer() -> Anonymizer: { COL_TEXT: ["Alice works at Acme", "Bob likes cats"], COL_REPLACED_TEXT: ["[REDACTED] works at [REDACTED]", "[REDACTED] likes cats"], + COL_FINAL_ENTITIES: entities, } ) replace_runner = Mock(spec=ReplacementWorkflow) @@ -76,11 +77,17 @@ def debug_messages( @pytest.mark.parametrize( "expected_substring", [ - "r1", - "r2", "input text lengths:", "detection config: threshold=", ], ) def test_debug_log_contains(debug_messages: list[str], expected_substring: str) -> None: assert any(expected_substring in m for m in debug_messages) + + +def test_debug_logs_exclude_failed_record_content(debug_messages: list[str]) -> None: + rendered = "\n".join(debug_messages) + assert "r1" not in rendered + assert "r2" not in rendered + assert "timeout" not in rendered + assert "parse error" not in rendered diff --git a/tools/intake_dogfood_runbook.md b/tools/intake_dogfood_runbook.md new file mode 100644 index 00000000..3ffd6771 --- /dev/null +++ b/tools/intake_dogfood_runbook.md @@ -0,0 +1,291 @@ + + + +# Local Intake dogfood runbook + +Use this runbook to start an isolated NeMo Platform Intake service with managed +ClickHouse, run Anonymizer's protected-only dogfood, and inspect the resulting +spans. This is a local development procedure, not a production deployment or a +production Intake integration. + +The current Anonymizer design uses a provisional **before-Intake** boundary. +Only protected payloads may enter the Intake service process. Do not enable the +raw characterization cases unless the deployment is isolated and the operator +has explicitly approved receiving raw synthetic fixtures. + +## Prerequisites + +You need: + +- a NeMo Platform source checkout with its Python environment bootstrapped; +- Docker with a reachable daemon; +- this Anonymizer worktree and its development environment; and +- an unused loopback port, normally `8080`. + +The procedure was validated with: + +- NeMo Platform revision `e1057736703bb8b167a4bd9013cea0caae2df63a`; +- ClickHouse 26.3; +- NeMo Platform at `/root/dev/nemo-platform`; and +- Anonymizer at `/root/dev/wt/Anonymizer-openshell-intake`. + +Other revisions may change CLI options, endpoint behavior, or storage layout. +Use NeMo Platform's `SETUP.md` and `services/intake/README.md` as the authority +when updating this procedure. + +If the NeMo Platform checkout is not bootstrapped, follow its canonical setup +guide. For a Python-only source environment, the relevant bootstrap target is: + +```bash +cd /root/dev/nemo-platform +make bootstrap-python +``` + +## 1. Check for an existing service + +Do not start a second platform blindly on the same port or data directory. +Check the ready endpoint and running commands first: + +```bash +curl --fail --silent --show-error \ + http://127.0.0.1:8080/health/ready || true + +pgrep -af '[n]emo services run' || true +``` + +If the ready endpoint returns `{"status":"ready"}`, either reuse that approved +deployment or choose another port and a separate data directory. Do not stop an +existing service or remove its container or data without operator approval. + +## 2. Create isolated local state + +Create a unique root and print it so another terminal or later session can +reuse the same deployment: + +```bash +INTAKE_DOGFOOD_ROOT="$(mktemp -d /tmp/nemo-intake-dogfood.XXXXXX)" +mkdir -p \ + "$INTAKE_DOGFOOD_ROOT/data" \ + "$INTAKE_DOGFOOD_ROOT/state" \ + "$INTAKE_DOGFOOD_ROOT/cache/uv" +printf 'Intake dogfood root: %s\n' "$INTAKE_DOGFOOD_ROOT" +``` + +The paths have distinct ownership: + +```text +$INTAKE_DOGFOOD_ROOT/ +├── data/ NeMo Platform SQLite state and managed ClickHouse data +├── state/nmp/ local-service state and runtime metadata +└── cache/uv/ uv cache for this isolated run +``` + +Retain the printed root for restarts. A new root creates a different managed +ClickHouse identity and an empty Intake deployment. + +## 3. Start Intake and managed ClickHouse + +Run the platform in the foreground from the NeMo Platform repository: + +```bash +cd /root/dev/nemo-platform + +env \ + NMP_DATA_DIR="$INTAKE_DOGFOOD_ROOT/data" \ + XDG_STATE_HOME="$INTAKE_DOGFOOD_ROOT/state" \ + UV_CACHE_DIR="$INTAKE_DOGFOOD_ROOT/cache/uv" \ + NMP_INTAKE_CLICKHOUSE_IMAGE=clickhouse/clickhouse-server:26.3 \ + uv run nemo services run \ + --services auth,entities,intake \ + --host 127.0.0.1 \ + --port 8080 +``` + +When `NMP_INTAKE_CLICKHOUSE_URL` is unset, Intake provisions a managed +ClickHouse container. It binds the ClickHouse HTTP port to a Docker-assigned +loopback port and stores data under +`$INTAKE_DOGFOOD_ROOT/data/intake-clickhouse/`. Run only one local platform +process against a given data directory. + +Keep this terminal open. Use another terminal for the remaining commands. + +## 4. Verify both service and storage + +First check aggregate readiness: + +```bash +curl --fail --silent --show-error \ + http://127.0.0.1:8080/health/ready +``` + +Expected response: + +```json +{"status":"ready"} +``` + +Then exercise Intake's ClickHouse-backed read path: + +```bash +curl --fail-with-body --silent --show-error --get \ + 'http://127.0.0.1:8080/apis/intake/v2/workspaces/default/spans' \ + --data-urlencode 'page=1' \ + --data-urlencode 'page_size=1' \ + | jq . +``` + +Continue only after this request returns HTTP 200. An empty `data` list is +healthy. HTTP 503 means Intake cannot reach ClickHouse even if the aggregate +ready endpoint succeeds. + +Inspect managed ClickHouse without changing it: + +```bash +docker ps --all \ + --filter 'label=nmp.nvidia.com/component=intake-clickhouse' \ + --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}' +``` + +If several managed containers exist, use their bind mounts and the printed +dogfood root to identify the current one: + +```bash +docker inspect \ + --format '{{.Name}} {{range .Mounts}}{{.Source}} -> {{.Destination}}{{end}}' \ + CONTAINER_NAME +``` + +## 5. Run protected-only Anonymizer dogfood + +Run the checked-in integration test from the Anonymizer worktree: + +```bash +cd /root/dev/wt/Anonymizer-openshell-intake + +env \ + PYTHONPATH=. \ + ANONYMIZER_INTAKE_DOGFOOD_BASE_URL=http://127.0.0.1:8080 \ + uv run --frozen pytest tests/streaming/test_intake_dogfood.py -q +``` + +This default profile sends only protected payloads and leaves its synthetic +rows under the deployment's retention policy. It does not provision, stop, or +clean up Intake or ClickHouse. + +To include a completed Sandbox Codex session created from +[`sandbox_agent_prompt.md`](../tests/fixtures/streaming/sandbox_agent_prompt.md), +add its run directory: + +```bash +env \ + PYTHONPATH=. \ + ANONYMIZER_INTAKE_DOGFOOD_BASE_URL=http://127.0.0.1:8080 \ + ANONYMIZER_SANDBOX_DOGFOOD_RUN_DIR=/stable-cache/sandbox/runs/RUN_NAME \ + uv run --frozen pytest tests/streaming/test_intake_dogfood.py -q +``` + +The Sandbox run must be completed successfully and must use the declared +synthetic values. The test-only adapter rejects unsupported completed event +types and never launches or manages Sandbox. + +Do not set `ANONYMIZER_INTAKE_DOGFOOD_ALLOW_RAW=1` during normal dogfood. That +flag intentionally sends raw synthetic fixtures and exists only to characterize +Intake on a separately approved isolated deployment. + +## 6. Query a stored trace + +List the newest distinct session IDs from the isolated deployment: + +```bash +curl --fail --silent --show-error --get \ + 'http://127.0.0.1:8080/apis/intake/v2/workspaces/default/spans' \ + --data-urlencode 'page=1' \ + --data-urlencode 'page_size=100' \ + | jq -r ' + .data + | sort_by(.ingested_at) + | reverse + | .[] + | [.ingested_at, .session_id] + | @tsv + ' \ + | awk '!seen[$2]++' +``` + +Replace `SESSION_ID` below with the relevant value: + +```bash +curl --fail --silent --show-error --get \ + 'http://127.0.0.1:8080/apis/intake/v2/workspaces/default/spans' \ + --data-urlencode 'filter[session_id]=SESSION_ID' \ + --data-urlencode 'page=1' \ + --data-urlencode 'page_size=100' \ + | jq . +``` + +For a compact topology and content view: + +```bash +curl --fail --silent --show-error --get \ + 'http://127.0.0.1:8080/apis/intake/v2/workspaces/default/spans' \ + --data-urlencode 'filter[session_id]=SESSION_ID' \ + --data-urlencode 'page=1' \ + --data-urlencode 'page_size=100' \ + | jq '.data[] | { + span_id, + parent_span_id, + kind, + name, + tool_name, + input, + output, + raw_attributes, + ingested_at + }' +``` + +In Intake's read model, `raw_attributes` means retained source attributes. For +the protected dogfood it contains the protected ATIF representation, not the +unprotected Sandbox trace. + +## 7. Stop without deleting data + +Return to the foreground service terminal and press `Ctrl-C`. A graceful stop +stops the managed ClickHouse container but does not remove the container or its +bind-mounted data. Restart with the same dogfood root and the command from step +3 to reuse the deployment. + +After stopping, verify the platform process is gone and record the container +state: + +```bash +pgrep -af '[n]emo services run' || true + +docker ps --all \ + --filter 'label=nmp.nvidia.com/component=intake-clickhouse' \ + --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}' +``` + +Removing the managed container or deleting `$INTAKE_DOGFOOD_ROOT` is a separate, +destructive operation. It requires explicit operator approval and must follow +NeMo Platform's managed teardown procedure. Do not use an unscoped `docker rm` +or recursive deletion command from this runbook. + +## Validated local instance + +The 2026-08-19 dogfood used this instance. These identifiers are evidence, not +reusable configuration: + +```text +Intake: http://127.0.0.1:8080 +Dogfood root: /tmp/nemo-intake-dogfood.AgO959 +ClickHouse: nmp-intake-clickhouse-b02569d8fb94 +Image: clickhouse/clickhouse-server:26.3 +ClickHouse HTTP 127.0.0.1:32768 -> 8123/tcp +``` + +The deployment stored protected ATIF, chat-completion, and OTLP/protobuf +fixtures, plus a protected ATIF trajectory derived from a real Sandbox Codex +session. See the +[Intake workload validation evidence](../docs/development/intake-workload-validation-evidence.md) +for the validated behavior and remaining contract limits. diff --git a/uv.lock b/uv.lock index 1cdea812..92fde0a7 100644 --- a/uv.lock +++ b/uv.lock @@ -1,5 +1,5 @@ version = 1 -revision = 2 +revision = 3 requires-python = ">=3.11" resolution-markers = [ "python_full_version >= '3.14'", @@ -2469,6 +2469,7 @@ dependencies = [ [package.dev-dependencies] dev = [ + { name = "opentelemetry-proto" }, { name = "pre-commit" }, { name = "pytest" }, { name = "pytest-cov" }, @@ -2511,6 +2512,7 @@ requires-dist = [ [package.metadata.requires-dev] dev = [ + { name = "opentelemetry-proto", specifier = ">=1.27.0,<2" }, { name = "pre-commit", specifier = ">=4.0.0,<5" }, { name = "pytest", specifier = ">=9.0.3,<10" }, { name = "pytest-cov", specifier = ">=7.0,<8" }, @@ -2697,6 +2699,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fe/dd/d95db7ac5bec037b3dcc8ea6fe64cd5dc4bc87017d2e06e7410318e85dc0/opentelemetry_exporter_prometheus-0.64b0-py3-none-any.whl", hash = "sha256:9979a15f8d007d442bc7a6e16f4cbde5e8e0c5e99689887ceb5f33da251f0655", size = 13031, upload-time = "2026-06-24T15:19:44.159Z" }, ] +[[package]] +name = "opentelemetry-proto" +version = "1.44.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "protobuf" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/64/01/40ac4ae9a149263cc52c2cee200ddd80cb6d8db1a4610abf8eabce0fe771/opentelemetry_proto-1.44.0.tar.gz", hash = "sha256:c547a79c2f8c0c515d31509154682e5921c7cfd5ca67b70e1f9266e2c3e103f3", size = 46488, upload-time = "2026-07-16T15:25:45.34Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/7c/8be563d68e93bbefa5c8affb82ddcff91b3ad858ce49957ba7b16fd3e0ab/opentelemetry_proto-1.44.0-py3-none-any.whl", hash = "sha256:898b155a0e1557afd867478fb6158e8122a46329ca0bb8dc53cc55e98f017f56", size = 72483, upload-time = "2026-07-16T15:25:28.429Z" }, +] + [[package]] name = "opentelemetry-sdk" version = "1.43.0"