diff --git a/.agent-loop/LOOP_STATE.md b/.agent-loop/LOOP_STATE.md index 374392941..cdbc99b04 100644 --- a/.agent-loop/LOOP_STATE.md +++ b/.agent-loop/LOOP_STATE.md @@ -5,45 +5,53 @@ - This authored file is reviewed planning/history context, not canonical live post-merge state. Canonical state is the signed schema-v2 output on `automation/loop-memory`. -- Active initiatives include parallel `WS-AUTH-001` and `WS-ART-001`. -- PR #132 merged `WS-AUTH-001-09A` into `main` as `299363a` on - 2026-07-17; signed schema-v2 memory recorded the stopped checkpoint. -- Active implementation chunk: `WS-AUTH-001-09B` on - `codex/ws-auth-001-09b-controlled-service-provisioning` after explicit user - start. -- AUTH-09B is bounded to one controlled service-actor provisioning route and - activates only `actor.service.provision`. AUTH-09C remains inactive. -- Start basis: trusted `main` at `299363a` after PR #132, integrated through - `a947b86` after PR #142 before publication. +- Active initiatives include independently owned `WS-AUTH-001`, `WS-ART-001`, + `WS-REV-001`, and `WS-CON-001`. The planning-only `WS-XINT-001` boundary + reconciliation merged through PR #139 as `5d353b6` and starts no runtime. +- AUTH's owner reconciliation merged through PR #140 as `d541521`; it defines + fixed-service admission, prepared mutation authority, and activation custody + without activating ART, REV, or CON feature behavior. +- AUTH-09A merged through PR #132 as `299363a`; its fixed service identity + foundation is preserved as migration `0023_service_actor_identity`. +- AUTH-09B merged through PR #143 as `053242b`; it adds controlled service + actor provisioning and leaves service runtime admission plus feature actions + inactive. +- Active ART implementation chunk: `WS-ART-001-02A3` on + `codex/ws-art-001-02a3-artifact-store-v2-local-clean-cut`. +- Parallel AUTH, REV, and CON worktrees remain independently owned. This ART + branch consumes their merged handoff contracts without editing or activating + their runtime behavior. +- Start basis: PR #129 merged ART-02A2 into `main` as `9a04434`, and PR #139 + then merged the cross-initiative boundary reconciliation as `5d353b6`. - PR #119 merged `WS-AUTH-001-05B` as `ad71c7e`. - PR #120 merged `WS-ART-001-OBJECT-STORAGE-AMENDMENT` as `4408256`. - PR #122 merged the first automated post-merge memory implementation as `fc89fb6`; its schema-v1 cross-initiative next pointer is superseded by the schema-v2 initiative-local clean cut. -- Current gate: PR #143's replacement Backend run passed all 1,242 tests but - exposed 89.75 percent authorization coverage at the new 90 percent gate. The - behavior-test repair passes every required reviewer track at exact SHA - `127615f` and projects 90.31 percent on the unchanged 1,600-statement - denominator. Replacement GitHub checks and evidence rebinding remain; no - service caller becomes executable before AUTH-09E. +- Current gate: publish PR #141 for GitHub Actions, CodeRabbit, and explicit + human review after reconciling merged AUTH-09B and rebinding exact-SHA ART + evidence. No later ART chunk starts automatically. - Scope checkpoint: AWS S3 is the only v0.1 production provider; MinIO is local/CI S3 protocol proof; LocalStorage is focused development/test; R2 and Flow Node are deferred. Product modules receive narrow artifact capabilities, and AWS cannot instantiate in production without release-bound live proof. - Authorization checkpoint: merged main contains 74 PermissionIds and 65 ActionIds, with the two actor-self and seven AUTH-08 administrative actions - active. AUTH-09A defines seven fixed artifact service identities and eleven - exact planned static matrix memberships. The - plan now requires availability-neutral ART/REV custody transfer, fixed-service - admission, prepared mutation authority, feature-owned hidden behavior, and - exact AUTH-only activation chunks. This planning amendment activates nothing. -- Parallel artifact checkpoint: `WS-ART-001-02A1` was explicitly started and - merged through PR #127 as `f64a8e5`; it is at the post-merge memory/stop - checkpoint. ART-02A2 merged through PR #129, and ART-02A3 is reviewed in its - isolated parallel worktree but has no open PR at this checkpoint. + active, plus the AUTH-09B `actor.service.provision` action. Merged AUTH-09A + defines seven fixed artifact + service identities and eleven exact planned static matrix memberships. ART + feature chunks supply hidden canonical behavior/resource composition. Merged + AUTH planning requires availability-neutral ART custody transfer, fixed-service + admission, prepared mutation authority, and exact AUTH-only activation chunks; + neither reconciliation PR activates feature behavior. +- Parallel artifact checkpoint: ART-02A1 merged through PR #127 as `f64a8e5` + and ART-02A2 merged through PR #129 as `9a04434`. ART-02A3 completed + merged-main deterministic repair, deterministic proof, and exact-SHA internal + review. PR #141 is open; refreshed evidence, external checks, and explicit + human merge approval remain. ART-02B1 remains inactive. - Authorization checkpoint: AUTH-07B, AUTH-08, and AUTH-09A merged through PRs - #130, #131, and #132. Signed memory stopped after 09A, and the user explicitly - started 09B. + #130, #131, and #132; AUTH-09B merged through PR #143 as `053242b`. + AUTH-09C remains inactive until a separate explicit user start. - Parallel coverage work: `WS-QUAL-001-01B2` remains paused. Its last official whole-app result is `6466/8159` statements (`79.249908%`); no replacement evidence exists. diff --git a/.agent-loop/REVIEW_LOG.md b/.agent-loop/REVIEW_LOG.md index 0990684e4..7c7971623 100644 --- a/.agent-loop/REVIEW_LOG.md +++ b/.agent-loop/REVIEW_LOG.md @@ -2136,3 +2136,173 @@ creation while preserving the stabilized submission artifact digest as lineage. Current gate: commit the planning baseline, run required internal plan review, repair valid findings, publish one planning PR, and stop. No runtime chunk starts from this record. + +## 2026-07-16 - WS-ART-001-02A3 Pre-Main Review Passed (Superseded) + +Reviewed code SHA: `935b1a2bb4663828ecde173b3f91c682250a1aed`. + +ArtifactStore v1 was replaced by the byte-only v2 contract, LocalStorage and +the empty pre-production schema were cleanly cut over, the deployment namespace +fence and scratch cleanup were activated, and no product ingest, verification, +recovery, S3, or AUTH action was activated. + +All nine required tracks passed for that pre-main revision after repair. +Deterministic proof includes +real PostgreSQL cancellation state, concurrent-writer migration refusal, 90 +percent changed-subsystem coverage, stale-contract/wording checks, and the +engineering evidence gate. + +Evidence: `.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-internal-review-evidence.md` + +Trust bundle: `.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-pr-trust-bundle.md` + +This review was invalidated for publication when merged-main reconciliation +changed the candidate. The current gate is final exact-SHA review and fresh +evidence for the merged-main ART-02A3 revision. Do not publish, merge, or start +`WS-ART-001-02B1` from this historical record. + +## 2026-07-17 - WS-ART-001-02A3 Merged-Main Review Passed Before Gate Repair + +Reviewed code SHA: `441d39230a341f2c43dd548776a2437ae6b2395d`. + +All nine required tracks passed with no remaining findings after atomic +provider-replica finalization and LocalStorage cleanup ownership were repaired. +Every reviewer session was closed. GitHub Agent Gates then found a process-only +successor-heading mismatch and a stale durable-state assertion. This record is +historical; the process-only repair must receive fresh exact-SHA review and +evidence before publication resumes. + +## 2026-07-17 - WS-ART-001-02A3 AUTH-Reconciled Review Repair + +The first internal review after merging AUTH PR #140 found four valid issues: +provider initialization preceded the startup namespace fence, the LocalStorage +fingerprint did not distinguish same-path root replacement, the Operator read +port used vague generic methods, and same-item `replay_required` recovery lacked +an end-to-end accounting/receipt regression. + +The repair now claims and validates the namespace before adapter construction, +binds LocalStorage to a pre-provisioned private root's hashed filesystem +identity, exposes exact Operator read method names, and proves one same-item +replay produces one content, replica, receipt, and accounting transition. +Deterministic proof passed 81 ART/PostgreSQL/conformance tests plus 177 +preparation/config/application tests with 93.35 percent combined changed-scope +coverage. The first review results are superseded; fresh exact-SHA review and +evidence remain mandatory before PR #141 is republished. + +## 2026-07-17 - WS-ART-001-02A3 Startup Claim Repair + +The first exact-SHA review of `ddc9dad` found a residual LocalStorage startup +race and a typed-factory sequencing mismatch. Namespace validation observed a +root before adapter construction, so a same-path replacement could still +receive layout mutation. Real `s3_compatible` startup also failed in local-only +namespace preflight instead of through the canonical typed unavailable-provider +error. Product/ops additionally found that the planned Operator port retained +free-form resource-type strings and one stale generic method summary. + +The repair keeps `ArtifactStore` byte-only and introduces a separate +composition-only bootstrap. The typed factory first opens and holds the existing +private root without layout mutation, PostgreSQL admits that exact namespace +identity, and initialization rechecks the configured path before writing only +through the held descriptor. `s3_compatible` now fails at the single typed +factory boundary before namespace work. Operator binding and audit lookups use +closed resource vocabularies and exact method names. All prior reviewer sessions +were closed; this candidate requires fresh deterministic coverage and all nine +exact-SHA reviewer tracks before evidence can be refreshed. + +The first review of the startup-claim candidate then found two Low issues. A +future provider descriptor could collide with canonical `backend`, `adapter`, +or `provider_profile` keys, and the glossary retained two v1 provider-reference +phrases. The interface now rejects every reserved descriptor key, regression +coverage pins that rule, and the glossary uses the exact opaque +`provider_object_ref` term. Those reviewer sessions were closed; no prior pass +or pass-with-risk result is reusable for the repaired revision. + +The next review round found one remaining Low documentation mismatch shared by +five tracks: `ArtifactOperationReceipt` still claimed a response digest and +provider/database timestamps that do not exist in the v2 model. The canonical +glossary and artifact specification now enumerate the exact Workstream-owned +put-acknowledgement fields, resolve adapter/namespace identity through the +linked replica, and explicitly forbid a response digest or provider receipt. +All reviewer sessions were closed, and this documentation repair requires one +final exact-SHA review round. + +That review found one High and one Medium implementation issue plus one Low +documentation mismatch. LocalStorage enforced private mode but not runtime-user +ownership, and `ArtifactStoreNamespaceIdentity` accepted a mutable list despite +being frozen. Local directory, marker, and object descriptor checks now require +the current effective UID; root revalidation also requires exact `0700` mode. +Namespace descriptor containers must be tuples, with regression tests for both +rules. ADR 0013 and the artifact specification now describe the actual +factory-bootstrap, PostgreSQL-claim, byte-store initialization sequence. All +reviewer sessions were closed; the repaired SHA requires fresh review. + +The next exact-SHA review found that provider descriptor keys were only +canonicalized, not closed by provider profile; the local layout marker accepted +owner-executable mode; and three active summaries still named direct +`ExternalServiceAdapterFactory[ArtifactStore]` construction. The namespace +identity now rejects unsupported profiles and any missing or unknown +`local-v2` key, the marker requires exact `0600`, and all active summaries use +the bootstrap -> PostgreSQL namespace claim -> byte-store initialization +sequence. The stale-evidence finding raised during the review was a sequencing +false positive: exact-SHA evidence is written only after reviewer sessions +complete. All reviewer sessions were closed; this repair requires fresh review. + +The next exact-SHA review found one Low LocalStorage layout issue: a root with a +valid v2 marker could retain unknown top-level entries. Initialization now +accepts only the marker, `objects`, `tmp`, and `locks` entries, with regression +proof that an added legacy directory fails closed. All reviewer sessions were +closed; the repaired SHA requires fresh review. + +During the next exact-SHA review, `origin/main` advanced through AUTH-09A PR +#132. Review stopped immediately because the ART branch still carried a sibling +`0023` migration and its diff would have obscured merged AUTH history. The +latest `main` is now merged in full. AUTH's `0023_service_actor_identity`, +service-identity runtime, merge intent, tests, and evidence remain unchanged; +At that checkpoint ART was renumbered to `0024_artifact_store_v2` and descended +from AUTH-09A. The +authored queue records AUTH-09A complete and ART-02A3 at its independent gate. +All reviewer sessions were closed; the merged candidate requires fresh +deterministic proof and exact-SHA review. + +The final CI-integrity review found that the cumulative artifact-foundation +coverage source set omitted the new closed product-capability interface module. +The 90 percent workflow gate and its deterministic command-shape assertion now +include `app/interfaces/artifact_operations.py`. All reviewer sessions were +closed; the repaired SHA requires fresh exact-SHA review. + +The next review found two Low closure gaps. The active and next ART focused +commands did not explicitly measure the operations interface, and bootstrap +initialization closed descriptors for configuration and operating-system errors +but not every sanitized `ArtifactStoreError`. Both focused commands now include +the operations interface, and initialization closes on all store errors with a +regression test for an integrity failure. All reviewer sessions were closed; +the repaired SHA requires fresh review. + +The next review found that claim validation still occurred before the cleanup +guard and that exact root names did not constrain nested LocalStorage contents. +All namespace-claim failures now close pinned descriptors. Existing marked +stores are validated before mutation against the real local grammar: one +`objects/sha256` tree with canonical digest names, bounded private put +temporaries, canonical digest locks, and matched crash-recovery hard links. +Foreign nested entries, owners, modes, link counts, and recovery links fail +closed. Regression proof covers mismatched/repeated claims, foreign entries at +every level, and reopening a valid populated store. All reviewer sessions were +closed; the repaired SHA requires fresh review. + +## 2026-07-17 - WS-ART-001-02A3 Merged-Main Final Review + +The branch merged trusted `main` at `0302bcf`, preserving the independently +owned REV planning and the canonical AUTH `0023_service_actor_identity` -> ART +then-current `0024_artifact_store_v2` migration order. Deterministic proof then +passed 268 +ART-focused tests at 93.18 percent changed-scope coverage, 56 byte-store +contract tests at 91.08 percent, four real PostgreSQL migration-safety cases, +Ruff, 92.0 percent docstring coverage, 80 agent-gate tests, stale-contract and +wording scans, Markdown links, and diff hygiene. + +The final internal review found no remaining implementation, architecture, QA, +security, product/ops, reuse, or test-delta defect. Docs and CI correctly +rejected the stale pre-final evidence record. This state-transition commit is +therefore the exact review target; its evidence-only descendant must bind the +reviewed SHA, record every reviewer run, pass the evidence gate, and then receive +final CI/docs confirmation before external checks resume. diff --git a/.agent-loop/WORK_QUEUE.md b/.agent-loop/WORK_QUEUE.md index 95f2d7c6d..c34f8e32e 100644 --- a/.agent-loop/WORK_QUEUE.md +++ b/.agent-loop/WORK_QUEUE.md @@ -4,7 +4,7 @@ | Chunk | Title | Risk | Status | |---|---|---:|---| -| `WS-AUTH-001-09B` | Controlled Service Actor Provisioning | L1 | PR #143 open; coverage-gate behavior repair internally reviewed; replacement checks pending | +| `WS-ART-001-02A3` | ArtifactStore v2 Local Clean Cut | L1 | Internal review and deterministic evidence passed; external checks pending | Live post-merge state remains read from signed `automation/loop-memory` output. This authored queue records the separately approved parallel chunks. @@ -17,7 +17,6 @@ output. This authored queue records the separately approved parallel chunks. | `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 | Inactive until 09B merge/memory and explicit user start | | `WS-QUAL-001-02` | Project Service Coverage | L1 | Inactive until 01B2 merge/memory plus explicit user start | | `WS-POL-002-04` | Locked Runtime Execution And Routing Hardening | L1 | Inactive pending relevant authorization proof and a separate explicit user start | -| `WS-ART-001-02A3` | ArtifactStore v2 Local Clean Cut | L1 | Reviewed in isolated parallel worktree; pending its own PR and merge | | `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Inactive until 02A3 merge and explicit user start | | `WS-ART-001-02C1` | Admission And Put-Attempt Foundation | L1 | Inactive until 02B1 merge and explicit user start | | `WS-ART-001-02C2` | Verification Publication And Fencing | L1 | Inactive until 02C1 merge and explicit user start | @@ -71,10 +70,11 @@ output. This authored queue records the separately approved parallel chunks. | `WS-AUTH-001-07A` | Closed Permission And Action Catalogue | L1 | Merged through PR #126 as `e9d72a1` on 2026-07-15 | | `WS-AUTH-001-07B` | Deny-By-Default Kernel And Self-Action Cutover | L1 | Merged through PR #130 as `90eca12` on 2026-07-15 | | `WS-AUTH-001-08` | Bootstrap Access Administrator Grant | L1 | Merged through PR #131 as `aa0fdcd` on 2026-07-16 | -| `WS-ART-001-02A2` | Committed Source And Local Preparation | L1 | Merged through PR #129 as `9a04434` on 2026-07-16 | | `WS-XINT-001-PLAN` | Lifecycle Boundary Reconciliation | L1 | Merged through PR #139 as `5d353b6` on 2026-07-17 | | `WS-ART-001-OBJECT-STORAGE-AMENDMENT` | AWS-First Object Storage Planning Amendment | L1 | Merged through PR #120 as `4408256` on 2026-07-14 | | `WS-ART-001-02A1` | External Service Adapter Foundation | L1 | Merged through PR #127 as `f64a8e5` on 2026-07-15 | +| `WS-ART-001-02A2` | Committed Source And Local Preparation | L1 | Merged through PR #129 as `9a04434` on 2026-07-16 | +| `WS-AUTH-001-XINT` | Lifecycle Boundary Plan Reconciliation | L1 | Merged through PR #140 as `d541521` on 2026-07-17 | | `WS-ENG-001-02` | Automated Post-Merge Memory | L1 | Merged through PR #122 as `fc89fb6`; schema-v1 output superseded by WS-ENG-001-03 | ## Proposed Next @@ -85,16 +85,17 @@ AUTH-06 merged through PR #124 as `f599551`. AUTH-07A, AUTH-07B, and AUTH-08 merged through PRs #126, #130, and #131. WS-XINT planning merged through PR #139, and its AUTH owner reconciliation merged through PR #140 as `d541521`. AUTH-09A merged through PR #132 as `299363a`, and signed schema-v2 memory -stopped. The user explicitly started AUTH-09B. Do not start AUTH-09C or +stopped. AUTH-09B merged through PR #143 as `053242b`. Do not start AUTH-09C or POL-002-04 automatically. Coverage R10 merged through PR #108. Do not start 01B2, chunk 02, or another coverage implementation chunk from this worktree. -`WS-ART-001-01`, the AWS-first planning amendment, and `02A1` are merged. R2 -and Flow Node are deferred. `02A2` merged through PR #129 as `9a04434`. -`02A3` is reviewed in its isolated worktree and awaits its own PR publication -and human review; later ART chunks remain inactive. +`WS-ART-001-01`, the AWS-first planning amendment, `02A1`, and `02A2` are +merged. R2 and Flow Node are deferred. The user explicitly started `02A3` on +2026-07-16. Its merged-main deterministic proof and exact-SHA internal review +are complete; external checks remain pending and `02B1` must not start +automatically. Coverage work proceeds independently in its own worktree and is not owned by this AUTH queue update. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md index b7c495969..e99f7c809 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md @@ -8,13 +8,13 @@ Each chunk is one PR. No later chunk starts automatically. | `WS-ART-001-01` | Artifact domain and LocalStorage v1 foundation. | L1 | Merged through PR #101 | | `WS-ART-001-OBJECT-STORAGE-AMENDMENT` | Make AWS S3 the v0.1 production provider with MinIO local/CI protocol proof; keep optional providers outside v0.1. | L1 | Merged through PR #120 as `4408256` | | `WS-ART-001-02A1` | Install only ADR 0014's small typed external-service adapter/factory foundation without migrating a capability. | L1 | Merged through PR #127 as `f64a8e5` | -| `WS-ART-001-02A2` | Add bounded committed-source preparation and inactive scratch-cleanup mechanics without changing the active v1 port. | L1 | Merged through PR #129 as `9a04434` | -| `WS-ART-001-02A3` | Replace ArtifactStore v1 with byte-only v2, activate API-startup and Celery Beat scratch cleanup, migrate schema/callers/factory, and remove `flow_node` in one atomic clean cut. | L1 | Reviewed in isolated worktree; PR publication pending | +| `WS-ART-001-02A2` | Add bounded committed-source preparation and inactive scratch-cleanup mechanics without changing the active v1 port. | L1 | Merged through PR #129 as `9a04434` on 2026-07-16 | +| `WS-ART-001-02A3` | Replace ArtifactStore v1 with byte-only v2, activate API-startup and Celery Beat scratch cleanup, migrate schema/callers/factory, and remove `flow_node` in one atomic clean cut. | L1 | Internal review and deterministic evidence passed; external checks pending | | `WS-ART-001-02B1` | Implement the S3-compatible adapter, MinIO integration, and AWS S3 production profile. | L1 | Proposed after 02A3 | | `WS-ART-001-02C1` | Add the generic durable-byte admission ledger and durable put-attempt state foundation without provider execution. | L1 | Proposed after 02B1 | | `WS-ART-001-02C2` | Add put resolution, verification publication, complete-object observation, immutable receipts, and PostgreSQL execution fencing without recovery attempts or routes. | L1 | Proposed after 02C1 | | `WS-ART-001-02C3` | Add the recovery-attempt model and exact idempotent source-job to retry-job chain without public or Operator routes. | L1 | Proposed after 02C2 | -| `WS-ART-001-02D` | Add hidden Operator content/job/retry/recovery/audit APIs, canonical resource composition, and production-readiness checks while actions and provider profiles remain inactive. | L1 | Proposed after 02C3, AUTH-09E, and AUTH custody registration | +| `WS-ART-001-02D` | Add hidden Operator content/job/retry/recovery/audit APIs, canonical resource composition, and production-readiness checks while actions and provider profiles remain inactive. | L1 | Proposed after 02C3, AUTH-09E, and `WS-AUTH-001-ART-CUSTODY` | | `WS-ART-001-03` | Store and bind guide-source bytes; add same-snapshot setup recovery through the authorized artifact reader. | L1 | Proposed after 02D | | `WS-ART-001-04A` | Add task-scoped upload sessions/items, trusted archive inspection, independent verification, immutable sealing, and artifact-set manifests. | L1 | Proposed after 03 | | `WS-ART-001-04B` | Execute authoritative pre-submit against sealed artifact sets and persist exact admissions with bounded infrastructure continuation. | L1 | Proposed after 04A | @@ -54,6 +54,8 @@ Physical deletion and semantic search require separate approved initiatives. - Artifact actions follow AUTH planned registration -> hidden ART behavior and canonical resource composition -> AUTH evaluator integration and activation. + `WS-AUTH-001-ART-CUSTODY` first transfers the 25 current ART actions to eight + exact AUTH activation custodians without changing mappings or availability. Protected service commands first pass AUTH-09E. They consume canonical `ActorProfile.id`, closed `ActionId` and `PermissionId` catalogues, and exact fixed service matrix rows. ART never changes action diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md index d233bd6e9..d36712f2a 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md @@ -79,12 +79,13 @@ access. S3CompatibleArtifactStore and LocalStorageAdapter are registered explicitly through -`ExternalServiceAdapterFactory[ArtifactStore]`. Only the artifact-storage -orchestration service receives the writable port through composition-root -dependency injection. Product modules and Celery jobs receive typed artifact +`ExternalServiceAdapterFactory[ArtifactStoreBootstrap]`. The composition root +claims the bootstrap's exact namespace in PostgreSQL before initialization +yields the writable `ArtifactStore`. Only artifact-storage orchestration +receives that port. Product modules and Celery jobs receive typed artifact operations from that owner. No service locator, plugin discovery, concrete -import, fallback constructor, dual factory, or bypassing writable-port -injection exists. +import, fallback constructor, dual factory, or bypassing writable-port injection +exists. ## D13 - Private S3 Deployment @@ -101,7 +102,11 @@ access mutation are denied. Static credentials are local/CI MinIO only. ## D14 - Clean Cut The `flow_node` backend value and ArtifactStore v1 contract are removed in -02A3. Pre-production data may be rebuilt. No backward compatibility is retained. +02A3. Migration `0025` refuses populated v1 artifact tables before DDL and +preserves their prior schema and rows. An empty pre-production environment may +be reprovisioned out of band and authoritative bytes reingested through v2; the +migration performs no fabricated backfill. No backward compatibility is +retained. ## D15 - Route Deployment Claims @@ -244,8 +249,8 @@ must not invent a service principal, inspect AUTH grants or static matrix membership, or activate an authorization action. The seven fixed service identities and complete 25-action custody transfer are -defined in -`../WS-XINT-001-lifecycle-boundary-reconciliation/AUTH_ART_HANDOFF.md`. Operator -verification retry remains an independently authorized human action. Service -identity, Celery executor identity, and execution-generation fencing never -substitute for one another. +defined canonically in +`../WS-AUTH-001-workstream-authorization-service/ACTIVATION_CUSTODY.md` and its +`WS-AUTH-001-ART-CUSTODY` chunk contract. Operator verification retry remains +an independently authorized human action. Service identity, Celery executor +identity, and execution-generation fencing never substitute for one another. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md index 6c834a363..ce764b8d7 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md @@ -11,8 +11,9 @@ deferred initiative input. ```text FastAPI / Celery composition roots --> ExternalServiceAdapterFactory[ArtifactStore] --> ArtifactStore v2 byte capability +-> ExternalServiceAdapterFactory[ArtifactStoreBootstrap] +-> PostgreSQL exact namespace claim +-> initialized ArtifactStore v2 byte capability -> LocalStorageAdapter local and focused tests -> S3CompatibleArtifactStore MinIO integration; AWS S3 production -> ArtifactService orchestration @@ -341,12 +342,13 @@ fact. AUTH-07 registers the closed artifact permissions, AUTH-08 defines applicable Operator grants, AUTH-09A defines the static service-action matrix, AUTH-09B provisions fixed service ActorProfiles and ActorIdentityLinks, and AUTH-09E -admits them at runtime. AUTH registers each planned action and its activation -custodian; the owning WS-ART chunk then supplies hidden canonical resource +admits them at runtime. `WS-AUTH-001-ART-CUSTODY` transfers every current ART +action to the exact AUTH activation custodian without changing mappings or +availability. The owning WS-ART chunk then supplies hidden canonical resource composition, guards, surface declarations, behavior, and tests while the real -kernel fails closed; AUTH finally integrates the evaluator and alone changes -availability to active. Later AUTH-12, AUTH-14, and AUTH-15 are not alternate -artifact activation paths. +kernel fails closed; the named AUTH activation chunk finally integrates the +evaluator and alone changes availability to active. Later AUTH-12, AUTH-14, and +AUTH-15 are not alternate artifact activation paths. Complete reads have an end-to-end verification deadline derived from the 512 MiB maximum and minimum supported throughput. The deadline is shorter than the @@ -415,8 +417,10 @@ atomicity. ## Migration Rules -- Pre-production data is rebuilt when an old caller-URI/hash record cannot be - converted from authoritative stored bytes. +- Migration `0025` refuses populated v1 artifact tables before DDL and preserves + the prior schema and rows. Pre-production reprovisioning happens out of band + into an empty database/storage namespace, followed by v2 reingest from + authoritative bytes; unavailable bytes are not fabricated or migrated. - `flow_node` configuration is rejected after the clean-cut settings migration. - ArtifactStore v1 methods are removed in the same chunk that migrates all LocalStorage callers and tests. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md index 685f7ba0f..fa254d124 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md @@ -20,7 +20,7 @@ | AWS configuration becomes active without live proof | Critical | Separate caller-ARN-bound readiness/runtime/negative probe records feed a credential-free coordinator; startup and every I/O require an exact activation refreshed every 5 minutes and expiring within 15 minutes. MinIO proof cannot create it. | | AWS operation outlives its activation proof | Critical | Admission requires remaining activation TTL to cover the total I/O deadline plus persistence/clock margins, and the terminal transaction rechecks the same unexpired activation. | | Registered permission is mistaken for executable artifact authority | Critical | AUTH registration remains planned; ART supplies hidden canonical resources, guards, surfaces, tests, and terminal revalidation; AUTH alone integrates the evaluator and activates the exact action afterward. | -| ART and AUTH both claim activation custody | Critical | Transfer all 25 current ART mappings to closed AUTH custodians, preserve mappings, remove unused ART owner enums, and reject partial or dual ownership through parity tests. | +| ART and AUTH both claim activation custody | Critical | `WS-AUTH-001-ART-CUSTODY` transfers all 25 current ART mappings to eight closed AUTH custodians, preserves mappings and availability, removes unused ART owner enums, and rejects partial or dual ownership through parity tests. | | Broker publication failure strands verification/recovery | High | Periodic PostgreSQL scanner with bounded SLA and duplicate-safe publication. | | Duplicate/stale Celery execution writes terminal state | Critical | PostgreSQL clock, fresh executor UUID, generation fencing, and zero-row stale finalization rollback. | | Slow progressive read outlives execution lease | High | End-to-end verification deadline shorter than lease by a persistence margin. | diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index 96672b23e..6cda9a86b 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -4,9 +4,15 @@ Original planning merged through PR #97, artifact/LocalStorage foundation merged through PR #101, the AWS-first object-storage amendment merged through -PR #120 as `4408256`, and the external-service adapter foundation merged -through PR #127 as `f64a8e5`. `WS-ART-001-02A2` merged through PR #129 as -`9a04434` on 2026-07-16. +PR #120 as `4408256`, the external-service adapter foundation merged through +PR #127 as `f64a8e5`, and committed-source preparation merged through PR #129 +as `9a04434` on 2026-07-16. The user explicitly started +`WS-ART-001-02A3` on 2026-07-16. + +The planning-only cross-initiative boundary reconciliation merged through +PR #139 as `5d353b6`, and AUTH's owner reconciliation merged through PR #140 as +`d541521`. ART now consumes AUTH's canonical activation-custody and prepared +mutation contracts without editing or activating AUTH runtime behavior. The Flow Node-focused amendment candidate `6cc422d` passed deterministic checks but failed internal review on recovery/API completeness. Before repair, the user @@ -25,21 +31,29 @@ approval or reusable evidence. Its source remains on branch ## Current Work -No artifact implementation chunk is active. `WS-ART-001-02A3` implementation -and review are complete at `935b1a2` in its isolated worktree and await its own -PR publication. Active ArtifactStore v1 behavior, provider selection, and -product lifecycle remain unchanged until that separate PR merges. +`WS-ART-001-02A3` implementation and merged-main deterministic repair are +complete. PR #141 is open. Exact-SHA review found and repaired a residual +LocalStorage startup race, typed-factory startup mismatch, and vague Operator +resource vocabulary. Fresh deterministic coverage and all required internal +reviewer tracks passed; external checks are now pending. Its approved +boundary atomically replaces ArtifactStore v1 with byte-only v2, migrates +LocalStorage and the empty pre-production artifact schema, installs the +immutable storage-namespace fence, +removes dormant `flow_node` configuration, and activates startup plus periodic +scratch cleanup. It does not activate product ingest, durable admission, +put-attempt resolution, verification jobs, or recovery. ## Next Proposed Chunk -`02A2` added committed-source preparation and narrowed LocalStorage internals -without changing the active port. `02A3` performs the atomic -ArtifactStore v2/LocalStorage/schema cut and removes `flow_node`. `02B1` then -owns MinIO and AWS S3. There is no active R2 chunk. +`02B1` owns the S3-compatible adapter, MinIO proof, and AWS S3 profile after +`02A3` merges and receives a separate explicit start. There is no active R2 or +Flow Node chunk. ## Gate -PR #129 merged `WS-ART-001-02A2` as `9a04434`. The current artifact gate is -publication and human review of the already reviewed `WS-ART-001-02A3` -worktree; no PR is attributed here until it is actually opened. No later -artifact chunk starts automatically, and only the user may approve a merge. +All reviews before the latest repairs are retained as history, not reused as +final provenance. Deterministic proof and exact-SHA internal review have passed. +The current gate is GitHub Actions, CodeRabbit, and explicit human review. +Durable admission, put attempts, verification publication, and recovery remain +in their later owning chunks. No +later artifact chunk starts automatically, and only the user may approve merge. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A1-external-service-adapter-foundation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A1-external-service-adapter-foundation.md index 98243f9a7..6a1975c8e 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A1-external-service-adapter-foundation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A1-external-service-adapter-foundation.md @@ -57,7 +57,7 @@ this chunk. ## Exact CI Coverage Gate ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A2-committed-source-local-preparation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A2-committed-source-local-preparation.md index 365e25846..603470d37 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A2-committed-source-local-preparation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A2-committed-source-local-preparation.md @@ -71,7 +71,7 @@ active ArtifactStore v1 contract and runtime behavior remain unchanged. ## Exact CI Coverage Gate ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 ``` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A3-artifact-store-v2-local-clean-cut.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A3-artifact-store-v2-local-clean-cut.md index dcf9ed503..0d4c66b0e 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A3-artifact-store-v2-local-clean-cut.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A3-artifact-store-v2-local-clean-cut.md @@ -1,6 +1,6 @@ # Chunk Contract: WS-ART-001-02A3 - ArtifactStore v2 Local Clean Cut -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02A2 +Initiative: `WS-ART-001` | Risk: L1 | Status: Internal review and deterministic evidence passed; external checks pending Artifact contract phase: `artifact_store_cutover` @@ -22,6 +22,8 @@ configuration. No compatibility path remains after this PR. startup cleanup wiring under `backend/app/workers/` and the composition root; - one Alembic migration and migration tests; - focused configuration, factory, LocalStorage, and v2 conformance tests; +- `backend/tests/test_projects.py` only for the real-PostgreSQL cumulative + `app/workers/*` coverage repair exposed by GitHub CI; no project runtime code; - `.github/workflows/backend.yml` only to expand the exact 90 percent scoped gate; - `scripts/check_stale_artifact_contracts.py` only to advance the artifact contract phase to `artifact_store_cutover` after the atomic clean cut; @@ -42,7 +44,8 @@ configuration. No compatibility path remains after this PR. - ArtifactStore exposes only `put(CommittedArtifactSource)`, read-only `observe_put_result(ArtifactCommitment)`, `open`, and `head`; - LocalStorage implements v2 with exclusive immutable publication, range reads, - exact replay, bounded async I/O, and sanitized errors; + exact replay, bounded async I/O, and sanitized errors. Startup fails closed + on a single-link provider temporary instead of deleting a possibly live file; - the migration removes provider retention/receipt semantics, adds immutable provider profile/storage namespace, and rebuilds or explicitly refuses incompatible pre-production rows; @@ -50,7 +53,14 @@ configuration. No compatibility path remains after this PR. non-secret namespace fingerprint. Startup and every provider operation insert-or-validate it transactionally before I/O; concurrent differing first writers prove one winner and one pre-I/O failure. Every replica references it. - v0.1 uses one deployment fence, not a per-operation router or hot switch; + LocalStorage startup requires a pre-provisioned owner-private durable root and + hashes its normalized path plus filesystem identity, so same-path replacement + fails before adapter layout mutation. One composition-only bootstrap pins the + root without layout mutation, PostgreSQL returns a claim for that exact + namespace identity, and initialization rechecks the path before writing + through the pinned descriptor. The byte-only `ArtifactStore` does not expose + this startup lifecycle. v0.1 uses one deployment fence, not a + per-operation router or hot switch; - upload state includes `replay_required`; provider acknowledgement creates `stored_pending_verification` and `pending/unknown/unknown`, never bindability; - v1 verify/retain/release/provider-receipt methods are absent from active code; @@ -81,7 +91,7 @@ configuration. No compatibility path remains after this PR. ## Exact CI Coverage Gate ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 @@ -92,7 +102,7 @@ coverage report --include='app/main.py' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_artifacts.py tests/test_config.py tests/test_artifact_store_conformance.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifacts --cov=app.modules.artifacts --cov=app.core.config --cov=app.main --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_artifacts.py tests/test_artifact_preparation.py tests/test_config.py tests/test_app.py tests/test_api_controls.py tests/test_artifact_architecture.py tests/test_artifact_cleanup_wiring.py tests/test_artifact_store_conformance.py tests/test_local_artifact_store.py -q --cov=app.adapters.artifacts --cov=app.core.cancellation --cov=app.core.file_locks --cov=app.interfaces.artifact_operations --cov=app.interfaces.artifacts --cov=app.modules.artifacts --cov=app.core.config --cov=app.main --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) (cd backend && .venv/bin/ruff check app tests) python3 scripts/check_stale_artifact_contracts.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md index 63e75cf8f..756acf1c9 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02B1-s3-compatible-minio-aws.md @@ -1,4 +1,4 @@ -# Chunk Contract: WS-ART-001-02B1 S3-Compatible MinIO And AWS +# Chunk Contract: WS-ART-001-02B1 - S3-Compatible MinIO And AWS Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02A3 @@ -76,7 +76,7 @@ credential-delivery contract. ## Exact CI Coverage Gate ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 @@ -88,7 +88,7 @@ coverage report --include='app/main.py' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis minio (cd backend && .venv/bin/ruff check app tests) -(cd backend && .venv/bin/pytest tests/test_artifact_store_conformance.py tests/test_s3_artifact_store.py tests/test_aws_credential_isolation.py tests/test_config.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifacts --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) +(cd backend && .venv/bin/pytest tests/test_artifact_store_conformance.py tests/test_s3_artifact_store.py tests/test_aws_credential_isolation.py tests/test_config.py -q --cov=app.adapters.artifacts --cov=app.interfaces.artifact_operations --cov=app.interfaces.artifacts --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) python3 scripts/check_stale_artifact_contracts.py python3 scripts/test_agent_gates.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md index 788dfd297..3306c5781 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md @@ -66,7 +66,7 @@ publication, recovery, Operator routes, and product cutovers inactive. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 @@ -78,7 +78,7 @@ coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_artifact_admission.py tests/test_config.py -q --cov=app.modules.artifacts --cov=app.modules.audit --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_artifact_admission.py tests/test_config.py -q --cov=app.interfaces.artifact_operations --cov=app.modules.artifacts --cov=app.modules.audit --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) (cd backend && .venv/bin/ruff check app tests) python3 scripts/check_stale_artifact_contracts.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md index bbc8ffa3f..26db8e486 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md @@ -81,7 +81,7 @@ Do not add recovery attempts, Operator routes, or product cutovers. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 @@ -93,7 +93,7 @@ coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_artifact_verification.py tests/test_config.py -q --cov=app.modules.artifacts --cov=app.modules.audit --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_artifact_verification.py tests/test_config.py -q --cov=app.interfaces.artifact_operations --cov=app.modules.artifacts --cov=app.modules.audit --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) (cd backend && .venv/bin/ruff check app tests) python3 scripts/check_stale_artifact_contracts.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md index e6cc26492..b29010047 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C3-recovery-attempt-idempotency.md @@ -58,7 +58,7 @@ retry-job chain without exposing routes or changing product lifecycle state. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md index af3b26ea5..ee82beafb 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02D-operator-artifact-operations.md @@ -1,6 +1,6 @@ # Chunk Contract: WS-ART-001-02D Operator Artifact Operations -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02C3, AUTH-09E, and AUTH custody registration +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02C3, AUTH-09E, and `WS-AUTH-001-ART-CUSTODY` Artifact contract phase: `artifact_store_cutover` @@ -45,8 +45,8 @@ product cutover. Chunk 07 owns every live AWS provider inspection. - exact Operator APIs exist for resource-scoped binding discovery, replicas, receipts, verification job, retry, recovery-attempt read, artifact audit listing, and read-only admission usage. -- AUTH-07A/07B, AUTH-08, AUTH-09A through AUTH-09E, and the reviewed - custody-registration checkpoint are merged before this chunk starts. They +- AUTH-07A/07B, AUTH-08, AUTH-09A through AUTH-09E, and + `WS-AUTH-001-ART-CUSTODY` are merged before this chunk starts. They provide the complete typed/SQL planned action registry, Operator grants, static service-action matrix, provisioned service ActorProfiles and ActorIdentityLinks, fixed service runtime admission, and AUTH activation @@ -124,7 +124,7 @@ product cutover. Chunk 07 owns every live AWS provider inspection. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 @@ -137,7 +137,7 @@ coverage report --include='app/api/router.py' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_artifact_operator_api.py tests/test_artifact_authorization.py tests/test_config.py -q --cov=app.modules.artifacts --cov=app.adapters.artifacts --cov=app.api.router --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_artifact_operator_api.py tests/test_artifact_authorization.py tests/test_config.py -q --cov=app.interfaces.artifact_operations --cov=app.modules.artifacts --cov=app.adapters.artifacts --cov=app.api.router --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) (cd backend && .venv/bin/ruff check app tests) python3 scripts/check_stale_authorization_docs.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md index 0e9c1e449..cc146e27a 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03-guide-source-cutover.md @@ -104,7 +104,7 @@ authorized Workstream artifact reader. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md index d05f4b1e7..58d4ba625 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04A-upload-inspection-sealing.md @@ -84,7 +84,7 @@ manifests without running pre-submit or creating submissions. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md index 347420faa..ad8e193e4 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-04B-pre-submit-admission.md @@ -85,7 +85,7 @@ when artifact infrastructure is temporarily unavailable. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md index f3e10a6db..d3038977f 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-05-submission-artifact-cutover.md @@ -90,7 +90,7 @@ enter `evaluation_pending` without manager finalization. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md index 693030f82..fc02995c8 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06A-checker-input-materialization.md @@ -71,7 +71,7 @@ bounded isolated checker workspaces. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 @@ -88,7 +88,7 @@ coverage report --include='app/modules/checkers/*' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_checker_artifacts.py tests/test_checker_materialization.py tests/test_artifact_scratch_manager.py -q --cov=app.modules.checkers --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_checker_artifacts.py tests/test_checker_materialization.py tests/test_artifact_scratch_manager.py -q --cov=app.interfaces.artifact_operations --cov=app.modules.checkers --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) (cd backend && .venv/bin/ruff check app tests) python3 scripts/check_stale_artifact_contracts.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md index 7a6d5a8ad..20a634d1a 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-06B-checker-output-routing.md @@ -79,7 +79,7 @@ lease, assignment, or decision. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 @@ -96,7 +96,7 @@ coverage report --include='app/modules/checkers/*' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_checker_artifacts.py tests/test_post_submit_checkers.py -q --cov=app.modules.checkers --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_checker_artifacts.py tests/test_post_submit_checkers.py -q --cov=app.interfaces.artifact_operations --cov=app.modules.checkers --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) (cd backend && .venv/bin/ruff check app tests) python3 scripts/check_stale_artifact_contracts.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md index 75fdbd8cd..8b9574a3d 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-07-recovery-live-proof.md @@ -121,7 +121,7 @@ without direct database inspection or Terminal Benchmark product coupling. ## Exact CI Coverage Gates ```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 +coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 coverage report --include='app/core/config.py' --precision=2 --fail-under=90 coverage report --include='app/workers/*' --precision=2 --fail-under=90 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-external-review-response.md new file mode 100644 index 000000000..8f10faaa2 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-external-review-response.md @@ -0,0 +1,68 @@ +# External Review Response: WS-ART-001-02A3 + +## Current State + +- Pull request: #141, open and not draft +- Reviewed implementation SHA: `c8eccaafb6b4f0615b1b37049edfcb0368dd6fb2` +- Latest published SHA before this response: `cdeef292b8137cf8c48cc040d82f1a93dffa4d2c` +- Prior GitHub and CodeRabbit results predate required AUTH/CON main + reconciliation and are not completion evidence for the current candidate +- Latest repaired SHA external checks: pending publication +- Human merge approval: pending + +Internal sub-agent review is recorded only in the separate internal-review +evidence. This file records CodeRabbit, GitHub, and later human PR feedback. + +## CodeRabbit Findings Addressed + +| Finding | Status | Response | +|---|---:|---| +| Local layout marker was not published atomically. | Fixed | Marker bytes are written and fsynced through a private temporary, installed with an exclusive hard link under a root lock, and recovered only through the exact validated one-link or same-inode two-link states. | +| Stored upload states admitted partial result metadata. | Fixed | Model and migration require both `content_id` and `provider_object_ref` for stored states and require both null otherwise. | +| Independent upload items were fenced by a shared session CAS. | Fixed | Provider I/O is fenced by the item attempt; finalization locks and validates the current open session without requiring an unchanged session CAS. | +| Reservation accounting silently clamped drift. | Fixed | Both counters are validated before exact subtraction; inconsistent accounting raises `ArtifactIntegrityError`. | +| Downgrade proof did not reject leftover v2 columns. | Fixed | Migration tests assert every v2-only replica, upload, receipt, and namespace column is absent after downgrade. | +| Architecture checks matched provider method names globally. | Fixed | The AST gate now follows `ArtifactStore` typed receiver provenance instead of unrelated `put`, `open`, or `head` calls. | +| Scratch-worker import gate ignored plain imports. | Fixed | Both `ast.Import` and `ast.ImportFrom` are inspected. | +| Test settings cache could leak temporary worker configuration. | Fixed | Cache teardown is registered as a test finalizer. | +| Concurrent fake acknowledgements used mutable arrival state. | Fixed | Each call snapshots its arrival number before the barrier. | +| Failure-path assertions did not prove handler exclusivity. | Fixed | Each branch asserts its required handler once and the alternate handler never. | +| Cutover gate test passed the old phase literal. | Fixed | Both scans use `gate.ARTIFACT_CONTRACT_PHASE`, which is `artifact_store_cutover`. | +| Focused commands and cumulative gates measured different source sets. | Fixed | Active and future ART commands include their exact owned operations/core sources; deterministic gate tests pin the workflow commands and cumulative retention. | +| Removed `provider_manifest_id` was absent from migration proof but not the ORM test. | Fixed | The active-model clean-cut test explicitly rejects the field. | +| A `#139` line could be parsed as a malformed heading. | Fixed | The PR reference is attached to normal prose. | + +## Finding Not Applied + +CodeRabbit's generic pre-merge widget reported 62 percent docstring coverage +against an 80 percent default. The repository's authoritative configured +docstring gate is 92.0 percent and passes. Lowering or duplicating that gate +would weaken the repository contract, so no change was made for the generic +widget warning. + +## Validation After Repairs + +```text +73 filesystem/architecture/wiring tests: PASS +59 final LocalStorage/conformance tests: PASS +9 critical real PostgreSQL orchestration tests: PASS +5 migration-chain tests, including 4 clean-cut safety tests: PASS +ART exact scoped coverage: 93.31% +Ruff: PASS +Repository docstring gate: 92.0%, PASS +Stale artifact/authorization/Workstream scans: PASS +Markdown links: PASS +Agent gates: 80 PASS +All nine exact-SHA internal reviewer tracks: PASS +Open sub-agent sessions: none +``` + +GitHub must rerun the isolated full repository suite, 78 percent whole-app +floor, all cumulative scoped 90 percent gates, and real API drill on the newly +published evidence-bound head. + +## Remaining External Gate + +Push the evidence-only descendant, wait for GitHub Actions and CodeRabbit on +that head, address every still-valid new finding, and stop for the user's +explicit merge decision. Do not start `WS-ART-001-02B1` automatically. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-internal-review-evidence.md new file mode 100644 index 000000000..4e7375066 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-internal-review-evidence.md @@ -0,0 +1,148 @@ +# Internal Review Evidence: WS-ART-001-02A3 + +## Chunk + +`WS-ART-001-02A3`: ArtifactStore v2 Local Clean Cut + +open sub-agent sessions: none + +valid findings addressed: yes + +## Reviewed Revision + +Reviewed code SHA: `c8eccaafb6b4f0615b1b37049edfcb0368dd6fb2` + +Reviewed at: 2026-07-18T01:31:17Z + +Reviewer run IDs: senior-engineering=019f72b5-2d70-7471-a006-3580f3844107; architecture=019f72b5-3692-70a2-9713-2cdd4b9d403e; QA/test=019f72b5-438d-71e3-bc2a-43a078865dbd; security/auth=019f72b5-52c6-7980-a3c7-451a66ce3f9d; product/ops=019f72ce-c029-76b1-9291-104b691f6014; reuse/dedup=019f72ce-cdb9-7fe2-8dea-d5820d675a38; CI-integrity=019f72ce-d8f1-7b00-a5bb-d8a7f835f2e4; test-delta=019f72d5-7d3c-7cd2-90c4-9b5facd31041; docs=019f72ce-e898-7d80-97da-9ec76ac6296a + +Only review artifacts may change after this reviewed SHA while follow-up review +and evidence closure complete. No implementation, migration, test, workflow, +policy, or chunk-contract change is permitted without invalidating this record. + +## Reviewed Change + +- Replaced ArtifactStore v1 with the four-operation byte-only v2 port and one + typed factory path. +- Rebuilt LocalStorage around immutable content-derived paths, exact replay, + bounded range reads, no-follow descriptors, exclusive publication, and + sanitized failures. +- Added the immutable storage-namespace fence and recorded provider + acknowledgement as `stored_pending_verification` with replica state + `pending/unknown/unknown`. +- Made LocalStorage layout-marker publication exclusive and crash recoverable, + including the durable marker-plus-temporary hard-link state. +- Made same-object replica finalization atomic across concurrent PostgreSQL + transactions while retaining one append-only receipt per upload item and + allowing independent items in one upload session to finalize concurrently. +- Guaranteed best-effort resource cleanup always attempts digest-lock release + without masking the original sanitized error or caller cancellation. +- Removed v1 provider-retention/receipt semantics, dormant Flow Node + configuration, and invented authorization ownership. +- Activated only bounded startup and Celery Beat scratch cleanup. Product + ingest, S3, verification, recovery, review, and AUTH actions remain inactive. + +## Reviewer Results + +| Reviewer | Result | Blocking findings | Notes | +|---|---:|---|---| +| senior engineering | PASS | None | The v2 port remains narrow; namespace validation, transaction boundaries, crash recovery, migration refusal, and cleanup activation have no remaining operational defect. | +| QA/test | PASS | None | V2 surface, crash states, namespace fencing, migration refusal, cleanup wiring, concurrency, and shared preparation helpers have complete focused regression proof. | +| security/auth | PASS | None | Filesystem ownership/mode/link checks, namespace fencing, sanitized errors, and ART/AUTH separation have no remaining actionable security finding. | +| product/ops | PASS | None | Acknowledgement remains non-bindable; no Operator route, review/revision decision, contribution, compensation, or reputation behavior is activated. | +| architecture | PASS | None | One ADR 0014 factory path remains; writable storage stays inside ART orchestration and product modules receive only closed capabilities. | +| ci integrity | PASS | None | The exact five cumulative 90 percent gates and the 78 percent whole-app floor remain fail closed with no workflow bypass. | +| docs | PASS | None | ADR, specification, glossary, migration wording, links, and future-chunk boundaries match the implementation. | +| reuse/dedup | PASS | None | Sealed-source preparation is centralized in the shared test helper; no factory bypass, production duplication, or cross-initiative runtime coupling remains. | +| test delta | PASS | None | V1 vectors were replaced by stronger v2 assertions without skips, weakened expectations, or false concurrency proof. | + +## Valid Findings Addressed + +- GitHub Backend CI passed the full repository coverage and first three focused + gates, then reported 88.20 percent cumulative `app/workers/*` coverage. Added + real-PostgreSQL coverage for the known `ProjectServiceError` worker path, + including sanitized result, persisted `setup_blocked` state, and bounded log + metadata. The new test executes the previously missed lines 174-193 without + narrowing or weakening the 90 percent worker gate. + +- Replaced the replica read-then-insert race with PostgreSQL + `ON CONFLICT DO NOTHING` plus a current-row load and strict content, + namespace, adapter, and provider-profile validation. +- Added a real two-session concurrency test proving two exact replay + finalizations produce one replica and two receipts. +- Made LocalStorage cleanup attempt digest-lock release in an independent + `finally` even when temporary unlink or fsync fails. +- Added regression coverage proving cleanup errors cannot leak the owned lock + or replace the caller's sanitized failure/cancellation semantics. +- Added a namespace-only populated-v2 downgrade refusal test so the deployment + fence cannot be discarded by a clean-cut rollback. +- Applied every still-valid CodeRabbit finding: atomic marker publication, + complete upload-result metadata constraints, per-item rather than shared + session CAS fencing, exact reservation accounting, full downgrade cleanup, + typed receiver provenance, complete import scanning, cache teardown, + mutually exclusive failure handlers, deterministic concurrent test results, + cutover-phase binding, exact coverage-source alignment, and removed-field + assertions. External dispositions are recorded separately. +- Restored AUTH status to trusted `main` and kept ART gate assertions within + the ART boundary. +- Reconciled the merged AUTH `0024` migration with ART `0025` and shifted only + inactive AUTH-10 through AUTH-15 reservations to `0026` through `0031`, with + an explicit superseding decision and no AUTH runtime activation. +- Strengthened the concurrent first-namespace proof through the real provider + operation: the winner observes the committed namespace before `put`, while + the conflicting loser performs zero provider I/O. +- Consolidated sealed-source test preparation into one shared helper after the + reuse reviewer found two residual duplicate paths; the repair received a + clean exact-SHA reuse review before the final nine-track fanout. + +## Commands Run + +```bash +cd backend && .venv/bin/ruff check app/modules/artifacts/repository.py app/modules/artifacts/service.py app/adapters/artifacts/local.py tests/test_artifacts.py tests/test_local_artifact_store.py +cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest -q tests/test_artifacts.py::test_concurrent_same_object_finalization_reuses_one_replica +cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest -q tests/test_artifacts.py::test_put_acknowledgement_stops_at_pending_verification tests/test_artifacts.py::test_concurrent_same_object_finalization_reuses_one_replica tests/test_artifacts.py::test_independent_items_in_one_session_finalize_without_shared_cas_replay tests/test_artifacts.py::test_database_rejects_partial_upload_result_metadata tests/test_artifacts.py::test_reservation_accounting_rejects_drift_without_clamping tests/test_artifacts.py::test_same_item_replay_finalizes_once_without_double_accounting tests/test_artifacts.py::test_terminal_put_failure_releases_reservation_and_fails_item tests/test_artifacts.py::test_namespace_mismatch_fails_before_provider_io tests/test_artifacts.py::test_concurrent_different_first_namespace_writers_have_one_winner +cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test COVERAGE_FILE=.coverage.art02a3 .venv/bin/coverage run -m pytest -q tests/test_artifacts.py tests/test_artifact_preparation.py tests/test_config.py tests/test_app.py tests/test_api_controls.py tests/test_artifact_architecture.py tests/test_artifact_cleanup_wiring.py tests/test_artifact_store_conformance.py tests/test_local_artifact_store.py +cd backend && .venv/bin/pytest -q tests/test_artifact_architecture.py tests/test_artifact_cleanup_wiring.py tests/test_artifact_store_conformance.py tests/test_local_artifact_store.py +cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/pytest -q tests/test_alembic.py::test_alembic_upgrade_and_downgrade tests/test_alembic.py::test_artifact_store_v2_empty_clean_cut_and_reversible_shape tests/test_alembic.py::test_artifact_store_v2_refuses_populated_v1_before_ddl tests/test_alembic.py::test_artifact_store_v2_refuses_populated_v2_downgrade_before_ddl tests/test_alembic.py::test_artifact_store_v2_waits_for_concurrent_v1_writer_and_refuses +cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest -q tests/test_projects.py::test_project_setup_worker_persists_sanitized_domain_failure +cd backend && .venv/bin/docstr-coverage --config .docstr.yaml +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_stale_authorization_docs.py +python3 scripts/check_stale_workstream_wording.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +Results: the prior complete focused run passed 268 ART tests; the final repaired +candidate passed 73 filesystem/architecture/wiring tests, 59 final +LocalStorage/conformance tests, nine critical real-PostgreSQL orchestration +cases, and five migration-chain cases, including all four ART clean-cut safety +cases. The corrected exact ART scope +coverage was 93.31 percent. Ruff, 92.0 percent docstring coverage, stale +contract/authorization/wording, +Markdown links, 80 agent-gate tests, and diff checks passed. +The added real-PostgreSQL worker test passed and executes the ten statements in +the previously missed known-domain-failure branch. GitHub must confirm the +cumulative worker percentage on the republished head. + +The isolated full repository suite, 78 percent whole-app floor, cumulative +worker/main gates, and real API drill remain authoritative in GitHub Backend CI. +The intentionally aborted broad local project/task run is not completion +evidence; it was stopped because full regression is assigned to GitHub Actions. + +## Remaining Risks + +- The migration intentionally refuses any populated pre-production v1 artifact + table rather than fabricating v2 provenance. +- LocalStorage remains development/test only. Hosted bytes remain disabled + until the separately approved S3-compatible adapter and AWS activation proof. +- `stored_pending_verification` remains non-bindable. Admission, verification, + publication, and recovery belong to later chunks. + +## Stop Condition + +Publish the evidence-bound head for GitHub CI, a fresh CodeRabbit review, and +human review. +Do not merge without explicit user approval and do not start +`WS-ART-001-02B1`. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-pr-trust-bundle.md new file mode 100644 index 000000000..9db5369eb --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02A3-pr-trust-bundle.md @@ -0,0 +1,171 @@ +# PR Trust Bundle: WS-ART-001-02A3 + +## Chunk + +`WS-ART-001-02A3` - ArtifactStore v2 Local Clean Cut + +Merge intent: `.agent-loop/merge-intents/WS-ART-001-02A3.json` + +## Goal + +Replace ArtifactStore v1 atomically with the byte-only v2 contract, migrate +LocalStorage and the empty pre-production schema, install an immutable storage +namespace fence, and activate bounded scratch cleanup without compatibility +paths or later product behavior. + +## Human-Approved Intent + +- Initiative: `.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md` +- Chunk contract: `.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02A3-artifact-store-v2-local-clean-cut.md` + +LocalStorage is development/test only. AWS S3 through a later S3-compatible +adapter is the v0.1 hosted direction. Flow Node is separately deferred. + +## What Changed + +- Reduced ArtifactStore to `put`, `observe_put_result`, `open`, and `head`. +- Rebuilt LocalStorage with content-derived paths, exact replay, bounded reads, + exclusive crash-safe publication, private no-follow I/O, and sanitized errors. +- Added a deployment storage-namespace singleton and provider-replica binding. +- Made concurrent exact-replay finalization converge on one immutable replica + while recording a receipt for each upload item. +- Guaranteed cleanup always attempts digest-lock release after temp cleanup + failures without masking caller cancellation or sanitized provider errors. +- Removed provider retention/receipt semantics, v1 helpers, dormant Flow Node + configuration, and invented service identity. +- Added startup and Celery Beat stale-scratch cleanup without activating product + ingestion, verification, recovery, S3, review, or authorization actions. + +## Why It Changed + +The prior interface mixed byte storage with lifecycle and retention semantics +that Workstream must own. The clean cut makes external storage a narrow +immutable-byte capability and leaves PostgreSQL authoritative for Workstream +state, receipts, verification, and later recovery. + +## Design Chosen + +`CommittedArtifactSource` binds server-computed SHA-256, exact size, media type, +and second-pass bytes. LocalStorage publishes to a deterministic private path. +PostgreSQL transaction A fences the namespace and reserved upload item, provider +I/O runs outside the transaction, and transaction B atomically stores or reuses +content and replica facts, appends the item receipt, and records +`stored_pending_verification`. + +## Alternatives Rejected + +- Compatibility aliases or dual v1/v2 factories: rejected for this empty + pre-production clean cut. +- Provider-owned verify/retain/release receipts: rejected because they mix + Workstream lifecycle authority into the byte adapter. +- A fabricated artifact service principal: rejected because AUTH alone owns + service identities and action activation. +- S3, admission, verification, recovery, or product routes in this chunk: + rejected as separately owned follow-up work. + +## Scope Control + +The change stays within the approved artifact interface, adapter, schema, +orchestrator, cleanup wiring, tests, CI gates, and related loop/docs files. It +adds no S3 SDK, public route, Operator route, verification job, recovery attempt, +guide/task/submission cutover, product review decision, or authorization action. + +## Product Behavior + +- [x] No contributor, reviewer, contribution, compensation, reputation, or product review behavior is activated. +- [x] Provider acknowledgement remains non-bindable pending verification. + +## Acceptance Criteria Proof + +- [x] Four-operation provider-neutral v2 port and one typed factory path. +- [x] Immutable local publication, exact replay, range reads, bounded I/O, and sanitized errors. +- [x] V1 retention/provider-receipt/configuration paths removed without fallback. +- [x] Namespace singleton checked before provider I/O and finalization. +- [x] Conflicting concurrent first writers prove one provider winner and one + zero-provider-call loser after the namespace transaction commits. +- [x] Concurrent exact replay produces one replica and independent receipts. +- [x] Migration refuses existing and concurrently inserted v1 facts before DDL. +- [x] Populated v2 namespace state prevents destructive downgrade. +- [x] Cancellation persists `replay_required` without premature durable facts. +- [x] Startup and named periodic scratch cleanup are wired. +- [x] Changed subsystems meet the 90 percent coverage requirement. + +## Tests And Checks + +```text +268 prior complete ART-focused tests PASS +73 final filesystem/architecture/wiring tests PASS +59 final LocalStorage/conformance tests PASS +9 final critical real PostgreSQL orchestration cases PASS +5 migration-chain tests PASS, including 4 ArtifactStore v2 safety tests +ART changed scope coverage 93.31% +Configuration coverage 96.92% +app/main.py coverage 90.35% +Ruff PASS +Docstring coverage 92.0% +80 agent-gate tests PASS +Stale artifact/authorization/Workstream wording, Markdown links, and diff PASS +``` + +GitHub Backend CI remains authoritative for the isolated full suite, the 78 +percent repository floor, cumulative scoped 90 percent gates, and real API drill. + +## Test Delta + +V1 operation/retention vectors were replaced by v2 contract, LocalStorage, +orchestration, migration, cancellation, concurrency, architecture, cleanup, and +configuration tests. No tests were skipped or weakened. + +## CI Integrity + +- [x] Repository 78 percent floor retained. +- [x] Existing cumulative 90 percent scoped gates retained. +- [x] Worker and FastAPI startup gates remain fail closed. +- [x] No `continue-on-error`, conditional bypass, or package-script weakening. + +## Internal Reviewer Results + +Reviewed code SHA: `c8eccaafb6b4f0615b1b37049edfcb0368dd6fb2` + +All nine required tracks passed the exact repaired SHA with no remaining +finding. Earlier valid findings covered incomplete test-helper consolidation, +cross-initiative migration custody, and provider-operation ordering proof. All +were repaired, every review session was closed, and the final exact-SHA fanout +passed. Reviewer IDs and final results are recorded in the internal review +evidence. + +## External Review + +| Source | Status | Notes | +|---|---:|---| +| GitHub Actions | Pending | Full isolated suite and coverage run after publication. | +| CodeRabbit | Pending | External review begins after publication. | +| Human review | Pending | Only the user may approve merge. | + +## Remaining Risks + +- The migration intentionally refuses any populated v1 artifact table. +- LocalStorage is not a hosted production provider. +- Acknowledged bytes remain unusable until later verification/publication work. + +## Follow-Up Work + +`WS-ART-001-02B1` adds the S3-compatible adapter, MinIO proof, and AWS profile +only after this PR merges and the user explicitly starts it. + +## Human Review Focus + +- Is the provider boundary byte-only and provider-neutral? +- Does LocalStorage fail closed while always releasing owned resources? +- Do concurrent exact replays converge without losing per-item receipts? +- Does migration refuse every existing or concurrent v1 fact before DDL? +- Does acknowledgement remain non-bindable and free of invented AUTH identity? +- Did the chunk avoid product ingest and future-chunk activation? + +## Human Merge Ownership + +- [ ] I can explain what changed and why. +- [ ] I know what could break. +- [ ] I accept the remaining risks. +- [ ] GitHub CI and external review pass. +- [ ] The user explicitly approved this specific PR for merge. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DECISIONS.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DECISIONS.md index 1e7d59935..695ae1fdc 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DECISIONS.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DECISIONS.md @@ -287,17 +287,20 @@ migration, audit-history treatment, cutover ownership, and rollback proof. WS-REV, WS-CON, and the artifact-storage specification continue to own their resources and state transitions. -Migration custody is reconciled with merged main: AUTH-05A owns `0018`, AUTH-05B -solely owns `0019`, AUTH-06 uses `0020`, AUTH-07A action evidence uses `0021`, -AUTH-08 uses `0022`, and the AUTH-09A fixed-service identity foundation uses -`0023`. AUTH-09B uses `0024` for service-link verification timestamps. AUTH-10 -uses `0025`, AUTH-11 uses `0026`, AUTH-12 uses `0027`, AUTH-13 uses `0028`, -AUTH-14 uses `0029`, and AUTH-15 uses `0030` +At D15 acceptance, migration custody was recorded as follows: AUTH-05A owned +`0018`, AUTH-05B solely owned `0019`, AUTH-06 used `0020`, AUTH-07A action +evidence used `0021`, AUTH-08 used `0022`, and the AUTH-09A fixed-service +identity foundation used `0023`. AUTH-09B used `0024` for service-link +verification timestamps. AUTH-10 used `0025`, AUTH-11 used `0026`, AUTH-12 used +`0027`, AUTH-13 used `0028`, AUTH-14 used `0029`, and AUTH-15 used `0030` so every new protected surface receives typed/PostgreSQL ActionId evidence parity in its owning cutover. Feature-gated REV/ART additive registrations use the next trusted-main migration head when their complete contracts become executable; they do not reserve a number while blocked. +D29 supersedes only D15's future AUTH-10 through AUTH-15 migration allocation. +The merged historical ownership through AUTH-09B `0024` remains unchanged. + ## D16: Split AUTH-07 at the catalogue and executable-kernel boundary Status: accepted as a required L1 preimplementation repair on 2026-07-15. @@ -454,7 +457,7 @@ idempotency hashes the requested role; revoke derives the role from the locked grant. Same key/different role mismatches, new-key duplicate same-role issue is a stable audited conflict, and replay reauthorizes before disclosure. -AUTH-10 replaces current typed and PostgreSQL validators in migration `0025` +AUTH-10 replaces current typed and PostgreSQL validators in migration `0026` without editing `0018`, `0019`, or `0022`. It fails closed if obsolete combined or replacement evidence exists and refuses an unsafe downgrade rather than converting or deleting evidence. Only issued and revoked success events remain. @@ -571,8 +574,8 @@ PermissionId mapping, canonical target, principal class, candidates, guards, surface, and revalidation rule before runtime edits. Each owning migration updates current typed and PostgreSQL audit validation; no chunk may promise an active surface whose ActionId is absent. AUTH-11 therefore owns migration -`0026`, and later migration reservations shift through AUTH-15 as recorded in -D15 and D28. AUTH-16 aggregates proof; it does not discover or backfill missing +`0027`, and later migration reservations shift through AUTH-15 as recorded in +D29. AUTH-16 aggregates proof; it does not discover or backfill missing registrations. ## D28: Service provisioning records an unverified issuer binding @@ -593,9 +596,10 @@ Provisioning is not proof that the service presented a token. AUTH-09B migration service links, removes its implicit default, and requires human links to remain verified. Human first access writes database time explicitly. New service profiles and links keep `last_seen_at` and `last_verified_at` null until AUTH-09E -successfully verifies that exact service token. The migration allocation shifts -AUTH-10 through AUTH-15 to `0025` through `0030`; historical migrations remain -immutable. +successfully verifies that exact service token. The migration allocation then +shifted AUTH-10 through AUTH-15 to `0025` through `0030`; D29 supersedes only +that future allocation after ART claimed the next trusted-main head. Historical +migrations remain immutable. Both central AUTH and legacy actor dependencies reject service subjects before actor resolution or timestamp mutation until AUTH-09E. Provisioning a service @@ -614,3 +618,17 @@ invalidation direction because the new binding invalidates cached negative identity-absence projections. It does not claim that a service permission was previously or is now executable. AUTH-09E remains the only runtime service admission owner. + +## D29: ART `0025` shifts future AUTH migration custody + +Status: accepted cross-initiative reconciliation on 2026-07-17. + +AUTH-09B remains the immutable owner of migration `0024`. WS-ART-001-02A3 owns +migration `0025` for the ArtifactStore v2 clean cut. Future, inactive AUTH +reservations therefore shift exactly once: AUTH-10 owns `0026`, AUTH-11 owns +`0027`, AUTH-12 owns `0028`, AUTH-13 owns `0029`, AUTH-14 owns `0030`, and +AUTH-15 owns `0031`. + +This decision supersedes only the future migration-number allocations in D15, +D27, and D28. It does not modify any merged migration, action ownership, +authorization behavior, or feature activation boundary. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DISCOVERY.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DISCOVERY.md index 234d37d13..56130778e 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DISCOVERY.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/DISCOVERY.md @@ -25,8 +25,9 @@ and test ownership, and combined schema, provisioning, reads, lifecycle mutations, service admission, and concurrency. Required L1 review split it into 09A through 09E before runtime implementation. AUTH-09A takes migration `0023`; 09B requires forward migration `0024` so provisioned service links remain -unverified until a token is actually verified. AUTH-10 through AUTH-15 therefore -shift to `0025` through `0030`, without allocating migrations to the +unverified until a token is actually verified. ART owns `0025` for the +ArtifactStore v2 clean cut. AUTH-10 through AUTH-15 therefore shift to `0026` +through `0031`, without allocating migrations to the availability-neutral custody or PREP chunks. ## Current behavior @@ -302,7 +303,7 @@ need an independently reviewable contract and production-code budget. `replaced_grant_id`, replacement events, and replacement reasons. - Migration `0022_bootstrap_admin_grants.py` recreates current PostgreSQL audit and linked-idempotency validators with the same combined/replacement values. - Historical migrations remain immutable; AUTH-10 `0025` must replace current + Historical migrations remain immutable; AUTH-10 `0026` must replace current validators and fail closed on incompatible evidence. - The current kernel has request-scoped `require()` but no AUTH-first prepared mutation handle. Cross-module mutations therefore lack the merged lock, diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/PLAN.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/PLAN.md index 4e91ccd49..ac00f8e08 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/PLAN.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/PLAN.md @@ -85,8 +85,9 @@ project role or an AdminRoleGrant. Every cutover chunk enumerates its exact ActionIds before implementation and updates current typed/PostgreSQL audit mapping validation in its own migration. After AUTH-09A migration `0023`, AUTH-09B uses `0024` for service-link -verification timestamp semantics. Core reservations are AUTH-10 `0025`, AUTH-11 -`0026`, AUTH-12 `0027`, AUTH-13 `0028`, AUTH-14 `0029`, and AUTH-15 `0030`. +verification timestamp semantics and ART owns `0025` for the ArtifactStore v2 +clean cut. Core reservations are AUTH-10 `0026`, AUTH-11 `0027`, AUTH-12 +`0028`, AUTH-13 `0029`, AUTH-14 `0030`, and AUTH-15 `0031`. Blocked cross-initiative additions allocate the next trusted-main migration head only when their complete feature contracts become executable. No migration number is reserved or allocated while the work remains blocked. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/STATUS.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/STATUS.md index b3440c918..ba97afdeb 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/STATUS.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/STATUS.md @@ -170,8 +170,9 @@ AUTH-10 additionally owns the clean cut across the typed `ProjectRole`, audit, and idempotency contracts plus the current PostgreSQL validators recreated by migration `0022`, removing obsolete `ProjectRole.BOTH` and replacement evidence. AUTH-09B owns migration `0024` for service-link verification timestamp -semantics. AUTH-10 through AUTH-15 own shifted migrations `0025` through `0030` -for their action/evidence parity. +semantics, and ART owns `0025` for the ArtifactStore v2 clean cut. AUTH-10 +through AUTH-15 own shifted migrations `0026` through `0031` for their +action/evidence parity. AUTH-05A and CAT post-merge memory have no remaining blocker and are merged. The combined AUTH-05 contract diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-09B-controlled-service-provisioning.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-09B-controlled-service-provisioning.md index bc0544ce0..7d3394fde 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-09B-controlled-service-provisioning.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-09B-controlled-service-provisioning.md @@ -195,8 +195,8 @@ The exact state table is: The migration downgrade first refuses while any service link has a null verification timestamp, then restores non-null/default semantics. Historical -migrations remain immutable. AUTH-10 through AUTH-15 move to `0025` through -`0030` respectively. +migrations remain immutable. ART owns `0025` for the ArtifactStore v2 clean +cut. AUTH-10 through AUTH-15 move to `0026` through `0031` respectively. ## Acceptance criteria diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-10-project-role-grants.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-10-project-role-grants.md index 86b5a8165..1d72a8680 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-10-project-role-grants.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-10-project-role-grants.md @@ -45,7 +45,7 @@ backend/app/modules/projects/repository.py backend/app/api/router.py backend/app/db/models.py backend/app/modules/audit/** -backend/alembic/versions/0025_*.py +backend/alembic/versions/0026_*.py backend/tests/test_actors.py backend/tests/test_projects.py backend/tests/test_auth.py @@ -140,9 +140,9 @@ editing migrations `0018`, `0019`, or `0022` project deny, pagination/count concealment, minimal-field, rate-limit, and inactive/non-human exclusion tests; no UUID must be recovered from logs or direct database access. -- Migration `0025` enforces exact three-role checks, composite snapshot/grant +- Migration `0026` enforces exact three-role checks, composite snapshot/grant ownership, partial unique/supporting indexes, database-time fields, and - immutability. Migration `0025` refuses upgrade when obsolete combined or + immutability. Migration `0026` refuses upgrade when obsolete combined or replacement evidence exists and never converts or deletes those rows. It replaces current audit/idempotency validators without editing historical migrations and refuses an unsafe downgrade without mutating evidence. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-11-project-read-cutover.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-11-project-read-cutover.md index 933814e38..15fa9711d 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-11-project-read-cutover.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-11-project-read-cutover.md @@ -3,7 +3,7 @@ ## Status Proposed and inactive. The exact project read/list ActionId inventory and -migration `0026` mapping delta must be added before implementation review. +migration `0027` mapping delta must be added before implementation review. ## Parent initiative @@ -43,7 +43,7 @@ backend/app/modules/projects/repository.py backend/app/modules/projects/schemas.py backend/app/modules/authorization/** backend/app/modules/audit/** -backend/alembic/versions/0026_*.py +backend/alembic/versions/0027_*.py backend/app/api/deps/auth.py backend/tests/test_projects.py backend/tests/test_auth.py @@ -91,7 +91,7 @@ issuer-role fallback or authorization pagination after unfiltered counts read/list surface migrated here has exactly one active `ActionId` declaration. - Before runtime edits, the contract enumerates every new ActionId, existing PermissionId mapping, canonical target, principal class, facts, guards, and - surface. Migration `0026` updates typed/PostgreSQL action-evidence parity and + surface. Migration `0027` updates typed/PostgreSQL action-evidence parity and proves prior-head upgrade, downgrade, re-upgrade, and fresh replay. - ProjectRepository remains the canonical project/guide/source persistence query owner and returns domain records. The project application service or a diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-12-project-mutation-cutover.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-12-project-mutation-cutover.md index c23025f96..c7583ccb1 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-12-project-mutation-cutover.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-12-project-mutation-cutover.md @@ -2,7 +2,7 @@ ## Status -Proposed and inactive. Exact ActionIds and migration `0027` mapping/provenance +Proposed and inactive. Exact ActionIds and migration `0028` mapping/provenance delta must be enumerated before implementation; AUTH-PREP is required. ## Parent initiative @@ -42,7 +42,7 @@ backend/app/modules/projects/** backend/app/modules/authorization/** backend/app/api/deps/auth.py backend/app/workers/project_setup.py -backend/alembic/versions/0027_*.py +backend/alembic/versions/0028_*.py backend/tests/test_projects.py backend/tests/test_auth.py backend/tests/test_alembic.py @@ -89,7 +89,7 @@ unscoped project-manager access or token role fallback declaration. - Approval provenance records matched local grant/actor/scope while preserving historical bootstrap provenance. -- Migration `0027` adds exact action-evidence parity plus matched local +- Migration `0028` adds exact action-evidence parity plus matched local grant/scope provenance and ownership constraints to project policy approval records without rewriting historical bootstrap values; prior-head upgrade, downgrade, and re-upgrade preserve diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-13-task-assignment-cutover.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-13-task-assignment-cutover.md index 3ab3c631f..9b41f90c4 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-13-task-assignment-cutover.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-13-task-assignment-cutover.md @@ -2,7 +2,7 @@ ## Status -Proposed and inactive. Exact ActionIds and migration `0028` must be enumerated +Proposed and inactive. Exact ActionIds and migration `0029` must be enumerated before implementation; AUTH-PREP is required for mutations. ## Parent initiative @@ -44,7 +44,7 @@ backend/app/modules/tasks/repository.py backend/app/modules/tasks/schemas.py backend/app/modules/tasks/models.py backend/app/modules/tasks/lifecycle.py -backend/alembic/versions/0028_*.py +backend/alembic/versions/0029_*.py backend/app/modules/authorization/** backend/app/modules/audit/** backend/app/api/deps/auth.py @@ -97,7 +97,7 @@ token role or legacy active-worker-profile fallback `operations.task.start_override` PermissionId/ActionId typed and PostgreSQL parity as planned metadata. This chunk promotes the action only with its task resource composer, Operator candidate, guards, surface declaration, reason, - evidence, and behavior tests. Migration `0028` owns task/assignment, + evidence, and behavior tests. Migration `0029` owns task/assignment, Contributor-field, and exact new ActionId evidence parity; it changes no PermissionId mapping. - Operator `operations.status.read` exposes a read-only cross-project task-queue @@ -138,7 +138,7 @@ token role or legacy active-worker-profile fallback remains bounded only because chunk 14 still owns the final submission compatibility consumer; task queue/claim/start no longer depend on it. - The assignment persistence column, model/schema/service fields, response - contract, and new audit payload keys use `contributor_id`. Migration `0028` + contract, and new audit payload keys use `contributor_id`. Migration `0029` preserves every existing assignment owner, supports downgrade, and removes the legacy storage name without exposing a public compatibility alias. - Full backend suite and API contract drill pass. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-14-submission-checker-cutover.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-14-submission-checker-cutover.md index d5379e501..c1030d0b5 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-14-submission-checker-cutover.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-14-submission-checker-cutover.md @@ -2,7 +2,7 @@ ## Status -Proposed and inactive. Exact ActionIds and migration `0029` must be enumerated +Proposed and inactive. Exact ActionIds and migration `0030` must be enumerated before implementation; AUTH-PREP is required for mutations. ## Parent initiative @@ -46,7 +46,7 @@ backend/app/modules/checkers/** backend/app/modules/projects/schemas.py backend/app/modules/projects/service.py backend/app/adapters/project_agents/openai_agent_sdk.py -backend/alembic/versions/0029_*.py +backend/alembic/versions/0030_*.py backend/app/modules/authorization/** backend/app/modules/audit/** backend/app/api/deps/auth.py @@ -101,7 +101,7 @@ legacy active-worker-profile or workflow-eligibility compatibility fallback PermissionId/ActionId typed and PostgreSQL parity as planned metadata. This chunk promotes each action only with its feature resource composer, Operator candidate, guards, surface declaration, reason, evidence, and behavior tests. - Migration `0029` owns submission/checker Contributor-field schema changes and + Migration `0030` owns submission/checker Contributor-field schema changes and exact new ActionId evidence parity; it changes no PermissionId mapping. - Contributor reads preserve ownership, hidden-result redaction, and concealed not-found behavior. @@ -121,7 +121,7 @@ legacy active-worker-profile or workflow-eligibility compatibility fallback `contributor_suggested_fix`, `contributor_evidence_refs`, and `contributor_visible` across persistence, models, schemas, services, runner contracts, audit payloads, and tests. Submission-policy JSON and derivation - contracts use `contributor_facing_fix`. Migration `0029` preserves all values, + contracts use `contributor_facing_fix`. Migration `0030` preserves all values, supports downgrade, and removes legacy storage/property names without public API aliases. - With the final consumer removed, the legacy `/api/v1/workers/me/profile` diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-15-worker-authority-removal.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-15-worker-authority-removal.md index 1e79fb6fd..0d93f5f50 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-15-worker-authority-removal.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-15-worker-authority-removal.md @@ -3,7 +3,7 @@ ## Status Proposed and inactive. Exact remaining command ActionIds, service identities, -and migration `0030` parity must be enumerated before implementation. +and migration `0031` parity must be enumerated before implementation. ## Parent initiative @@ -51,7 +51,7 @@ backend/app/core/permissions.py backend/app/schemas/auth.py backend/app/api/deps/auth.py backend/app/modules/audit/** -backend/alembic/versions/0030_*.py +backend/alembic/versions/0031_*.py backend/scripts/api_contract_e2e.py backend/tests/test_projects.py backend/tests/test_tasks.py @@ -112,7 +112,7 @@ review/contribution/compensation implementation - Scanner has regression tests and runs in CI. - Scanner regression includes a known-bad fixture for each forbidden authority pattern and proves the gate fails rather than silently allowlisting it. -- Migration `0030` adds exact remaining command ActionId evidence parity and any +- Migration `0031` adds exact remaining command ActionId evidence parity and any approved fixed-service constraints; it changes no existing permission mapping and proves prior-head/fresh upgrade, downgrade, and re-upgrade. - Full backend suite and API contract drill pass. diff --git a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-CAT-action-resource-catalogue-reconciliation.md b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-CAT-action-resource-catalogue-reconciliation.md index 077eaee4e..e9af2bae6 100644 --- a/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-CAT-action-resource-catalogue-reconciliation.md +++ b/.agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/chunks/WS-AUTH-001-CAT-action-resource-catalogue-reconciliation.md @@ -86,8 +86,9 @@ starting AUTH-05B runtime implementation in this chunk transition ownership. - AUTH-07 through AUTH-16 retain at least 90 percent authorization-subsystem coverage; the repository-wide 78 percent baseline remains unchanged. -- AUTH-05A retains migration `0018`, AUTH-05B solely owns `0019`, and the later - AUTH schema chunks use the non-conflicting `0020` through `0026` sequence. +- AUTH-05A retains migration `0018`, AUTH-05B solely owns `0019`, and merged + AUTH schema chunks through AUTH-09B use `0020` through `0024`. Future + migration custody follows the current AUTH plan and trusted main. - The rejected root proposal is absent from the working tree and patch. ## Verification commands diff --git a/.agent-loop/merge-intents/WS-ART-001-02A3.json b/.agent-loop/merge-intents/WS-ART-001-02A3.json new file mode 100644 index 000000000..423125520 --- /dev/null +++ b/.agent-loop/merge-intents/WS-ART-001-02A3.json @@ -0,0 +1,9 @@ +{ + "chunk_id": "WS-ART-001-02A3", + "chunk_title": "ArtifactStore v2 Local Clean Cut", + "initiative_id": "WS-ART-001", + "next_chunk_id": "WS-ART-001-02B1", + "next_chunk_title": "S3-Compatible MinIO And AWS", + "next_requires_explicit_start": true, + "schema_version": 2 +} diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml index afeecaa54..83b071e2f 100644 --- a/.github/workflows/backend.yml +++ b/.github/workflows/backend.yml @@ -72,7 +72,7 @@ jobs: working-directory: backend run: >- coverage report - --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifacts.py,app/modules/artifacts/*' + --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 @@ -92,6 +92,22 @@ jobs: --precision=2 --fail-under=90 + - name: Artifact worker wiring coverage + working-directory: backend + run: >- + coverage report + --include='app/workers/*' + --precision=2 + --fail-under=90 + + - name: FastAPI startup wiring coverage + working-directory: backend + run: >- + coverage report + --include='app/main.py' + --precision=2 + --fail-under=90 + - name: Actor subsystem coverage working-directory: backend run: >- diff --git a/backend/alembic/versions/0025_artifact_store_v2_clean_cut.py b/backend/alembic/versions/0025_artifact_store_v2_clean_cut.py new file mode 100644 index 000000000..77946c69f --- /dev/null +++ b/backend/alembic/versions/0025_artifact_store_v2_clean_cut.py @@ -0,0 +1,422 @@ +"""replace artifact provider v1 with byte-only v2 + +Revision ID: 0025_artifact_store_v2 +Revises: 0024_service_link_verification +Create Date: 2026-07-16 +""" + +from __future__ import annotations + +from alembic import op +import sqlalchemy as sa + + +revision = "0025_artifact_store_v2" +down_revision = "0024_service_link_verification" +branch_labels = depends_on = None + +_ARTIFACT_TABLES = ( + "artifact_operation_receipts", + "artifact_replicas", + "artifact_bindings", + "artifact_upload_items", + "artifact_contents", + "artifact_upload_sessions", +) + + +def _refuse_populated_artifact_rows(*, include_namespace: bool) -> None: + """Refuse to invent v2 provenance or downgrade durable v2 facts.""" + connection = op.get_bind() + tables = _ARTIFACT_TABLES + (("artifact_storage_namespaces",) if include_namespace else ()) + connection.execute( + sa.text(f"lock table {', '.join(tables)} in access exclusive mode") + ) + populated = [ + table_name + for table_name in tables + if connection.execute( + sa.text(f"select exists(select 1 from {table_name})") + ).scalar() + ] + if populated: + raise RuntimeError("artifact storage clean cut requires empty pre-production tables") + + +def upgrade() -> None: + """Install v2 only when no v1 artifact fact requires fabrication.""" + _refuse_populated_artifact_rows(include_namespace=False) + + op.create_table( + "artifact_storage_namespaces", + sa.Column("id", sa.String(20), primary_key=True), + sa.Column("backend", sa.String(50), nullable=False), + sa.Column("adapter", sa.String(50), nullable=False), + sa.Column("provider_profile", sa.String(100), nullable=False), + sa.Column("namespace_descriptor", sa.JSON(), nullable=False), + sa.Column("namespace_fingerprint", sa.String(71), nullable=False), + sa.Column( + "created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False + ), + sa.CheckConstraint("id = 'primary'", name="singleton_id"), + sa.CheckConstraint( + "namespace_fingerprint ~ '^sha256:[0-9a-f]{64}$'", name="fingerprint_shape" + ), + sa.UniqueConstraint( + "namespace_fingerprint", name="uq_artifact_storage_namespace_fingerprint" + ), + ) + op.execute( + """ + create trigger trg_artifact_storage_namespaces_immutable + before update or delete on artifact_storage_namespaces + for each row execute function reject_artifact_fact_mutation() + """ + ) + + op.drop_constraint( + op.f("ck_artifact_upload_items_ready_result_required"), + "artifact_upload_items", + type_="check", + ) + op.drop_constraint( + op.f("ck_artifact_upload_items_state"), "artifact_upload_items", type_="check" + ) + op.alter_column( + "artifact_upload_items", + "provider_operation_reference", + new_column_name="provider_object_ref", + type_=sa.String(1024), + existing_type=sa.String(200), + ) + op.create_check_constraint( + "state", + "artifact_upload_items", + "state in ('reserved', 'uploading', 'replay_required', " + "'stored_pending_verification', 'ready', 'failed', 'cancelled')", + ) + op.create_check_constraint( + "stored_result_required", + "artifact_upload_items", + "((state in ('stored_pending_verification', 'ready')) and " + "content_id is not null and provider_object_ref is not null) or " + "((state not in ('stored_pending_verification', 'ready')) and " + "content_id is null and provider_object_ref is null)", + ) + + op.drop_constraint( + op.f("uq_artifact_replica_provider"), "artifact_replicas", type_="unique" + ) + for constraint in ( + "ck_artifact_replicas_verification_state", + "ck_artifact_replicas_retention_state", + "ck_artifact_replicas_availability_state", + "ck_artifact_replicas_integrity_state", + ): + op.drop_constraint(op.f(constraint), "artifact_replicas", type_="check") + op.drop_column("artifact_replicas", "retention_state") + op.drop_column("artifact_replicas", "provider_manifest_id") + op.alter_column( + "artifact_replicas", + "provider_artifact_id", + new_column_name="provider_object_ref", + type_=sa.String(1024), + existing_type=sa.String(200), + ) + op.add_column( + "artifact_replicas", sa.Column("storage_namespace_id", sa.String(20), nullable=False) + ) + op.add_column( + "artifact_replicas", sa.Column("namespace_fingerprint", sa.String(71), nullable=False) + ) + op.add_column( + "artifact_replicas", sa.Column("provider_profile", sa.String(100), nullable=False) + ) + op.create_foreign_key( + op.f("fk_artifact_replicas_storage_namespace_id_artifact_storage_namespaces"), + "artifact_replicas", + "artifact_storage_namespaces", + ["storage_namespace_id"], + ["id"], + ondelete="RESTRICT", + ) + op.create_index( + op.f("ix_artifact_replicas_storage_namespace_id"), + "artifact_replicas", + ["storage_namespace_id"], + ) + op.create_unique_constraint( + op.f("uq_artifact_replica_provider_object"), + "artifact_replicas", + ["storage_namespace_id", "provider_object_ref"], + ) + op.create_check_constraint( + "verification_state", + "artifact_replicas", + "verification_state in ('pending', 'verified', 'missing', 'integrity_mismatch')", + ) + op.create_check_constraint( + "availability_state", + "artifact_replicas", + "availability_state in ('unknown', 'available', 'unavailable')", + ) + op.create_check_constraint( + "integrity_state", + "artifact_replicas", + "integrity_state in ('unknown', 'valid', 'invalid')", + ) + op.create_check_constraint( + "fingerprint_shape", + "artifact_replicas", + "namespace_fingerprint ~ '^sha256:[0-9a-f]{64}$'", + ) + + op.execute("drop trigger trg_artifact_operation_receipts_immutable on artifact_operation_receipts") + op.drop_table("artifact_operation_receipts") + _create_v2_receipts() + + +def _create_v2_receipts() -> None: + op.create_table( + "artifact_operation_receipts", + sa.Column("id", sa.String(36), primary_key=True), + sa.Column("upload_item_id", sa.String(36), nullable=False), + sa.Column("replica_id", sa.String(36), nullable=False), + sa.Column("operation", sa.String(30), nullable=False), + sa.Column("idempotency_key", sa.String(200), nullable=False), + sa.Column("request_digest", sa.String(71), nullable=False), + sa.Column("provider_object_ref", sa.String(1024), nullable=False), + sa.Column("replayed", sa.Boolean(), nullable=False), + sa.Column("outcome", sa.String(30), nullable=False), + sa.Column("attempt_number", sa.Integer(), nullable=False), + sa.Column("correlation_id", sa.String(100), nullable=False), + sa.Column("details", sa.JSON(), nullable=False), + sa.Column( + "created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False + ), + sa.ForeignKeyConstraint( + ["upload_item_id"], ["artifact_upload_items.id"], ondelete="RESTRICT" + ), + sa.ForeignKeyConstraint(["replica_id"], ["artifact_replicas.id"], ondelete="RESTRICT"), + sa.UniqueConstraint("upload_item_id", name="uq_artifact_receipt_upload_item"), + sa.CheckConstraint( + "request_digest ~ '^sha256:[0-9a-f]{64}$'", name="request_digest_shape" + ), + sa.CheckConstraint("operation = 'put'", name="operation"), + sa.CheckConstraint("outcome = 'stored_pending_verification'", name="outcome"), + sa.CheckConstraint("attempt_number > 0", name="attempt_positive"), + ) + op.create_index( + "ix_artifact_operation_receipts_upload_item_id", + "artifact_operation_receipts", + ["upload_item_id"], + ) + op.create_index( + "ix_artifact_operation_receipts_replica_id", + "artifact_operation_receipts", + ["replica_id"], + ) + op.execute( + """ + create trigger trg_artifact_operation_receipts_immutable + before update or delete on artifact_operation_receipts + for each row execute function reject_artifact_fact_mutation() + """ + ) + + +def downgrade() -> None: + """Restore the empty v1 shape without converting any v2 fact.""" + _refuse_populated_artifact_rows(include_namespace=True) + + op.execute("drop trigger trg_artifact_operation_receipts_immutable on artifact_operation_receipts") + op.drop_table("artifact_operation_receipts") + + for constraint in ( + "ck_artifact_replicas_verification_state", + "ck_artifact_replicas_availability_state", + "ck_artifact_replicas_integrity_state", + "ck_artifact_replicas_fingerprint_shape", + ): + op.drop_constraint(op.f(constraint), "artifact_replicas", type_="check") + op.drop_constraint( + op.f("uq_artifact_replica_provider_object"), "artifact_replicas", type_="unique" + ) + op.drop_index( + op.f("ix_artifact_replicas_storage_namespace_id"), table_name="artifact_replicas" + ) + op.drop_constraint( + op.f("fk_artifact_replicas_storage_namespace_id_artifact_storage_namespaces"), + "artifact_replicas", + type_="foreignkey", + ) + op.drop_column("artifact_replicas", "provider_profile") + op.drop_column("artifact_replicas", "namespace_fingerprint") + op.drop_column("artifact_replicas", "storage_namespace_id") + op.alter_column( + "artifact_replicas", + "provider_object_ref", + new_column_name="provider_artifact_id", + type_=sa.String(200), + existing_type=sa.String(1024), + ) + op.add_column( + "artifact_replicas", sa.Column("provider_manifest_id", sa.String(200), nullable=True) + ) + op.add_column( + "artifact_replicas", sa.Column("retention_state", sa.String(30), nullable=False) + ) + op.create_unique_constraint( + "uq_artifact_replica_provider", + "artifact_replicas", + ["adapter", "provider_artifact_id"], + ) + op.create_check_constraint( + "verification_state", + "artifact_replicas", + "verification_state in ('pending', 'verified', 'failed')", + ) + op.create_check_constraint( + "retention_state", + "artifact_replicas", + "retention_state in ('unretained', 'retained', 'released')", + ) + op.create_check_constraint( + "availability_state", + "artifact_replicas", + "availability_state in ('available', 'unavailable', 'missing')", + ) + op.create_check_constraint( + "integrity_state", + "artifact_replicas", + "integrity_state in ('unknown', 'valid', 'quarantined')", + ) + + op.drop_constraint( + op.f("ck_artifact_upload_items_stored_result_required"), + "artifact_upload_items", + type_="check", + ) + op.drop_constraint( + op.f("ck_artifact_upload_items_state"), + "artifact_upload_items", + type_="check", + ) + op.alter_column( + "artifact_upload_items", + "provider_object_ref", + new_column_name="provider_operation_reference", + type_=sa.String(200), + existing_type=sa.String(1024), + ) + op.create_check_constraint( + "state", + "artifact_upload_items", + "state in ('reserved', 'uploading', 'provider_committed', " + "'replay_required', 'ready', 'failed', 'cancelled')", + ) + op.create_check_constraint( + "ready_result_required", + "artifact_upload_items", + "(state = 'ready') = " + "(content_id is not null and provider_operation_reference is not null)", + ) + + op.execute("drop trigger trg_artifact_storage_namespaces_immutable on artifact_storage_namespaces") + op.drop_table("artifact_storage_namespaces") + _create_v1_receipts() + + +def _create_v1_receipts() -> None: + """Restore the exact empty receipt shape expected by migration 0016.""" + op.create_table( + "artifact_operation_receipts", + sa.Column("id", sa.String(36), primary_key=True), + sa.Column("upload_item_id", sa.String(36), nullable=True), + sa.Column("replica_id", sa.String(36), nullable=True), + sa.Column("adapter", sa.String(50), nullable=False), + sa.Column("service_principal", sa.String(200), nullable=False), + sa.Column("operation", sa.String(30), nullable=False), + sa.Column("idempotency_key", sa.String(200), nullable=False), + sa.Column("request_digest", sa.String(71), nullable=False), + sa.Column("response_digest", sa.String(71), nullable=False), + sa.Column("provider_receipt_id", sa.String(200), nullable=False), + sa.Column("provider_operation_reference", sa.String(200), nullable=False), + sa.Column("outcome", sa.String(30), nullable=False), + sa.Column("attempt_number", sa.Integer(), nullable=False), + sa.Column("correlation_id", sa.String(100), nullable=False), + sa.Column("retention_reference", sa.String(200), nullable=True), + sa.Column("retention_class", sa.String(100), nullable=True), + sa.Column("retention_owner", sa.String(200), nullable=True), + sa.Column("provider_recorded_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("details", sa.JSON(), nullable=False), + sa.Column( + "created_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False + ), + sa.ForeignKeyConstraint( + ["upload_item_id"], ["artifact_upload_items.id"], ondelete="RESTRICT" + ), + sa.ForeignKeyConstraint(["replica_id"], ["artifact_replicas.id"], ondelete="RESTRICT"), + sa.UniqueConstraint( + "adapter", + "service_principal", + "operation", + "idempotency_key", + name="uq_artifact_receipt_operation", + ), + sa.CheckConstraint( + "request_digest ~ '^sha256:[0-9a-f]{64}$'", name="request_digest_shape" + ), + sa.CheckConstraint( + "response_digest ~ '^sha256:[0-9a-f]{64}$'", name="response_digest_shape" + ), + sa.CheckConstraint( + "operation in ('store', 'verify', 'retain', 'release')", name="operation" + ), + sa.CheckConstraint( + "outcome in ('stored', 'verified', 'retained', 'released')", name="outcome" + ), + sa.CheckConstraint( + "(operation = 'store' and outcome = 'stored') or " + "(operation = 'verify' and outcome = 'verified') or " + "(operation = 'retain' and outcome = 'retained') or " + "(operation = 'release' and outcome = 'released')", + name="operation_outcome", + ), + sa.CheckConstraint("attempt_number > 0", name="attempt_positive"), + sa.CheckConstraint( + "operation != 'retain' or " + "(retention_reference is not null and retention_class is not null " + "and retention_owner is not null)", + name="retain_fields", + ), + sa.CheckConstraint( + "operation != 'release' or " + "(retention_reference is not null and retention_class is not null " + "and retention_owner is not null)", + name="release_reference", + ), + sa.CheckConstraint( + "operation in ('retain', 'release') or " + "(retention_reference is null and retention_class is null " + "and retention_owner is null)", + name="non_retention_fields_empty", + ), + ) + op.create_index( + "ix_artifact_operation_receipts_upload_item_id", + "artifact_operation_receipts", + ["upload_item_id"], + ) + op.create_index( + "ix_artifact_operation_receipts_replica_id", + "artifact_operation_receipts", + ["replica_id"], + ) + op.execute( + """ + create trigger trg_artifact_operation_receipts_immutable + before update or delete on artifact_operation_receipts + for each row execute function reject_artifact_fact_mutation() + """ + ) diff --git a/backend/app/adapters/artifacts/__init__.py b/backend/app/adapters/artifacts/__init__.py index bc1863355..1805b7d1c 100644 --- a/backend/app/adapters/artifacts/__init__.py +++ b/backend/app/adapters/artifacts/__init__.py @@ -1,33 +1,83 @@ -"""Artifact-store adapter resolution.""" +"""Artifact-store composition and shared scratch construction.""" from __future__ import annotations from app.core.config import Settings -from app.interfaces.artifacts import ArtifactConfigurationError, ArtifactStore +from app.interfaces.artifacts import ( + ARTIFACT_STORE_CAPABILITY_KEY, + ArtifactConfigurationError, + ArtifactStoreBootstrap, +) +from app.interfaces.external_services import ExternalServiceAdapterFactory +from app.modules.artifacts.preparation import ( + ArtifactPreparationLimits, + ArtifactScratchManager, +) -def resolve_artifact_store(settings: Settings) -> ArtifactStore: - """Resolve the configured artifact provider without implying future support. +def create_artifact_store_bootstrap(settings: Settings) -> ArtifactStoreBootstrap: + """Construct the selected store bootstrap through one typed factory. Args: settings: Validated application settings. Returns: - Configured artifact store. + Non-mutating configured artifact store bootstrap. Raises: - ArtifactConfigurationError: If storage is disabled or not implemented. + ExternalServiceConfigurationError: If the provider is not registered. """ - if settings.artifact_store_backend == "local": - from app.adapters.artifacts.local import LocalStorageAdapter + from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap + factory = ExternalServiceAdapterFactory[ArtifactStoreBootstrap]( + ARTIFACT_STORE_CAPABILITY_KEY + ) + + def create_local_store() -> ArtifactStoreBootstrap: + """Pin the configured development-only LocalStorage provider root.""" if settings.artifact_local_root is None: raise ArtifactConfigurationError("local artifact root is not configured") - return LocalStorageAdapter( - root=settings.artifact_local_root, - buffer_bytes=settings.artifact_stream_buffer_bytes, - lock_timeout_seconds=settings.artifact_operation_lock_timeout_seconds, + return LocalStorageBootstrap( + LocalStorageAdapter( + root=settings.artifact_local_root, + buffer_bytes=settings.artifact_stream_buffer_bytes, + lock_timeout_seconds=settings.artifact_operation_lock_timeout_seconds, + ) ) - if settings.artifact_store_backend == "flow_node": - raise ArtifactConfigurationError("configured artifact adapter is not available") - raise ArtifactConfigurationError("artifact storage is disabled") + + factory.register("local", create_local_store) + return factory.create(settings.artifact_store_backend) + + +def artifact_preparation_limits(settings: Settings) -> ArtifactPreparationLimits: + """Map validated settings to the one process-independent scratch contract.""" + return ArtifactPreparationLimits( + aggregate_reserved_bytes=settings.artifact_scratch_aggregate_reserved_bytes, + maximum_files=settings.artifact_scratch_maximum_files, + maximum_concurrency=settings.artifact_scratch_maximum_concurrency, + minimum_free_bytes=settings.artifact_scratch_minimum_free_bytes, + reservation_ttl_seconds=settings.artifact_scratch_reservation_ttl_seconds, + total_deadline_seconds=settings.artifact_preparation_total_deadline_seconds, + cleanup_margin_seconds=settings.artifact_scratch_cleanup_margin_seconds, + stream_buffer_bytes=settings.artifact_stream_buffer_bytes, + maximum_source_bytes=settings.artifact_maximum_bytes, + ) + + +def create_artifact_scratch_manager(settings: Settings) -> ArtifactScratchManager: + """Construct a scratch manager from the canonical settings mapping.""" + if settings.artifact_scratch_root is None: + raise ArtifactConfigurationError("artifact scratch root is not configured") + return ArtifactScratchManager( + root=settings.artifact_scratch_root, + limits=artifact_preparation_limits(settings), + ) + + +async def cleanup_stale_artifact_scratch(settings: Settings) -> int: + """Run one database-independent stale cleanup with shared construction.""" + manager = create_artifact_scratch_manager(settings) + try: + return await manager.cleanup_stale() + finally: + manager.close() diff --git a/backend/app/adapters/artifacts/local.py b/backend/app/adapters/artifacts/local.py index 5629b7481..638780ae1 100644 --- a/backend/app/adapters/artifacts/local.py +++ b/backend/app/adapters/artifacts/local.py @@ -1,68 +1,98 @@ -"""Private development artifact storage with provider-contract parity.""" +"""Private local implementation of the immutable ArtifactStore v2 port.""" from __future__ import annotations import asyncio -from collections.abc import AsyncIterable, AsyncIterator +from collections.abc import AsyncIterator from contextlib import contextmanager -from datetime import UTC, datetime +import errno import fcntl import hashlib -import json import math import os from pathlib import Path import re -import stat as stat_module -import time -from typing import Any, BinaryIO, Iterator +import stat +from typing import Any, Iterator from uuid import uuid4 from app.core.cancellation import ( await_cancellation_resistant, run_blocking_cancellation_resistant, ) -from app.core.hashing import canonical_json_hash from app.core.file_locks import acquire_exclusive_file_lock from app.interfaces.artifacts import ( + ARTIFACT_STORE_CAPABILITY_KEY, ArtifactByteRange, - ArtifactIdempotencyMismatchError, + ArtifactConfigurationError, ArtifactInputMismatchError, ArtifactIntegrityError, ArtifactLimitExceededError, - ArtifactMalformedRequestError, - ArtifactNotFoundError, - ArtifactOperation, - ArtifactOperationResult, - ArtifactRetentionConflictError, - ArtifactStatus, + ArtifactObjectHead, + ArtifactObjectMissingError, + ArtifactOperationConflictError, + ArtifactPutObservation, + ArtifactPutResult, + ArtifactRangeInvalidError, + ArtifactStoreError, + ArtifactStoreNamespaceClaim, + ArtifactStoreNamespaceIdentity, ArtifactStoreUnavailableError, - AvailabilityState, - IdempotencyIdentity, - IntegrityState, - OperationReceipt, - ReceiptDetail, - ReceiptOutcome, - RetentionState, - RetentionSummary, - StoreArtifactRequest, - StoredArtifact, - VerificationState, - canonical_release_request_digest, - canonical_retain_request_digest, - canonical_store_request_digest, - canonical_verify_request_digest, + artifact_store_namespace_material, ) +from app.interfaces.external_services import ExternalServiceAdapterIdentity +from app.core.hashing import canonical_json_hash +from app.modules.artifacts.preparation import HARD_MAXIMUM_ARTIFACT_BYTES +from app.modules.artifacts.sources import ArtifactCommitment, CommittedArtifactSource -_OPAQUE_ID = re.compile(r"^art_[0-9a-f]{32}$") -_DIGEST = re.compile(r"^sha256:[0-9a-f]{64}$") -_OPERATION_REF = re.compile(r"^op_[0-9a-f]{32}$") +_IDENTITY = ExternalServiceAdapterIdentity(ARTIFACT_STORE_CAPABILITY_KEY, "local") +_LAYOUT_MARKER = ".workstream-artifact-store-v2" +_LAYOUT_MARKER_TEMPORARY = ".workstream-artifact-store-v2.initializing" +_LAYOUT_MARKER_BYTES = b"workstream-artifact-store-v2\n" +_LAYOUT_ENTRY_NAMES = frozenset( + {_LAYOUT_MARKER, _LAYOUT_MARKER_TEMPORARY, "locks", "objects", "tmp"} +) +_PROVIDER_OBJECT_REF = re.compile(r"^sha256/([0-9a-f]{2})/([0-9a-f]{62})$") +_DIGEST_PREFIX_NAME = re.compile(r"^[0-9a-f]{2}$") +_DIGEST_OBJECT_NAME = re.compile(r"^[0-9a-f]{62}$") +_TEMPORARY_NAME = re.compile(r"^\.put\.[0-9a-f]{32}\.tmp$") +_LOCK_NAME = re.compile(r"^[0-9a-f]{64}\.lock$") -class LocalStorageAdapter: - """Filesystem-backed artifact provider for development and CI only.""" - adapter_name = "local" +class LocalStorageBootstrap: + """Composition-only namespace lifecycle for one pinned local store.""" + + def __init__(self, adapter: LocalStorageAdapter) -> None: + """Own one unopened-layout adapter created by the composition root.""" + if type(adapter) is not LocalStorageAdapter: + raise ValueError("local artifact bootstrap adapter is invalid") + self._adapter = adapter + + @property + def identity(self) -> ExternalServiceAdapterIdentity: + """Return the canonical artifact-store/local adapter identity.""" + return self._adapter.identity + + @property + def namespace_identity(self) -> ArtifactStoreNamespaceIdentity: + """Return the namespace identity of the pinned local root.""" + return self._adapter._namespace_identity + + def initialize_after_namespace_claim( + self, + claim: ArtifactStoreNamespaceClaim, + ) -> LocalStorageAdapter: + """Return the byte store only after exact namespace admission.""" + return self._adapter._initialize_after_namespace_claim(claim) + + def close(self) -> None: + """Release the pinned local store resources.""" + self._adapter.close() + + +class LocalStorageAdapter: + """Filesystem-backed immutable byte provider for development and tests.""" def __init__( self, @@ -71,17 +101,8 @@ def __init__( buffer_bytes: int = 1024 * 1024, lock_timeout_seconds: float = 1800.0, ) -> None: - """Initialize a private storage root. - - Args: - root: Dedicated non-symlink storage directory. - buffer_bytes: Maximum bytes used by one filesystem read or write. - lock_timeout_seconds: Maximum wait for one cross-process operation lock. - - Raises: - ValueError: If the root or buffer is unsafe. - """ - if buffer_bytes <= 0 or buffer_bytes > 1024 * 1024: + """Pin one pre-provisioned private root without mutating its layout.""" + if type(buffer_bytes) is not int or not 1 <= buffer_bytes <= 1024 * 1024: raise ValueError("artifact buffer must be between 1 byte and 1 MiB") if ( isinstance(lock_timeout_seconds, bool) @@ -90,1319 +111,834 @@ def __init__( or lock_timeout_seconds <= 0 ): raise ValueError("artifact lock timeout is invalid") - self._root = root.resolve(strict=False) - self._buffer_bytes = buffer_bytes - self._lock_timeout_seconds = lock_timeout_seconds - self._initialize_root(root) - self._root_fd = os.open( - self._root, - os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0), - ) - self._cleanup_abandoned_temporary_files() - def close(self) -> None: - """Release the pinned local storage root descriptor.""" - descriptor = getattr(self, "_root_fd", -1) - if descriptor >= 0: - os.close(descriptor) - self._root_fd = -1 - - def __del__(self) -> None: - """Best-effort descriptor cleanup when an adapter leaves scope.""" + self._buffer_bytes = buffer_bytes + self._lock_timeout_seconds = float(lock_timeout_seconds) + self._root_fd = -1 + self._sha256_fd = -1 + self._tmp_fd = -1 + self._locks_fd = -1 + self._configured_root = os.path.abspath(os.fspath(root)) if isinstance(root, Path) else "" + self._root_device = -1 + self._root_inode = -1 + self._initialized = False try: + self._open_root(root) + except (ArtifactStoreError, ValueError): self.close() + raise except OSError: - pass + self.close() + raise ArtifactConfigurationError("local artifact storage is unavailable") from None + + @property + def identity(self) -> ExternalServiceAdapterIdentity: + """Return the canonical artifact-store/local adapter identity.""" + return _IDENTITY + + @property + def _namespace_identity(self) -> ArtifactStoreNamespaceIdentity: + """Return the canonical identity of the pinned local namespace.""" + root_identity = canonical_json_hash( + { + "private_root": self._configured_root, + "device": self._root_device, + "inode": self._root_inode, + } + ) + return ArtifactStoreNamespaceIdentity( + provider_profile="local-v2", + descriptor_items=( + ("private_prefix", "objects/sha256"), + ("private_root_identity", root_identity), + ), + ) - async def store( + def _initialize_after_namespace_claim( self, - stream: AsyncIterable[bytes], - request: StoreArtifactRequest, - ) -> StoredArtifact: - """Store a stream or prove an exact replay without duplicating bytes.""" - self._validate_store_request(request) - scope = self._scope_key(request.idempotency) - lock: tuple[Any, int] | None = None + claim: ArtifactStoreNamespaceClaim, + ) -> LocalStorageAdapter: + """Create or validate layout through the same root accepted by PostgreSQL.""" try: - lock = await self._acquire_lock_async(scope) - intent = await self._read_json_optional(self._path("intents", scope, ".json")) - if intent is not None: - self._assert_same_request(intent, request.idempotency) - provider_id = self._validated_provider_id(intent["provider_artifact_id"]) - artifact_lock = await self._acquire_lock_async( - self._artifact_lock_scope(provider_id) + if ( + type(claim) is not ArtifactStoreNamespaceClaim + or claim.adapter_identity != self.identity + or claim.namespace_identity != self._namespace_identity + or claim.namespace_fingerprint != self._expected_namespace_fingerprint() + ): + raise ArtifactConfigurationError( + "artifact namespace claim does not match provider" ) - try: - return await self._recover_or_replay(stream, request, intent, scope) - finally: - await self._run_io(self._release_lock, artifact_lock) - - provider_artifact_id = f"art_{uuid4().hex}" - operation_reference = f"op_{uuid4().hex}" - intent = { - "service_principal": request.idempotency.service_principal, - "operation": request.idempotency.operation.value, - "idempotency_key": request.idempotency.key, - "request_digest": request.idempotency.request_digest, - "provider_artifact_id": provider_artifact_id, - "provider_operation_reference": operation_reference, - "expected_sha256": request.expected_sha256, - "expected_size": request.expected_size, - "media_type": request.media_type, - } - await self._write_json_exclusive(self._path("intents", scope, ".json"), intent) - sha256, byte_count = await self._write_stream(stream, provider_artifact_id, request) - metadata = self._new_metadata( - provider_artifact_id=provider_artifact_id, - sha256=sha256, - byte_count=byte_count, - media_type=request.media_type, - ) - await self._write_json_exclusive( - self._path("metadata", provider_artifact_id, ".json"), metadata - ) - receipt = self._make_receipt( - identity=request.idempotency, - operation_reference=operation_reference, - outcome=ReceiptOutcome.STORED, - provider_artifact_id=provider_artifact_id, - sha256=sha256, - byte_count=byte_count, - ) - await self._write_json_exclusive( - self._path("receipts", scope, ".json"), self._receipt_to_json(receipt) - ) - return StoredArtifact( - provider_artifact_id=provider_artifact_id, - sha256=sha256, - byte_count=byte_count, - media_type=request.media_type, - receipt=receipt, - replayed=False, - ) - except asyncio.CancelledError as cancellation: - if lock is not None: - try: - await await_cancellation_resistant(self._cleanup_unpublished(scope)) - except BaseException: - raise cancellation from None - raise - except ( - ArtifactInputMismatchError, - ArtifactIntegrityError, - ArtifactIdempotencyMismatchError, - ArtifactLimitExceededError, - ): + if self._initialized: + raise ArtifactConfigurationError( + "local artifact storage is already initialized" + ) + self._assert_pinned_root_is_current() + self._initialize_layout() + self._initialized = True + return self + except (ArtifactStoreError, ValueError): + self.close() raise - except OSError as exc: - raise ArtifactStoreUnavailableError("local artifact operation failed") from exc - finally: - if lock is not None: - await self._run_io(self._release_lock, lock) - - async def _recover_or_replay( - self, - stream: AsyncIterable[bytes], - request: StoreArtifactRequest, - intent: dict[str, Any], - scope: str, - ) -> StoredArtifact: - """Recover committed bytes or validate a no-commitment exact replay.""" - provider_id = self._validated_provider_id(intent["provider_artifact_id"]) - receipt_json = await self._read_json_optional(self._path("receipts", scope, ".json")) - metadata = await self._read_json_optional(self._path("metadata", provider_id, ".json")) - expected = intent.get("expected_sha256") - if metadata is None and not await self._object_exists(provider_id): - if receipt_json is not None: - self._receipt_from_json(receipt_json) - raise ArtifactIntegrityError("artifact receipt has no committed object") - replay_sha256, replay_size = await self._write_stream( - stream, provider_id, request - ) - metadata = self._new_metadata( - provider_artifact_id=provider_id, - sha256=replay_sha256, - byte_count=replay_size, - media_type=intent["media_type"], - ) - await self._write_json_exclusive( - self._path("metadata", provider_id, ".json"), metadata - ) - else: - replay_sha256, replay_size = await self._consume_stream( - stream, request.maximum_bytes - ) - if expected is not None and replay_sha256 != expected: - await self._quarantine_provider_object(provider_id, "replay_commitment_mismatch") - raise ArtifactIntegrityError("artifact replay does not match its commitment") - if intent.get("expected_size") is not None and replay_size != intent["expected_size"]: - await self._quarantine_provider_object(provider_id, "replay_size_mismatch") - raise ArtifactIntegrityError("artifact replay size does not match its commitment") - - if metadata is None: - observed_sha256, observed_size = await self._hash_object(provider_id) - if observed_sha256 != replay_sha256 or observed_size != replay_size: - await self._quarantine_provider_object(provider_id, "recovery_mismatch") - raise ArtifactIntegrityError("artifact recovery commitment failed") - if intent.get("expected_size") is not None and observed_size != intent["expected_size"]: - await self._quarantine_provider_object(provider_id, "recovery_size_mismatch") - raise ArtifactIntegrityError("artifact recovery size commitment failed") - metadata = self._new_metadata( - provider_artifact_id=provider_id, - sha256=observed_sha256, - byte_count=observed_size, - media_type=intent.get("media_type"), - ) - await self._write_json_exclusive( - self._path("metadata", provider_id, ".json"), metadata - ) - - if replay_sha256 != metadata.get("sha256") or replay_size != metadata.get("byte_count"): - await self._quarantine_provider_object(provider_id, "replay_mismatch") - raise ArtifactIntegrityError("artifact replay does not match persisted bytes") - - await self._assert_metadata_matches_object(provider_id, metadata) - if receipt_json is None: - receipt = self._make_receipt( - identity=request.idempotency, - operation_reference=intent["provider_operation_reference"], - outcome=ReceiptOutcome.STORED, - provider_artifact_id=provider_id, - sha256=metadata["sha256"], - byte_count=metadata["byte_count"], - ) - await self._write_json_exclusive( - self._path("receipts", scope, ".json"), self._receipt_to_json(receipt) - ) - else: - try: - receipt = self._receipt_from_json(receipt_json) - self._assert_receipt_matches(receipt, request.idempotency, metadata) - except ArtifactIntegrityError: - await self._quarantine_provider_object(provider_id, "receipt_mismatch") - raise - return StoredArtifact( - provider_artifact_id=provider_id, - sha256=metadata["sha256"], - byte_count=metadata["byte_count"], - media_type=metadata.get("media_type"), - receipt=receipt, - replayed=True, + except OSError: + self.close() + raise ArtifactConfigurationError("local artifact storage is unavailable") from None + + def _expected_namespace_fingerprint(self) -> str: + """Return the shared canonical fingerprint for this pinned local namespace.""" + _, fingerprint = artifact_store_namespace_material( + backend="local", + adapter_identity=self.identity, + namespace_identity=self._namespace_identity, ) + return fingerprint + + def close(self) -> None: + """Release pinned private directory descriptors.""" + for attribute in ("_locks_fd", "_tmp_fd", "_sha256_fd", "_root_fd"): + descriptor = getattr(self, attribute, -1) + if descriptor >= 0: + try: + os.close(descriptor) + except OSError: + pass + setattr(self, attribute, -1) + self._initialized = False - async def recover_committed_store(self, request: StoreArtifactRequest) -> StoredArtifact: - """Recover a committed local object from a persisted client commitment.""" - self._validate_store_request(request) - if request.expected_sha256 is None or request.expected_size is None: - raise ArtifactInputMismatchError("artifact recovery requires an exact commitment") - scope = self._scope_key(request.idempotency) + def __del__(self) -> None: + """Best-effort descriptor cleanup.""" + self.close() + + async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: + """Publish sealed bytes exclusively or verify an exact immutable replay.""" + self._require_initialized() + if type(source) is not CommittedArtifactSource: + raise ArtifactInputMismatchError("artifact source is not sealed") + commitment = source.commitment + if commitment.byte_count > HARD_MAXIMUM_ARTIFACT_BYTES: + raise ArtifactLimitExceededError("artifact source exceeds provider limit") + + provider_object_ref = self._provider_object_ref(commitment) lock: tuple[Any, int] | None = None + descriptor: int | None = None + temporary_name: str | None = None try: - lock = await self._acquire_lock_async(scope) - intent = await self._read_json_optional(self._path("intents", scope, ".json")) - if intent is None: - raise ArtifactIntegrityError("committed artifact recovery intent is missing") - self._assert_same_request(intent, request.idempotency) - provider_id = self._validated_provider_id(intent["provider_artifact_id"]) - artifact_lock = await self._acquire_lock_async( - self._artifact_lock_scope(provider_id) - ) + lock = await self._acquire_lock_async(commitment.sha256[7:]) try: - return await self._recover_without_replay(request, intent, scope) - finally: - await self._run_io(self._release_lock, artifact_lock) - except OSError as exc: - raise ArtifactStoreUnavailableError("local artifact recovery failed") from exc - finally: - if lock is not None: - await self._run_io(self._release_lock, lock) + await self._run_io(self._head_object, provider_object_ref) + except FileNotFoundError: + pass + else: + await self._run_io( + self._verify_exact_under_lock, + provider_object_ref, + commitment, + ) + return ArtifactPutResult(provider_object_ref, replayed=True) + + descriptor, temporary_name = await self._create_temporary_async() + digest = hashlib.sha256() + byte_count = 0 + async for source_chunk in source.stream(): + if not isinstance(source_chunk, bytes): + raise ArtifactInputMismatchError("artifact source must yield bytes") + view = memoryview(source_chunk) + for offset in range(0, len(view), self._buffer_bytes): + chunk = view[offset : offset + self._buffer_bytes] + byte_count += len(chunk) + if byte_count > commitment.byte_count: + raise ArtifactInputMismatchError( + "artifact source exceeds committed byte count" + ) + digest.update(chunk) + await self._run_io(self._write_all, descriptor, chunk) - async def _recover_without_replay( - self, - request: StoreArtifactRequest, - intent: dict[str, Any], - scope: str, - ) -> StoredArtifact: - """Reconstruct metadata and receipt after independently hashing an object.""" - provider_id = self._validated_provider_id(intent["provider_artifact_id"]) - if not await self._object_exists(provider_id): - raise ArtifactIntegrityError("committed artifact object is missing") - observed_sha256, observed_size = await self._hash_object(provider_id) - if observed_sha256 != request.expected_sha256 or observed_size != request.expected_size: - await self._quarantine_provider_object(provider_id, "recovery_mismatch") - raise ArtifactIntegrityError("artifact recovery commitment failed") - metadata_path = self._path("metadata", provider_id, ".json") - metadata = await self._read_json_optional(metadata_path) - if metadata is None: - metadata = self._new_metadata( - provider_artifact_id=provider_id, - sha256=observed_sha256, - byte_count=observed_size, - media_type=intent.get("media_type"), + observed_sha256 = f"sha256:{digest.hexdigest()}" + if ( + byte_count != commitment.byte_count + or observed_sha256 != commitment.sha256 + ): + raise ArtifactInputMismatchError("artifact source violates commitment") + + await self._run_io(self._seal_temporary, descriptor) + published = await self._run_io( + self._publish_exclusive, + temporary_name, + provider_object_ref, ) - await self._write_json_exclusive(metadata_path, metadata) - else: - await self._assert_metadata_matches_object(provider_id, metadata) - receipt_path = self._path("receipts", scope, ".json") - receipt_payload = await self._read_json_optional(receipt_path) - if receipt_payload is None: - receipt = self._make_receipt( - identity=request.idempotency, - operation_reference=intent["provider_operation_reference"], - outcome=ReceiptOutcome.STORED, - provider_artifact_id=provider_id, - sha256=observed_sha256, - byte_count=observed_size, + temporary_name = None + if not published: + await self._run_io( + self._verify_exact_under_lock, + provider_object_ref, + commitment, + ) + return ArtifactPutResult(provider_object_ref, replayed=True) + return ArtifactPutResult(provider_object_ref, replayed=False) + except asyncio.CancelledError as cancellation: + await await_cancellation_resistant( + asyncio.to_thread(self._cleanup_resources, descriptor, temporary_name, lock) ) - await self._write_json_exclusive(receipt_path, self._receipt_to_json(receipt)) - else: - receipt = self._receipt_from_json(receipt_payload) - self._assert_receipt_matches(receipt, request.idempotency, metadata) - return StoredArtifact( - provider_artifact_id=provider_id, - sha256=observed_sha256, - byte_count=observed_size, - media_type=metadata.get("media_type"), - receipt=receipt, - replayed=True, - ) + descriptor = None + temporary_name = None + lock = None + raise cancellation + except ArtifactStoreError: + raise + except OSError: + raise ArtifactStoreUnavailableError("local artifact operation failed") from None + finally: + if descriptor is not None or temporary_name is not None or lock is not None: + await self._run_io( + self._cleanup_resources, + descriptor, + temporary_name, + lock, + ) + + async def observe_put_result( + self, + commitment: ArtifactCommitment, + ) -> ArtifactPutObservation: + """Read and validate the complete deterministic object when it exists.""" + self._require_initialized() + if type(commitment) is not ArtifactCommitment: + raise ArtifactOperationConflictError("artifact commitment is invalid") + provider_object_ref = self._provider_object_ref(commitment) + observed = await self.head(provider_object_ref) + if not observed.exists: + return ArtifactPutObservation(provider_object_ref, committed=False) + await self._verify_exact(provider_object_ref, commitment) + return ArtifactPutObservation(provider_object_ref, committed=True) def open( self, - provider_artifact_id: str, + provider_object_ref: str, byte_range: ArtifactByteRange | None = None, ) -> AsyncIterator[bytes]: - """Stream an available object using bounded reads.""" - - async def iterator() -> AsyncIterator[bytes]: - """Yield the selected object range and close its file deterministically.""" + """Stream a full object or range through bounded off-loop reads.""" + self._require_initialized() + self._parse_provider_object_ref(provider_object_ref) + if byte_range is not None and type(byte_range) is not ArtifactByteRange: + raise ArtifactOperationConflictError("artifact byte range is invalid") + selected_range = byte_range or ArtifactByteRange() + + async def iterate() -> AsyncIterator[bytes]: + """Yield bounded chunks while retaining one validated descriptor.""" + descriptor: int | None = None try: - provider_id = self._validated_provider_id(provider_artifact_id) - metadata = await self._load_available_metadata(provider_id) - size = int(metadata["byte_count"]) - selected = byte_range or ArtifactByteRange() - if selected.offset > size: - raise ArtifactMalformedRequestError("artifact range starts past object end") - remaining = size - selected.offset - if selected.length is not None: - remaining = min(remaining, selected.length) - handle = await self._run_io(self._open_regular_object, provider_id) try: - await self._run_io(handle.seek, selected.offset) - while remaining > 0: - chunk = await self._run_io( - handle.read, min(self._buffer_bytes, remaining) - ) - if not chunk: - raise ArtifactIntegrityError("artifact ended before recorded size") - remaining -= len(chunk) - yield chunk - finally: - await self._run_io(handle.close) - except OSError as exc: - raise ArtifactStoreUnavailableError("local artifact read failed") from exc - - return iterator() - - async def stat(self, provider_artifact_id: str) -> ArtifactStatus: - """Return provider observations without exposing filesystem paths.""" - provider_id = self._validated_provider_id(provider_artifact_id) - artifact_lock = await self._acquire_lock_async(self._artifact_lock_scope(provider_id)) - try: - try: - metadata = await self._read_json_required( - self._path("metadata", provider_id, ".json") - ) - except ArtifactIntegrityError as exc: - try: - await self._quarantine_provider_object(provider_id, "metadata_invalid") - except (ArtifactIntegrityError, OSError): - pass - raise exc - if metadata.get("integrity_state") == "quarantined": - try: - await self._run_io( - self._assert_regular_file, - self._path("quarantine", provider_id, ".blob"), + descriptor, byte_count = await self._open_object_async( + provider_object_ref ) except FileNotFoundError: - metadata = { - **metadata, - "availability_state": "missing", - "integrity_state": "unknown", - } - else: - if not await self._object_exists(provider_id): - metadata = { - **metadata, - "availability_state": "missing", - "integrity_state": "unknown", - } - else: - await self._assert_metadata_matches_object(provider_id, metadata) - try: - return self._status_from_metadata(metadata) - except ArtifactIntegrityError: - await self._quarantine_provider_object(provider_id, "metadata_invalid") + raise ArtifactObjectMissingError("artifact object is missing") from None + if selected_range.offset > byte_count: + raise ArtifactRangeInvalidError("artifact range starts past object end") + remaining = byte_count - selected_range.offset + if selected_range.length is not None: + remaining = min(remaining, selected_range.length) + position = selected_range.offset + while remaining: + chunk = await self._run_io( + os.pread, + descriptor, + min(self._buffer_bytes, remaining), + position, + ) + if not chunk: + raise ArtifactIntegrityError("artifact object was truncated") + if len(chunk) > self._buffer_bytes or len(chunk) > remaining: + raise ArtifactIntegrityError("artifact read exceeded its bound") + position += len(chunk) + remaining -= len(chunk) + yield chunk + except asyncio.CancelledError as cancellation: + if descriptor is not None: + await await_cancellation_resistant(asyncio.to_thread(os.close, descriptor)) + descriptor = None + raise cancellation + except ArtifactStoreError: raise - finally: - await self._run_io(self._release_lock, artifact_lock) + except OSError: + raise ArtifactStoreUnavailableError("local artifact read failed") from None + finally: + if descriptor is not None: + await self._run_io(os.close, descriptor) - async def verify( - self, - provider_artifact_id: str, - expected_sha256: str, - expected_size: int, - idempotency: IdempotencyIdentity, - ) -> ArtifactOperationResult: - """Independently hash and count a complete provider object.""" - self._validate_identity(idempotency, ArtifactOperation.VERIFY) - self._validate_digest(expected_sha256) - if expected_size < 0: - raise ArtifactMalformedRequestError("expected artifact size must be nonnegative") - provider_id = self._validated_provider_id(provider_artifact_id) - if idempotency.request_digest != canonical_verify_request_digest( - provider_id, expected_sha256, expected_size - ): - raise ArtifactIdempotencyMismatchError("verify request digest is not canonical") - scope = self._scope_key(idempotency) - lock: tuple[Any, int] | None = None - artifact_lock: tuple[Any, int] | None = None + return iterate() + + async def head(self, provider_object_ref: str) -> ArtifactObjectHead: + """Return an existing or missing observation without exposing paths.""" + self._require_initialized() + self._parse_provider_object_ref(provider_object_ref) try: - lock = await self._acquire_lock_async(scope) - artifact_lock = await self._acquire_lock_async( - self._artifact_lock_scope(provider_id) - ) - existing = await self._read_json_optional(self._path("receipts", scope, ".json")) - if existing is not None: - try: - receipt = self._receipt_from_json(existing) - self._assert_operation_receipt( - receipt, - idempotency, - provider_id, - ReceiptOutcome.VERIFIED, - ) - return ArtifactOperationResult(receipt=receipt, replayed=True) - except ArtifactIntegrityError: - await self._quarantine_provider_object(provider_id, "receipt_mismatch") - raise - observed_sha256, observed_size = await self._hash_object(provider_id) - if observed_sha256 != expected_sha256 or observed_size != expected_size: - await self._quarantine_provider_object(provider_id, "verify_mismatch") - raise ArtifactIntegrityError("artifact verification failed") - await self._update_metadata(provider_id, verification_state="verified") - receipt = self._make_receipt( - identity=idempotency, - operation_reference=f"op_{uuid4().hex}", - outcome=ReceiptOutcome.VERIFIED, - provider_artifact_id=provider_id, - sha256=observed_sha256, - byte_count=observed_size, - ) - await self._write_json_exclusive( - self._path("receipts", scope, ".json"), self._receipt_to_json(receipt) + try: + byte_count = await self._run_io( + self._head_object_with_recovery, + provider_object_ref, + ) + except FileNotFoundError: + return ArtifactObjectHead(provider_object_ref, exists=False) + return ArtifactObjectHead( + provider_object_ref, + exists=True, + byte_count=byte_count, ) - return ArtifactOperationResult(receipt=receipt, replayed=False) - except FileNotFoundError as exc: - raise ArtifactNotFoundError("artifact is unavailable") from exc - except OSError as exc: - raise ArtifactStoreUnavailableError("local artifact verification failed") from exc - finally: - await self._release_locks(artifact_lock, lock) + except ArtifactStoreError: + raise + except OSError: + raise ArtifactStoreUnavailableError("local artifact head failed") from None - async def retain( + async def _verify_exact( self, - provider_artifact_id: str, - retention_reference: str, - retention_class: str, - idempotency: IdempotencyIdentity, - ) -> ArtifactOperationResult: - """Add one exact retention reference idempotently.""" - self._validate_identity(idempotency, ArtifactOperation.RETAIN) - self._validate_opaque(retention_reference, "retention reference") - self._validate_retention_class(retention_class) - provider_id = self._validated_provider_id(provider_artifact_id) - if idempotency.request_digest != canonical_retain_request_digest( - provider_id, retention_reference, retention_class + provider_object_ref: str, + commitment: ArtifactCommitment, + ) -> None: + """Independently hash and count one complete existing object.""" + digest = hashlib.sha256() + byte_count = 0 + async for chunk in self.open(provider_object_ref): + digest.update(chunk) + byte_count += len(chunk) + if ( + byte_count != commitment.byte_count + or f"sha256:{digest.hexdigest()}" != commitment.sha256 ): - raise ArtifactIdempotencyMismatchError("retain request digest is not canonical") - return await self._change_retention( - provider_artifact_id, - retention_reference, - retention_class, - idempotency, - releasing=False, - ) + raise ArtifactIntegrityError("artifact object violates commitment") - async def release( - self, - provider_artifact_id: str, - retention_reference: str, - idempotency: IdempotencyIdentity, - ) -> ArtifactOperationResult: - """Release only the exact retention reference supplied.""" - self._validate_identity(idempotency, ArtifactOperation.RELEASE) - self._validate_opaque(retention_reference, "retention reference") - provider_id = self._validated_provider_id(provider_artifact_id) - if idempotency.request_digest != canonical_release_request_digest( - provider_id, retention_reference - ): - raise ArtifactIdempotencyMismatchError("release request digest is not canonical") - return await self._change_retention( - provider_artifact_id, - retention_reference, - None, - idempotency, - releasing=True, - ) + async def _run_io(self, function: Any, *args: Any) -> Any: + """Complete blocking filesystem work before preserving cancellation.""" + return await run_blocking_cancellation_resistant(function, *args) - async def _change_retention( - self, - provider_artifact_id: str, - retention_reference: str, - retention_class: str | None, - identity: IdempotencyIdentity, - *, - releasing: bool, - ) -> ArtifactOperationResult: - """Apply one locked, reference-counted retention change.""" - provider_id = self._validated_provider_id(provider_artifact_id) - scope = self._scope_key(identity) - operation_lock: tuple[Any, int] | None = None - artifact_lock: tuple[Any, int] | None = None + async def _acquire_lock_async(self, digest_hex: str) -> tuple[Any, int]: + """Acquire a cross-process digest lock without leaking it on cancellation.""" + task = asyncio.create_task(asyncio.to_thread(self._acquire_lock, digest_hex)) try: - operation_lock = await self._acquire_lock_async(scope) - artifact_lock = await self._acquire_lock_async( - self._artifact_lock_scope(provider_id) - ) - existing = await self._read_json_optional(self._path("receipts", scope, ".json")) - if existing is not None: - try: - receipt = self._receipt_from_json(existing) - self._assert_operation_receipt( - receipt, - identity, - provider_id, - ReceiptOutcome.RELEASED if releasing else ReceiptOutcome.RETAINED, - retention_reference=retention_reference, - ) - return ArtifactOperationResult(receipt=receipt, replayed=True) - except ArtifactIntegrityError: - await self._quarantine_provider_object(provider_id, "receipt_mismatch") - raise - metadata = await self._load_available_metadata(provider_id) - references = dict(metadata.get("retention_references", {})) - intent_path = self._path("intents", scope, ".json") - mutation_intent = await self._read_json_optional(intent_path) - if mutation_intent is not None: - self._assert_retention_intent( - mutation_intent, - identity, - provider_id, - retention_reference, - retention_class, - releasing=releasing, - ) - if releasing: - recorded = references.get(retention_reference) - if recorded is None and mutation_intent is None: - raise ArtifactRetentionConflictError("retention reference is not active") - if recorded is not None and recorded.get("owner") != identity.service_principal: - raise ArtifactRetentionConflictError("retention reference owner changed") - recorded_class = ( - recorded["retention_class"] - if recorded is not None - else mutation_intent["retention_class"] + return await asyncio.shield(task) + except asyncio.CancelledError as cancellation: + try: + lock = await await_cancellation_resistant(task) + except Exception: + raise cancellation from None + await await_cancellation_resistant(asyncio.to_thread(self._release_lock, lock)) + raise cancellation + + async def _create_temporary_async(self) -> tuple[int, str]: + """Create a temporary file without losing ownership on cancellation.""" + task = asyncio.create_task(asyncio.to_thread(self._create_temporary)) + try: + return await asyncio.shield(task) + except asyncio.CancelledError as cancellation: + try: + descriptor, temporary_name = await await_cancellation_resistant(task) + except Exception: + raise cancellation from None + await await_cancellation_resistant( + asyncio.to_thread( + self._cleanup_resources, + descriptor, + temporary_name, + None, ) - else: - recorded_class = retention_class - existing_reference = references.get(retention_reference) - if existing_reference is not None and existing_reference != { - "retention_class": retention_class, - "owner": identity.service_principal, - }: - raise ArtifactIdempotencyMismatchError("retention reference class changed") - - if mutation_intent is None: - mutation_intent = { - "service_principal": identity.service_principal, - "operation": identity.operation.value, - "idempotency_key": identity.key, - "request_digest": identity.request_digest, - "provider_artifact_id": provider_id, - "provider_operation_reference": f"op_{uuid4().hex}", - "retention_reference": retention_reference, - "retention_class": recorded_class, - } - await self._write_json_exclusive(intent_path, mutation_intent) - - if releasing: - references.pop(retention_reference, None) - else: - references[retention_reference] = { - "retention_class": retention_class, - "owner": identity.service_principal, - } - retention_state = "retained" if references else "released" - await self._update_metadata( - provider_id, - retention_references=references, - retention_state=retention_state, - ) - receipt = self._make_receipt( - identity=identity, - operation_reference=mutation_intent["provider_operation_reference"], - outcome=(ReceiptOutcome.RELEASED if releasing else ReceiptOutcome.RETAINED), - provider_artifact_id=provider_id, - sha256=metadata["sha256"], - byte_count=metadata["byte_count"], - retention_reference=retention_reference, - retention_class=recorded_class, - ) - await self._write_json_exclusive( - self._path("receipts", scope, ".json"), self._receipt_to_json(receipt) ) - return ArtifactOperationResult(receipt=receipt, replayed=False) - except OSError as exc: - raise ArtifactStoreUnavailableError("local artifact retention failed") from exc - finally: - await self._release_locks(artifact_lock, operation_lock) + raise cancellation - async def get_operation_receipt( - self, - service_principal: str, - operation: ArtifactOperation, - idempotency_key: str, - ) -> OperationReceipt | None: - """Load the authoritative receipt for a complete operation identity.""" - identity = IdempotencyIdentity( - service_principal=service_principal, - operation=operation, - key=idempotency_key, - request_digest="sha256:" + "0" * 64, + async def _open_object_async(self, provider_object_ref: str) -> tuple[int, int]: + """Open an object without losing descriptor ownership on cancellation.""" + task = asyncio.create_task( + asyncio.to_thread(self._open_object_with_recovery, provider_object_ref) ) - self._validate_service_principal(service_principal) - self._validate_opaque(idempotency_key, "idempotency key") - payload = await self._read_json_optional( - self._path("receipts", self._scope_key(identity), ".json") - ) - return None if payload is None else self._receipt_from_json(payload) + try: + return await asyncio.shield(task) + except asyncio.CancelledError as cancellation: + try: + descriptor, _ = await await_cancellation_resistant(task) + except Exception: + raise cancellation from None + await await_cancellation_resistant(asyncio.to_thread(os.close, descriptor)) + raise cancellation - def _initialize_root(self, configured_root: Path) -> None: - """Create and verify the private local layout.""" + def _open_root(self, configured_root: Path) -> None: + """Open and pin one existing owner-private root without mutation.""" + if not isinstance(configured_root, Path): + raise ValueError("artifact root must be a path") if configured_root.is_symlink(): - raise ValueError("artifact root must not be a symlink") - configured_root.mkdir(parents=True, exist_ok=True, mode=0o700) - if configured_root.resolve() != self._root or not configured_root.is_dir(): - raise ValueError("artifact root must be a dedicated directory") - os.chmod(self._root, 0o700) - for name in ("objects", "metadata", "receipts", "intents", "locks", "tmp", "quarantine"): - path = self._root / name - path.mkdir(mode=0o700, exist_ok=True) - if path.is_symlink() or not path.is_dir(): - raise ValueError("artifact storage layout is unsafe") - os.chmod(path, 0o700) - - def _cleanup_abandoned_temporary_files(self) -> None: - """Remove unpublished local temporary files left by interrupted writes.""" - intents: dict[str, str] = {} - intent_directory = self._open_directory_fd(self._root / "intents") - try: - for name in os.listdir(intent_directory): - match = re.fullmatch(r"([0-9a-f]{64})\.json", name) - if match is None: - continue - try: - intent = self._read_json_sync( - self._path("intents", match.group(1), ".json") - ) - except (ArtifactIntegrityError, OSError, ValueError, json.JSONDecodeError): - continue - provider_id = intent.get("provider_artifact_id") - if isinstance(provider_id, str) and _OPAQUE_ID.fullmatch(provider_id): - intents[provider_id] = match.group(1) - finally: - os.close(intent_directory) + raise ArtifactConfigurationError("local artifact root is unsafe") + self._root_fd = os.open( + configured_root, + os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0), + ) + self._assert_private_directory(self._root_fd) + details = os.fstat(self._root_fd) + self._root_device = details.st_dev + self._root_inode = details.st_ino - directory = self._open_directory_fd(self._root / "tmp") + def _assert_pinned_root_is_current(self) -> None: + """Reject same-path replacement before any layout mutation.""" try: - for name in os.listdir(directory): - match = re.fullmatch(r"(art_[0-9a-f]{32})_[0-9a-f]{32}\.part", name) - if match is None or match.group(1) not in intents: - continue - details = os.stat(name, dir_fd=directory, follow_symlinks=False) - if time.time() - details.st_mtime < 3600: - continue - lock = self._acquire_lock(intents[match.group(1)]) - try: - details = os.stat(name, dir_fd=directory, follow_symlinks=False) - if time.time() - details.st_mtime >= 3600: - os.unlink(name, dir_fd=directory) - except FileNotFoundError: - continue - finally: - self._release_lock(lock) - os.fsync(directory) - finally: - os.close(directory) - - def _path(self, directory: str, opaque: str, suffix: str) -> Path: - """Return a contained private path from a validated internal token.""" - if not re.fullmatch(r"[a-zA-Z0-9_-]{1,128}", opaque): - raise ArtifactInputMismatchError("artifact identifier is invalid") - path = self._root / directory / f"{opaque}{suffix}" - if path.parent.parent != self._root: - raise ArtifactInputMismatchError("artifact identifier is invalid") - return path - - def _scope_key(self, identity: IdempotencyIdentity) -> str: - """Derive a private stable filename key from operation identity.""" - material = f"{identity.service_principal}\0{identity.operation.value}\0{identity.key}" - return hashlib.sha256(material.encode("utf-8")).hexdigest() - - @staticmethod - def _artifact_lock_scope(provider_id: str) -> str: - """Derive the shared metadata-mutation lock for one provider object.""" - return hashlib.sha256(f"artifact\0{provider_id}".encode()).hexdigest() - - def _validate_store_request(self, request: StoreArtifactRequest) -> None: - """Validate bounded store metadata before filesystem work.""" - self._validate_identity(request.idempotency, ArtifactOperation.STORE) - if request.maximum_bytes < 0: - raise ArtifactMalformedRequestError("maximum artifact bytes must be nonnegative") - if request.expected_size is not None and request.expected_size < 0: - raise ArtifactMalformedRequestError("expected artifact size must be nonnegative") - if request.expected_sha256 is not None: - self._validate_digest(request.expected_sha256) + current = os.lstat(self._configured_root) + except OSError: + raise ArtifactConfigurationError("local artifact root changed after validation") from None if ( - not request.media_type - or len(request.media_type) > 255 - or any(ord(character) < 32 or ord(character) == 127 for character in request.media_type) + not stat.S_ISDIR(current.st_mode) + or current.st_uid != os.geteuid() + or stat.S_IMODE(current.st_mode) != 0o700 + or current.st_dev != self._root_device + or current.st_ino != self._root_inode ): - raise ArtifactMalformedRequestError("artifact media type is invalid") - if len(request.metadata) > 32: - raise ArtifactMalformedRequestError("artifact metadata exceeds its item limit") - for name, value in request.metadata.items(): - if ( - not re.fullmatch(r"[a-z][a-z0-9_]{0,63}", name) - or not isinstance(value, str) - or len(value) > 256 - ): - raise ArtifactMalformedRequestError("artifact metadata is invalid") - if request.idempotency.request_digest != canonical_store_request_digest(request): - raise ArtifactIdempotencyMismatchError("store request digest is not canonical") - - def _validate_identity( - self, identity: IdempotencyIdentity, expected_operation: ArtifactOperation - ) -> None: - """Validate operation identity and request commitment.""" - if identity.operation != expected_operation: - raise ArtifactMalformedRequestError("artifact operation identity is invalid") - self._validate_service_principal(identity.service_principal) - self._validate_opaque(identity.key, "idempotency key") - self._validate_digest(identity.request_digest) - - @staticmethod - def _validate_opaque(value: str, label: str) -> None: - """Require one bounded provider-neutral opaque identifier.""" - if not re.fullmatch(r"(?!\.{1,2}$)[A-Za-z0-9][A-Za-z0-9._~-]{0,127}", value): - raise ArtifactMalformedRequestError(f"{label} is invalid") - - @staticmethod - def _validate_service_principal(value: str) -> None: - """Require one bounded provider-neutral service principal.""" - if not re.fullmatch(r"[a-z0-9](?:[a-z0-9._-]{1,126}[a-z0-9])", value) or len( - value - ) > 100: - raise ArtifactMalformedRequestError("service principal is invalid") - - @staticmethod - def _validate_retention_class(value: str) -> None: - """Require the provider-neutral retention-class grammar.""" - if not re.fullmatch(r"[a-z][a-z0-9_-]{0,63}", value): - raise ArtifactMalformedRequestError("retention class is invalid") + raise ArtifactConfigurationError("local artifact root changed after validation") - @staticmethod - def _validate_digest(value: str) -> None: - """Require canonical lower-case SHA-256 values.""" - if not _DIGEST.fullmatch(value): - raise ArtifactMalformedRequestError("artifact digest is invalid") + def _initialize_layout(self) -> None: + """Initialize only an empty or already-valid v2 layout on the pinned root.""" + fcntl.flock(self._root_fd, fcntl.LOCK_EX) + try: + entries = set(os.listdir(self._root_fd)) + if not entries.issubset(_LAYOUT_ENTRY_NAMES): + raise ArtifactConfigurationError("local artifact layout is incompatible") + if _LAYOUT_MARKER_TEMPORARY in entries: + self._recover_layout_marker(entries) + entries = set(os.listdir(self._root_fd)) + if _LAYOUT_MARKER in entries: + self._validate_existing_layout(entries) + else: + if entries: + raise ArtifactConfigurationError("local artifact layout is incompatible") + self._publish_layout_marker() - @staticmethod - def _validated_provider_id(value: str) -> str: - """Require an opaque provider artifact identifier.""" - if not _OPAQUE_ID.fullmatch(value): - raise ArtifactNotFoundError("artifact is unavailable") - return value + self._validate_marker() - async def _write_stream( - self, - stream: AsyncIterable[bytes], - provider_id: str, - request: StoreArtifactRequest, - ) -> tuple[str, int]: - """Write and publish an independently hashed bounded stream.""" - temporary = self._path("tmp", f"{provider_id}_{uuid4().hex}", ".part") - final = self._path("objects", provider_id, ".blob") - handle = await self._run_io(self._open_exclusive, temporary) - write_complete = False - close_complete = False - try: - sha256, total = await self._write_bounded_stream_to_private_file( - stream, - handle, - request.maximum_bytes, - ) - write_complete = True - finally: + objects_fd = self._ensure_directory(self._root_fd, "objects") try: - await self._run_io(os.close, handle) - close_complete = True + self._sha256_fd = self._ensure_directory(objects_fd, "sha256") finally: - if not write_complete or not close_complete: - await self._run_io(self._unlink_optional, temporary) - if request.expected_sha256 is not None and sha256 != request.expected_sha256: - await self._run_io(self._unlink_optional, temporary) - raise ArtifactInputMismatchError("artifact bytes do not match expected digest") - if request.expected_size is not None and total != request.expected_size: - await self._run_io(self._unlink_optional, temporary) - raise ArtifactInputMismatchError("artifact bytes do not match expected size") - try: - await self._run_io(self._publish_exclusive, temporary, final) - except BaseException: - await self._run_io(self._unlink_optional, temporary) - raise - return sha256, total - - async def _write_bounded_stream_to_private_file( - self, - stream: AsyncIterable[bytes], - descriptor: int, - maximum_bytes: int, - ) -> tuple[str, int]: - """Hash and write one bounded stream to an already-private file.""" - digest = hashlib.sha256() - total = 0 - async for source_chunk in stream: - if not isinstance(source_chunk, bytes): - raise ArtifactInputMismatchError("artifact stream must yield bytes") - view = memoryview(source_chunk) - for offset in range(0, len(view), self._buffer_bytes): - chunk = view[offset : offset + self._buffer_bytes] - total += len(chunk) - if total > maximum_bytes: - raise ArtifactLimitExceededError("artifact exceeds maximum bytes") - digest.update(chunk) - await self._run_io(self._write_all, descriptor, chunk) - await self._run_io(os.fsync, descriptor) - return f"sha256:{digest.hexdigest()}", total - - async def _consume_stream( - self, stream: AsyncIterable[bytes], maximum_bytes: int - ) -> tuple[str, int]: - """Fully hash and count a replay without storing a second object.""" - digest = hashlib.sha256() - total = 0 - async for source_chunk in stream: - if not isinstance(source_chunk, bytes): - raise ArtifactInputMismatchError("artifact stream must yield bytes") - view = memoryview(source_chunk) - for offset in range(0, len(view), self._buffer_bytes): - chunk = view[offset : offset + self._buffer_bytes] - total += len(chunk) - if total > maximum_bytes: - raise ArtifactLimitExceededError("artifact exceeds maximum bytes") - digest.update(chunk) - return f"sha256:{digest.hexdigest()}", total - - async def _hash_object(self, provider_id: str) -> tuple[str, int]: - """Independently hash and count one regular provider object.""" - handle = await self._run_io(self._open_regular_object, provider_id) - digest = hashlib.sha256() - total = 0 - try: - while True: - chunk = await self._run_io(handle.read, self._buffer_bytes) - if not chunk: - break - digest.update(chunk) - total += len(chunk) + os.close(objects_fd) + self._tmp_fd = self._ensure_directory(self._root_fd, "tmp") + self._locks_fd = self._ensure_directory(self._root_fd, "locks") finally: - await self._run_io(handle.close) - return f"sha256:{digest.hexdigest()}", total - - async def _assert_metadata_matches_object( - self, provider_id: str, metadata: dict[str, Any] - ) -> None: - """Quarantine any object that differs from provider metadata.""" - observed_sha256, observed_size = await self._hash_object(provider_id) - if observed_sha256 != metadata.get("sha256") or observed_size != metadata.get("byte_count"): - await self._quarantine_provider_object(provider_id, "object_metadata_mismatch") - raise ArtifactIntegrityError("artifact provider facts are inconsistent") - - async def _load_available_metadata(self, provider_id: str) -> dict[str, Any]: - """Load metadata only for an available integrity-valid object.""" - metadata = await self._read_json_required(self._path("metadata", provider_id, ".json")) - if metadata.get("integrity_state") != "valid" or metadata.get("availability_state") != "available": - raise ArtifactNotFoundError("artifact is unavailable") - await self._run_io(self._assert_regular_file, self._path("objects", provider_id, ".blob")) - return metadata - - async def _update_metadata(self, provider_id: str, **changes: Any) -> dict[str, Any]: - """Atomically update mutable provider observations.""" - path = self._path("metadata", provider_id, ".json") - metadata = await self._read_json_required(path) - metadata.update(changes) - await self._replace_json(path, metadata) - return metadata - - async def _quarantine_provider_object(self, provider_id: str, reason: str) -> None: - """Durably make a corrupt provider object unavailable.""" - source = self._path("objects", provider_id, ".blob") - target = self._path("quarantine", provider_id, ".blob") - observed: tuple[str, int] | None = None - try: - observed = await self._hash_object(provider_id) - except FileNotFoundError: - pass - try: - await self._run_io(self._move_to_quarantine, source, target) - except FileNotFoundError: - pass - metadata_path = self._path("metadata", provider_id, ".json") - try: - persisted_metadata = await self._read_json_optional(metadata_path) - metadata_exists = persisted_metadata is not None - metadata = persisted_metadata or {} - except ArtifactIntegrityError: - metadata_exists = True - metadata = {} - if observed is not None: - metadata.update( - provider_artifact_id=provider_id, - sha256=observed[0], - byte_count=observed[1], - ) - media_type = metadata.get("media_type") - if not isinstance(media_type, str) or not media_type: - metadata["media_type"] = "application/octet-stream" - metadata.update( - verification_state="failed", - retention_state="unretained", - retention_references={}, - integrity_state="quarantined", - availability_state="unavailable", - quarantine_reason=reason, + fcntl.flock(self._root_fd, fcntl.LOCK_UN) + + def _publish_layout_marker(self) -> None: + """Durably publish the marker from one private temporary file.""" + marker = os.open( + _LAYOUT_MARKER_TEMPORARY, + os.O_CREAT | os.O_EXCL | os.O_WRONLY | getattr(os, "O_NOFOLLOW", 0), + 0o600, + dir_fd=self._root_fd, ) - if metadata_exists: - await self._replace_json(metadata_path, metadata) - else: - await self._write_json_exclusive(metadata_path, metadata) - - async def _object_exists(self, provider_id: str) -> bool: - """Return whether a regular immutable object exists.""" - try: - await self._run_io(self._assert_regular_file, self._path("objects", provider_id, ".blob")) - return True - except FileNotFoundError: - return False - - async def _cleanup_unpublished(self, scope: str) -> None: - """Remove only temporary state for an uncommitted operation.""" - intent = await self._read_json_optional(self._path("intents", scope, ".json")) - if intent is None: - return - provider_id = intent.get("provider_artifact_id") - if isinstance(provider_id, str) and _OPAQUE_ID.fullmatch(provider_id): - await self._run_io(self._cleanup_provider_temporary_files, provider_id) - - def _cleanup_provider_temporary_files(self, provider_id: str) -> None: - """Remove matching temporary files through a verified directory descriptor.""" - directory = self._open_directory_fd(self._root / "tmp") try: - prefix = f"{provider_id}_" - for name in os.listdir(directory): - if name.startswith(prefix) and name.endswith(".part"): - os.unlink(name, dir_fd=directory) - os.fsync(directory) + self._write_all(marker, memoryview(_LAYOUT_MARKER_BYTES)) + os.fsync(marker) finally: - os.close(directory) - - def _new_metadata( - self, - *, - provider_artifact_id: str, - sha256: str, - byte_count: int, - media_type: str | None, - ) -> dict[str, Any]: - """Build provider-owned object metadata.""" - return { - "provider_artifact_id": provider_artifact_id, - "sha256": sha256, - "byte_count": byte_count, - "media_type": media_type, - "verification_state": "pending", - "retention_state": "unretained", - "availability_state": "available", - "integrity_state": "valid", - "retention_references": {}, - } - - def _make_receipt( - self, - *, - identity: IdempotencyIdentity, - operation_reference: str, - outcome: ReceiptOutcome, - provider_artifact_id: str, - sha256: str, - byte_count: int, - retention_reference: str | None = None, - retention_class: str | None = None, - ) -> OperationReceipt: - """Build a bounded receipt and its canonical response digest.""" - receipt_id = f"receipt_{uuid4().hex}" - response = self._receipt_response_payload( - receipt_id=receipt_id, - operation_reference=operation_reference, - provider_artifact_id=provider_artifact_id, - sha256=sha256, - byte_count=byte_count, - outcome=outcome, - retention_reference=retention_reference, - retention_class=retention_class, + os.close(marker) + os.link( + _LAYOUT_MARKER_TEMPORARY, + _LAYOUT_MARKER, + src_dir_fd=self._root_fd, + dst_dir_fd=self._root_fd, + follow_symlinks=False, ) - return OperationReceipt( - receipt_id=receipt_id, - provider_operation_reference=operation_reference, - identity=identity, - response_digest=canonical_json_hash(response), - outcome=outcome, - provider_artifact_id=provider_artifact_id, - sha256=sha256, - byte_count=byte_count, - recorded_at=datetime.now(UTC), - retention_reference=retention_reference, - retention_class=retention_class, + os.fsync(self._root_fd) + os.unlink(_LAYOUT_MARKER_TEMPORARY, dir_fd=self._root_fd) + os.fsync(self._root_fd) + + def _recover_layout_marker(self, entries: set[str]) -> None: + """Complete or remove one exact durable marker publication remnant.""" + temporary_details = self._validate_marker( + _LAYOUT_MARKER_TEMPORARY, + recoverable=True, ) - - @staticmethod - def _receipt_response_payload( - *, - receipt_id: str, - operation_reference: str, - provider_artifact_id: str | None, - sha256: str | None, - byte_count: int | None, - outcome: ReceiptOutcome, - retention_reference: str | None, - retention_class: str | None, - ) -> dict[str, Any]: - """Build the one canonical operation-response payload.""" - response: dict[str, Any] = { - "receipt_id": receipt_id, - "provider_operation_id": operation_reference, - "provider_artifact_id": provider_artifact_id, - "sha256": sha256, - "byte_count": byte_count, - "outcome": outcome.value, - } - if retention_reference is not None: - response["retention_reference"] = retention_reference - if retention_class is not None: - response["retention_class"] = retention_class - return response - - @classmethod - def _receipt_response_digest(cls, receipt: OperationReceipt) -> str: - """Recompute the canonical operation response commitment.""" - response = cls._receipt_response_payload( - receipt_id=receipt.receipt_id, - operation_reference=receipt.provider_operation_reference, - provider_artifact_id=receipt.provider_artifact_id, - sha256=receipt.sha256, - byte_count=receipt.byte_count, - outcome=receipt.outcome, - retention_reference=receipt.retention_reference, - retention_class=receipt.retention_class, - ) - return canonical_json_hash(response) - - @staticmethod - def _receipt_to_json(receipt: OperationReceipt) -> dict[str, Any]: - """Serialize a receipt for private provider persistence.""" - return { - "receipt_id": receipt.receipt_id, - "provider_operation_reference": receipt.provider_operation_reference, - "service_principal": receipt.identity.service_principal, - "operation": receipt.identity.operation.value, - "idempotency_key": receipt.identity.key, - "request_digest": receipt.identity.request_digest, - "response_digest": receipt.response_digest, - "outcome": receipt.outcome.value, - "provider_artifact_id": receipt.provider_artifact_id, - "sha256": receipt.sha256, - "byte_count": receipt.byte_count, - "recorded_at": receipt.recorded_at.isoformat(), - "retention_reference": receipt.retention_reference, - "retention_class": receipt.retention_class, - "details": [ - {"name": detail.name, "value": detail.value} for detail in receipt.details - ], - } - - def _receipt_from_json(self, payload: dict[str, Any]) -> OperationReceipt: - """Parse and validate private provider receipt evidence.""" - try: - identity = IdempotencyIdentity( - service_principal=payload["service_principal"], - operation=ArtifactOperation(payload["operation"]), - key=payload["idempotency_key"], - request_digest=payload["request_digest"], - ) - self._validate_identity(identity, identity.operation) - provider_id = self._validated_provider_id(payload["provider_artifact_id"]) - self._validate_digest(payload["response_digest"]) - self._validate_digest(payload["sha256"]) - if not _OPERATION_REF.fullmatch(payload["provider_operation_reference"]): - raise ValueError - byte_count = int(payload["byte_count"]) - details_payload = payload.get("details", []) + if _LAYOUT_MARKER in entries: + marker_details = self._validate_marker(_LAYOUT_MARKER, recoverable=True) if ( - byte_count < 0 - or not isinstance(details_payload, list) - or len(details_payload) > 16 + temporary_details.st_dev != marker_details.st_dev + or temporary_details.st_ino != marker_details.st_ino + or temporary_details.st_nlink != 2 + or marker_details.st_nlink != 2 ): - raise ValueError - details = tuple( - ReceiptDetail(name=detail["name"], value=detail["value"]) - for detail in details_payload - ) - if any( - set(detail) != {"name", "value"} - or not isinstance(detail["name"], str) - or not re.fullmatch(r"[a-z][a-z0-9_]{0,63}", detail["name"]) - or not isinstance(detail["value"], str) - or len(detail["value"]) > 512 - for detail in details_payload + raise ArtifactConfigurationError("local artifact layout is incompatible") + os.unlink(_LAYOUT_MARKER_TEMPORARY, dir_fd=self._root_fd) + else: + if ( + entries != {_LAYOUT_MARKER_TEMPORARY} + or temporary_details.st_nlink != 1 ): - raise ValueError - receipt = OperationReceipt( - receipt_id=payload["receipt_id"], - provider_operation_reference=payload["provider_operation_reference"], - identity=identity, - response_digest=payload["response_digest"], - outcome=ReceiptOutcome(payload["outcome"]), - provider_artifact_id=provider_id, - sha256=payload["sha256"], - byte_count=byte_count, - recorded_at=datetime.fromisoformat(payload["recorded_at"]), - retention_reference=payload.get("retention_reference"), - retention_class=payload.get("retention_class"), - details=details, + raise ArtifactConfigurationError("local artifact layout is incompatible") + os.link( + _LAYOUT_MARKER_TEMPORARY, + _LAYOUT_MARKER, + src_dir_fd=self._root_fd, + dst_dir_fd=self._root_fd, + follow_symlinks=False, ) - if not re.fullmatch(r"receipt_[0-9a-f]{32}", receipt.receipt_id): - raise ValueError - if receipt.response_digest != self._receipt_response_digest(receipt): - raise ValueError - return receipt - except ( - ArtifactInputMismatchError, - ArtifactMalformedRequestError, - ArtifactNotFoundError, - KeyError, - TypeError, - ValueError, - ) as exc: - raise ArtifactIntegrityError("artifact operation receipt is invalid") from exc - - def _assert_same_request( - self, intent: dict[str, Any], identity: IdempotencyIdentity - ) -> None: - """Reject reuse of one operation identity with changed request content.""" - if intent.get("request_digest") != identity.request_digest: - raise ArtifactIdempotencyMismatchError("artifact operation request changed") - - def _assert_retention_intent( + os.fsync(self._root_fd) + os.unlink(_LAYOUT_MARKER_TEMPORARY, dir_fd=self._root_fd) + os.fsync(self._root_fd) + + def _validate_existing_layout(self, root_entries: set[str]) -> None: + """Validate the complete existing v2 grammar before any mutation.""" + self._validate_marker() + temporary_links: set[tuple[int, int]] = set() + if "tmp" in root_entries: + temporary_fd = self._open_existing_directory(self._root_fd, "tmp") + try: + temporary_links = self._validate_temporary_entries(temporary_fd) + finally: + os.close(temporary_fd) + object_links: set[tuple[int, int]] = set() + if "objects" in root_entries: + objects_fd = self._open_existing_directory(self._root_fd, "objects") + try: + object_links = self._validate_object_entries(objects_fd) + finally: + os.close(objects_fd) + if object_links != temporary_links: + raise ArtifactIntegrityError("local artifact recovery links are inconsistent") + if "locks" in root_entries: + locks_fd = self._open_existing_directory(self._root_fd, "locks") + try: + self._validate_lock_entries(locks_fd) + finally: + os.close(locks_fd) + + def _validate_object_entries(self, objects_fd: int) -> set[tuple[int, int]]: + """Require only canonical immutable objects and recoverable hard links.""" + object_entries = set(os.listdir(objects_fd)) + if not object_entries.issubset({"sha256"}): + raise ArtifactConfigurationError("local artifact layout is incompatible") + if "sha256" not in object_entries: + return set() + sha256_fd = self._open_existing_directory(objects_fd, "sha256") + recovery_links: set[tuple[int, int]] = set() + try: + for prefix in os.listdir(sha256_fd): + if _DIGEST_PREFIX_NAME.fullmatch(prefix) is None: + raise ArtifactConfigurationError("local artifact layout is incompatible") + prefix_fd = self._open_existing_directory(sha256_fd, prefix) + try: + for name in os.listdir(prefix_fd): + if _DIGEST_OBJECT_NAME.fullmatch(name) is None: + raise ArtifactConfigurationError( + "local artifact layout is incompatible" + ) + details = self._stat_layout_file( + prefix_fd, + name, + recoverable=True, + ) + if details is None: + continue + if stat.S_IMODE(details.st_mode) != 0o400: + raise ArtifactIntegrityError("local artifact object is mutable") + if details.st_nlink == 2: + recovery_links.add((details.st_dev, details.st_ino)) + finally: + os.close(prefix_fd) + finally: + os.close(sha256_fd) + return recovery_links + + def _validate_temporary_entries(self, temporary_fd: int) -> set[tuple[int, int]]: + """Require only bounded private put temporaries and recovery links.""" + recovery_links: set[tuple[int, int]] = set() + for name in os.listdir(temporary_fd): + if _TEMPORARY_NAME.fullmatch(name) is None: + raise ArtifactConfigurationError("local artifact layout is incompatible") + details = self._stat_layout_file(temporary_fd, name, recoverable=True) + if details is None: + continue + mode = stat.S_IMODE(details.st_mode) + if mode not in {0o400, 0o600} or (details.st_nlink == 2 and mode != 0o400): + raise ArtifactIntegrityError("local artifact temporary is unsafe") + if details.st_nlink == 1: + raise ArtifactConfigurationError( + "local artifact startup requires orphan temporary cleanup" + ) + recovery_links.add((details.st_dev, details.st_ino)) + return recovery_links + + def _validate_lock_entries(self, locks_fd: int) -> None: + """Require only canonical private digest-lock files.""" + for name in os.listdir(locks_fd): + if _LOCK_NAME.fullmatch(name) is None: + raise ArtifactConfigurationError("local artifact layout is incompatible") + details = self._stat_layout_file(locks_fd, name, recoverable=False) + if details is not None and stat.S_IMODE(details.st_mode) != 0o600: + raise ArtifactIntegrityError("local artifact lock is unsafe") + + def _require_initialized(self) -> None: + """Reject provider operations before PostgreSQL namespace admission.""" + if not self._initialized: + raise ArtifactConfigurationError("artifact store namespace is not initialized") + + def _validate_marker( self, - intent: dict[str, Any], - identity: IdempotencyIdentity, - provider_artifact_id: str, - retention_reference: str, - retention_class: str | None, + name: str = _LAYOUT_MARKER, *, - releasing: bool, - ) -> None: - """Validate a durable retention intent before completing recovery.""" - expected_class = None if releasing else retention_class - if ( - intent.get("service_principal") != identity.service_principal - or intent.get("operation") != identity.operation.value - or intent.get("idempotency_key") != identity.key - or intent.get("request_digest") != identity.request_digest - or intent.get("provider_artifact_id") != provider_artifact_id - or intent.get("retention_reference") != retention_reference - or (not releasing and intent.get("retention_class") != expected_class) - or not _OPERATION_REF.fullmatch( - str(intent.get("provider_operation_reference", "")) + recoverable: bool = False, + ) -> os.stat_result: + """Require the exact private v2 layout marker.""" + descriptor = os.open( + name, + os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0), + dir_fd=self._root_fd, + ) + try: + details = ( + self._assert_recoverable_object(descriptor) + if recoverable + else self._assert_private_regular(descriptor) ) - ): - raise ArtifactIdempotencyMismatchError("artifact retention intent changed") + if stat.S_IMODE(details.st_mode) != 0o600: + raise ArtifactConfigurationError("local artifact layout is incompatible") + content = os.read(descriptor, len(_LAYOUT_MARKER_BYTES) + 1) + if content != _LAYOUT_MARKER_BYTES: + raise ArtifactConfigurationError("local artifact layout is incompatible") + return details + finally: + os.close(descriptor) - def _assert_receipt_identity( - self, receipt: OperationReceipt, identity: IdempotencyIdentity - ) -> None: - """Require exact operation identity and request digest.""" - if receipt.identity != identity: - raise ArtifactIdempotencyMismatchError("artifact operation request changed") + def _ensure_directory(self, parent_fd: int, name: str) -> int: + """Create or open one fixed private no-follow directory.""" + try: + os.mkdir(name, 0o700, dir_fd=parent_fd) + except FileExistsError: + pass + return self._open_existing_directory(parent_fd, name) - def _assert_operation_receipt( - self, - receipt: OperationReceipt, - identity: IdempotencyIdentity, - provider_artifact_id: str, - outcome: ReceiptOutcome, - *, - retention_reference: str | None = None, - ) -> None: - """Require replay evidence to describe the requested provider effect.""" - self._assert_receipt_identity(receipt, identity) - if ( - receipt.provider_artifact_id != provider_artifact_id - or receipt.outcome != outcome - or receipt.retention_reference != retention_reference - ): - raise ArtifactIntegrityError("artifact receipt facts do not match the operation") + def _open_existing_directory(self, parent_fd: int, name: str) -> int: + """Open and validate one existing private directory without leaking it.""" + descriptor = os.open( + name, + os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0), + dir_fd=parent_fd, + ) + try: + self._assert_private_directory(descriptor) + return descriptor + except BaseException: + os.close(descriptor) + raise - def _assert_receipt_matches( + def _stat_layout_file( self, - receipt: OperationReceipt, - identity: IdempotencyIdentity, - metadata: dict[str, Any], - ) -> None: - """Reject altered receipt facts and quarantine through caller.""" - self._assert_receipt_identity(receipt, identity) - if ( - receipt.outcome != ReceiptOutcome.STORED - or receipt.provider_artifact_id != metadata.get("provider_artifact_id") - or receipt.sha256 != metadata.get("sha256") - or receipt.byte_count != metadata.get("byte_count") - or receipt.retention_reference is not None - or receipt.retention_class is not None - ): - raise ArtifactIntegrityError("artifact receipt does not match provider metadata") - - def _status_from_metadata(self, metadata: dict[str, Any]) -> ArtifactStatus: - """Convert private metadata into a provider-neutral status.""" + parent_fd: int, + name: str, + *, + recoverable: bool, + ) -> os.stat_result | None: + """Open one raced layout file safely and return its validated metadata.""" try: - media_type = metadata["media_type"] - if not isinstance(media_type, str) or not media_type: - raise ValueError - self._validate_digest(metadata["sha256"]) - byte_count = int(metadata["byte_count"]) - if byte_count < 0: - raise ValueError - return ArtifactStatus( - provider_artifact_id=self._validated_provider_id( - metadata["provider_artifact_id"] - ), - sha256=metadata["sha256"], - byte_count=byte_count, - media_type=media_type, - verification_state=VerificationState(metadata["verification_state"]), - retention_state=RetentionState(metadata["retention_state"]), - availability_state=AvailabilityState(metadata["availability_state"]), - integrity_state=IntegrityState(metadata["integrity_state"]), - active_retentions=tuple( - RetentionSummary( - retention_reference=reference, - retention_class=value["retention_class"], - state="active", - ) - for reference, value in sorted( - metadata.get("retention_references", {}).items() - ) - ), + descriptor = os.open( + name, + os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0), + dir_fd=parent_fd, ) - except ( - ArtifactMalformedRequestError, - ArtifactNotFoundError, - KeyError, - TypeError, - ValueError, - ) as exc: - raise ArtifactIntegrityError("artifact provider metadata is invalid") from exc - - async def _read_json_optional(self, path: Path) -> dict[str, Any] | None: - """Read a regular private JSON file when it exists.""" - try: - return await self._run_io(self._read_json_sync, path) except FileNotFoundError: return None - except (OSError, ValueError, json.JSONDecodeError) as exc: - raise ArtifactIntegrityError("artifact provider metadata is invalid") from exc + try: + if recoverable: + return self._assert_recoverable_object(descriptor) + return self._assert_private_regular(descriptor) + finally: + os.close(descriptor) - async def _read_json_required(self, path: Path) -> dict[str, Any]: - """Read required private JSON without leaking its path.""" - value = await self._read_json_optional(path) - if value is None: - raise ArtifactNotFoundError("artifact is unavailable") - return value + def _provider_object_ref(self, commitment: ArtifactCommitment) -> str: + """Derive one identity-free object reference from canonical SHA-256.""" + digest_hex = commitment.sha256[7:] + return f"sha256/{digest_hex[:2]}/{digest_hex[2:]}" - async def _write_json_exclusive(self, path: Path, value: dict[str, Any]) -> None: - """Atomically publish one immutable private JSON record.""" - await self._run_io(self._write_json_sync, path, value, True) + @staticmethod + def _parse_provider_object_ref(provider_object_ref: str) -> tuple[str, str]: + """Validate the exact opaque local provider-reference grammar.""" + if not isinstance(provider_object_ref, str): + raise ArtifactOperationConflictError("artifact provider reference is invalid") + match = _PROVIDER_OBJECT_REF.fullmatch(provider_object_ref) + if match is None: + raise ArtifactOperationConflictError("artifact provider reference is invalid") + return match.group(1), match.group(2) + + def _open_prefix(self, prefix: str, *, create: bool) -> int: + """Open one digest-prefix directory without following links.""" + if create: + try: + os.mkdir(prefix, 0o700, dir_fd=self._sha256_fd) + except FileExistsError: + pass + try: + descriptor = os.open( + prefix, + os.O_RDONLY + | getattr(os, "O_DIRECTORY", 0) + | getattr(os, "O_NOFOLLOW", 0), + dir_fd=self._sha256_fd, + ) + except OSError as exc: + if exc.errno in {errno.ELOOP, errno.ENOTDIR}: + raise ArtifactIntegrityError("local artifact prefix is unsafe") from None + raise + self._assert_private_directory(descriptor) + if create: + os.fsync(self._sha256_fd) + return descriptor - async def _replace_json(self, path: Path, value: dict[str, Any]) -> None: - """Atomically replace one mutable provider observation record.""" - await self._run_io(self._write_json_sync, path, value, False) + def _create_temporary(self) -> tuple[int, str]: + """Create one unpublished private temporary object.""" + name = f".put.{uuid4().hex}.tmp" + descriptor = os.open( + name, + os.O_CREAT | os.O_EXCL | os.O_WRONLY | getattr(os, "O_NOFOLLOW", 0), + 0o600, + dir_fd=self._tmp_fd, + ) + self._assert_private_regular(descriptor) + return descriptor, name - async def _run_io(self, function: Any, *args: Any) -> Any: - """Complete one blocking filesystem call before propagating cancellation.""" - return await run_blocking_cancellation_resistant(function, *args) + @staticmethod + def _seal_temporary(descriptor: int) -> None: + """Make one complete temporary object read-only and durable.""" + os.fchmod(descriptor, 0o400) + os.fsync(descriptor) + + def _publish_exclusive(self, temporary_name: str, provider_object_ref: str) -> bool: + """Hard-link one complete object without ever replacing existing bytes.""" + prefix, filename = self._parse_provider_object_ref(provider_object_ref) + prefix_fd = self._open_prefix(prefix, create=True) + try: + try: + os.link( + temporary_name, + filename, + src_dir_fd=self._tmp_fd, + dst_dir_fd=prefix_fd, + follow_symlinks=False, + ) + except FileExistsError: + os.unlink(temporary_name, dir_fd=self._tmp_fd) + os.fsync(self._tmp_fd) + return False + os.fsync(prefix_fd) + os.unlink(temporary_name, dir_fd=self._tmp_fd) + os.fsync(self._tmp_fd) + return True + finally: + os.close(prefix_fd) + + def _head_object(self, provider_object_ref: str) -> int: + """Return an exact size after validating one private regular object.""" + descriptor, byte_count = self._open_object(provider_object_ref) + os.close(descriptor) + return byte_count + + def _head_object_with_recovery(self, provider_object_ref: str) -> int: + """Recover an interrupted publication under its digest lock before head.""" + prefix, filename = self._parse_provider_object_ref(provider_object_ref) + with self._locked_file(prefix + filename): + return self._head_object(provider_object_ref) + + def _open_object_with_recovery(self, provider_object_ref: str) -> tuple[int, int]: + """Recover an interrupted publication under its digest lock before open.""" + prefix, filename = self._parse_provider_object_ref(provider_object_ref) + with self._locked_file(prefix + filename): + return self._open_object(provider_object_ref) + + def _open_object(self, provider_object_ref: str) -> tuple[int, int]: + """Open one immutable no-follow object and return its exact size.""" + prefix, filename = self._parse_provider_object_ref(provider_object_ref) + prefix_fd = self._open_prefix(prefix, create=False) + descriptor = -1 + try: + try: + descriptor = os.open( + filename, + os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0), + dir_fd=prefix_fd, + ) + except OSError as exc: + if exc.errno in {errno.ELOOP, errno.ENOTDIR}: + raise ArtifactIntegrityError("local artifact object is unsafe") from None + raise + details = self._assert_recoverable_object(descriptor) + if details.st_nlink == 2: + self._recover_interrupted_publication(descriptor, details, prefix_fd) + details = self._assert_private_regular(descriptor) + if stat.S_IMODE(details.st_mode) != 0o400: + raise ArtifactIntegrityError("local artifact object is mutable") + return descriptor, details.st_size + except BaseException: + if descriptor >= 0: + os.close(descriptor) + raise + finally: + os.close(prefix_fd) - async def _release_locks(self, *locks: tuple[Any, int] | None) -> None: - """Release every owned lock before propagating cancellation or failure.""" - cancellation: asyncio.CancelledError | None = None - failure: Exception | None = None - for lock in locks: - if lock is None: + def _recover_interrupted_publication( + self, + descriptor: int, + object_stat: os.stat_result, + prefix_fd: int, + ) -> None: + """Remove the one sealed temp hard link left by a crashed publisher.""" + matches: list[str] = [] + for name in os.listdir(self._tmp_fd): + if _TEMPORARY_NAME.fullmatch(name) is None: continue try: - await self._run_io(self._release_lock, lock) - except asyncio.CancelledError as exc: - cancellation = cancellation or exc - except Exception as exc: - failure = failure or exc - if cancellation is not None: - raise cancellation - if failure is not None: - raise failure - - async def _acquire_lock_async(self, scope: str) -> tuple[Any, int]: - """Acquire a lock without leaking it when task cancellation wins the race.""" - task = asyncio.create_task(asyncio.to_thread(self._acquire_lock, scope)) + candidate = os.stat(name, dir_fd=self._tmp_fd, follow_symlinks=False) + except FileNotFoundError: + continue + if candidate.st_dev == object_stat.st_dev and candidate.st_ino == object_stat.st_ino: + if ( + not stat.S_ISREG(candidate.st_mode) + or candidate.st_mode & 0o077 + or stat.S_IMODE(candidate.st_mode) != 0o400 + or candidate.st_nlink != 2 + ): + raise ArtifactIntegrityError("local artifact recovery link is unsafe") + matches.append(name) + if len(matches) != 1: + raise ArtifactIntegrityError("local artifact publication is ambiguous") + os.unlink(matches[0], dir_fd=self._tmp_fd) + os.fsync(self._tmp_fd) + os.fsync(prefix_fd) + if os.fstat(descriptor).st_nlink != 1: + raise ArtifactIntegrityError("local artifact publication recovery failed") + + def _verify_exact_under_lock( + self, + provider_object_ref: str, + commitment: ArtifactCommitment, + ) -> None: + """Hash one object while the caller owns its digest lock.""" + descriptor, byte_count = self._open_object(provider_object_ref) + digest = hashlib.sha256() + observed_bytes = 0 try: - return await asyncio.shield(task) - except asyncio.CancelledError as cancellation: - try: - lock = await await_cancellation_resistant(task) - except Exception: - raise cancellation from None - await await_cancellation_resistant(asyncio.to_thread(self._release_lock, lock)) - raise cancellation + while observed_bytes < byte_count: + chunk = os.read(descriptor, min(self._buffer_bytes, byte_count - observed_bytes)) + if not chunk: + raise ArtifactIntegrityError("artifact object was truncated") + observed_bytes += len(chunk) + digest.update(chunk) + finally: + os.close(descriptor) + if ( + observed_bytes != commitment.byte_count + or f"sha256:{digest.hexdigest()}" != commitment.sha256 + ): + raise ArtifactIntegrityError("artifact object violates commitment") @contextmanager - def _locked_file(self, path: Path) -> Iterator[int]: - """Open and exclusively lock one private regular file.""" - flags = os.O_CREAT | os.O_RDWR | getattr(os, "O_NOFOLLOW", 0) - directory = self._open_directory_fd(path.parent) - descriptor: int | None = None + def _locked_file(self, digest_hex: str) -> Iterator[int]: + """Own one bounded cross-process lock for a canonical digest.""" + descriptor = os.open( + f"{digest_hex}.lock", + os.O_CREAT | os.O_RDWR | getattr(os, "O_NOFOLLOW", 0), + 0o600, + dir_fd=self._locks_fd, + ) try: - descriptor = os.open(path.name, flags, 0o600, dir_fd=directory) - self._assert_private_descriptor(descriptor) + self._assert_private_regular(descriptor) try: acquire_exclusive_file_lock( descriptor, @@ -1414,245 +950,97 @@ def _locked_file(self, path: Path) -> Iterator[int]: ) from None yield descriptor finally: - if descriptor is not None: - try: - fcntl.flock(descriptor, fcntl.LOCK_UN) - finally: - os.close(descriptor) - os.close(directory) + try: + fcntl.flock(descriptor, fcntl.LOCK_UN) + finally: + os.close(descriptor) - def _acquire_lock(self, scope: str) -> tuple[Any, int]: - """Acquire a cross-process operation lock.""" - manager = self._locked_file(self._path("locks", scope, ".lock")) + def _acquire_lock(self, digest_hex: str) -> tuple[Any, int]: + """Acquire one validated digest lock.""" + if re.fullmatch(r"[0-9a-f]{64}", digest_hex) is None: + raise ArtifactOperationConflictError("artifact commitment is invalid") + manager = self._locked_file(digest_hex) descriptor = manager.__enter__() return manager, descriptor @staticmethod def _release_lock(lock: tuple[Any, int]) -> None: - """Release a cross-process operation lock.""" + """Release one owned digest lock.""" manager, _ = lock manager.__exit__(None, None, None) - def _open_exclusive(self, path: Path) -> int: - """Open a new no-follow private file.""" - directory = self._open_directory_fd(path.parent) + def _cleanup_resources( + self, + descriptor: int | None, + temporary_name: str | None, + lock: tuple[Any, int] | None, + ) -> None: + """Release all private put resources after failure or cancellation.""" try: - descriptor = os.open( - path.name, - os.O_CREAT | os.O_EXCL | os.O_WRONLY | getattr(os, "O_NOFOLLOW", 0), - 0o600, - dir_fd=directory, - ) - self._assert_private_descriptor(descriptor) - return descriptor + if descriptor is not None: + try: + os.close(descriptor) + except OSError: + pass + if temporary_name is not None and _TEMPORARY_NAME.fullmatch(temporary_name): + try: + os.unlink(temporary_name, dir_fd=self._tmp_fd) + os.fsync(self._tmp_fd) + except FileNotFoundError: + pass + except OSError: + pass finally: - os.close(directory) + if lock is not None: + try: + self._release_lock(lock) + except OSError: + pass @staticmethod def _write_all(descriptor: int, chunk: memoryview) -> None: - """Write one bounded memory view completely.""" + """Write one already-bounded view completely.""" remaining = chunk while remaining: written = os.write(descriptor, remaining) if written <= 0: - raise OSError("short artifact write") + raise OSError(errno.EIO, "short artifact write") remaining = remaining[written:] - def _publish_exclusive(self, temporary: Path, final: Path) -> None: - """Publish without overwriting an immutable object.""" - source_directory = self._open_directory_fd(temporary.parent) - target_directory = self._open_directory_fd(final.parent) - try: - os.link( - temporary.name, - final.name, - src_dir_fd=source_directory, - dst_dir_fd=target_directory, - follow_symlinks=False, - ) - os.unlink(temporary.name, dir_fd=source_directory) - os.fsync(source_directory) - os.fsync(target_directory) - finally: - os.close(source_directory) - os.close(target_directory) - - def _unlink_optional(self, path: Path) -> None: - """Remove a private temporary file when present.""" - directory = self._open_directory_fd(path.parent) - try: - try: - os.unlink(path.name, dir_fd=directory) - except FileNotFoundError: - pass - finally: - os.close(directory) - - def _open_regular_object(self, provider_id: str) -> BinaryIO: - """Open one no-follow regular object with exactly one link.""" - path = self._path("objects", provider_id, ".blob") - directory = self._open_directory_fd(path.parent) - try: - descriptor = os.open( - path.name, - os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0), - dir_fd=directory, - ) - finally: - os.close(directory) - self._assert_private_descriptor(descriptor) - return os.fdopen(descriptor, "rb", buffering=0) - - def _assert_regular_file(self, path: Path) -> None: - """Require a no-follow regular file with one hard link.""" - directory = self._open_directory_fd(path.parent) - descriptor: int | None = None - try: - descriptor = os.open( - path.name, - os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0), - dir_fd=directory, - ) - self._assert_private_descriptor(descriptor) - finally: - if descriptor is not None: - os.close(descriptor) - os.close(directory) - - def _read_json_sync(self, path: Path) -> dict[str, Any]: - """Read bounded JSON from a no-follow regular file.""" - directory = self._open_directory_fd(path.parent) - descriptor: int | None = None - try: - descriptor = os.open( - path.name, - os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0), - dir_fd=directory, - ) - self._assert_private_descriptor(descriptor) - raw = os.read(descriptor, 65_537) - if len(raw) > 65_536 or os.read(descriptor, 1): - raise ValueError("provider metadata exceeds limit") - finally: - if descriptor is not None: - os.close(descriptor) - os.close(directory) - value = json.loads(raw.decode("utf-8")) - if not isinstance(value, dict): - raise ValueError("provider metadata must be an object") - return value - - def _write_json_sync(self, path: Path, value: dict[str, Any], exclusive: bool) -> None: - """Atomically publish and synchronize one private JSON file.""" - raw = json.dumps( - value, sort_keys=True, separators=(",", ":"), ensure_ascii=False, allow_nan=False - ).encode("utf-8") - if len(raw) > 65_536: - raise ValueError("provider metadata exceeds limit") - temporary = path.with_name(f".{path.name}.{uuid4().hex}.tmp") - descriptor = self._open_exclusive(temporary) - try: - self._write_all(descriptor, memoryview(raw)) - os.fsync(descriptor) - finally: - os.close(descriptor) - try: - if exclusive: - source_directory = self._open_directory_fd(temporary.parent) - target_directory = self._open_directory_fd(path.parent) - try: - os.link( - temporary.name, - path.name, - src_dir_fd=source_directory, - dst_dir_fd=target_directory, - follow_symlinks=False, - ) - os.unlink(temporary.name, dir_fd=source_directory) - os.fsync(source_directory) - os.fsync(target_directory) - finally: - os.close(source_directory) - os.close(target_directory) - else: - self._assert_regular_file(path) - source_directory = self._open_directory_fd(temporary.parent) - target_directory = self._open_directory_fd(path.parent) - try: - os.replace( - temporary.name, - path.name, - src_dir_fd=source_directory, - dst_dir_fd=target_directory, - ) - os.fsync(source_directory) - os.fsync(target_directory) - finally: - os.close(source_directory) - os.close(target_directory) - except BaseException: - self._unlink_optional(temporary) - raise - - def _fsync_directory(self, path: Path) -> None: - """Synchronize one directory entry update.""" - descriptor = self._open_directory_fd(path) - try: - os.fsync(descriptor) - finally: - os.close(descriptor) - - def _move_to_quarantine(self, source: Path, target: Path) -> None: - """Move a provider object to private quarantine without overwriting.""" - try: - source_directory = self._open_directory_fd(source.parent) - target_directory = self._open_directory_fd(target.parent) - try: - os.link( - source.name, - target.name, - src_dir_fd=source_directory, - dst_dir_fd=target_directory, - follow_symlinks=False, - ) - os.unlink(source.name, dir_fd=source_directory) - os.fsync(source_directory) - os.fsync(target_directory) - finally: - os.close(source_directory) - os.close(target_directory) - except FileNotFoundError: - try: - self._assert_regular_file(target) - except FileNotFoundError: - raise + @staticmethod + def _assert_private_directory(descriptor: int) -> os.stat_result: + """Require one owner-private directory descriptor.""" + details = os.fstat(descriptor) + if ( + not stat.S_ISDIR(details.st_mode) + or details.st_uid != os.geteuid() + or stat.S_IMODE(details.st_mode) != 0o700 + ): + raise ArtifactConfigurationError("local artifact directory is unsafe") + return details - def _open_directory_fd(self, path: Path) -> int: - """Open a verified storage subdirectory relative to the pinned root.""" - if path.parent != self._root or path.name not in { - "objects", - "metadata", - "receipts", - "intents", - "locks", - "tmp", - "quarantine", - }: - raise ArtifactIntegrityError("artifact provider directory is invalid") - descriptor = os.open( - path.name, - os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0), - dir_fd=self._root_fd, - ) + @staticmethod + def _assert_private_regular(descriptor: int) -> os.stat_result: + """Require one owner-private, single-link regular file.""" details = os.fstat(descriptor) - if not stat_module.S_ISDIR(details.st_mode): - os.close(descriptor) - raise ArtifactIntegrityError("artifact provider directory is invalid") - return descriptor + if ( + not stat.S_ISREG(details.st_mode) + or details.st_uid != os.geteuid() + or details.st_mode & 0o077 + or details.st_nlink != 1 + ): + raise ArtifactIntegrityError("local artifact object is unsafe") + return details @staticmethod - def _assert_private_descriptor(descriptor: int) -> None: - """Require an opened provider file to be regular and single-linked.""" + def _assert_recoverable_object(descriptor: int) -> os.stat_result: + """Allow only the one extra link produced by interrupted publication.""" details = os.fstat(descriptor) - if not stat_module.S_ISREG(details.st_mode) or details.st_nlink != 1: - raise ArtifactIntegrityError("artifact provider file is invalid") + if ( + not stat.S_ISREG(details.st_mode) + or details.st_uid != os.geteuid() + or details.st_mode & 0o077 + or details.st_nlink not in {1, 2} + ): + raise ArtifactIntegrityError("local artifact object is unsafe") + return details diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 2c42d290d..8685880be 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -91,9 +91,8 @@ class Settings(BaseSettings): api_first_access_rate_window_seconds: int = Field(default=60, ge=1, le=3_600) api_admin_mutation_rate_limit: int = Field(default=30, ge=1, le=10_000) api_admin_mutation_rate_window_seconds: int = Field(default=60, ge=1, le=3_600) - artifact_store_backend: Literal["disabled", "local", "flow_node"] = "disabled" + artifact_store_backend: Literal["disabled", "local", "s3_compatible"] = "disabled" artifact_local_root: Path | None = None - artifact_retention_policy_version: str | None = None artifact_maximum_bytes: int = Field(default=512 * 1024 * 1024, gt=0) artifact_stream_buffer_bytes: int = Field(default=1024 * 1024, gt=0, le=1024 * 1024) artifact_operation_lock_timeout_seconds: float = Field( @@ -127,6 +126,11 @@ class Settings(BaseSettings): gt=0.0, le=3600.0, ) + artifact_scratch_cleanup_interval_seconds: int = Field( + default=300, + gt=0, + le=86_400, + ) model_config = SettingsConfigDict( env_prefix="WORKSTREAM_", @@ -236,10 +240,8 @@ def validate_artifact_storage(self) -> Settings: ) if self.artifact_store_backend == "disabled": return self - if not self.artifact_retention_policy_version or not ( - self.artifact_retention_policy_version.strip() - ): - raise ValueError("enabled artifact storage requires a retention policy version") + if self.artifact_scratch_root is None: + raise ValueError("enabled artifact storage requires an artifact scratch root") if self.artifact_store_backend == "local": if self.environment not in {"local", "dev", "development", "test"}: raise ValueError("local artifact storage is restricted to development and test") diff --git a/backend/app/interfaces/artifact_operations.py b/backend/app/interfaces/artifact_operations.py new file mode 100644 index 000000000..523095cdc --- /dev/null +++ b/backend/app/interfaces/artifact_operations.py @@ -0,0 +1,290 @@ +"""Closed product capabilities owned by artifact orchestration.""" + +from __future__ import annotations + +from collections.abc import AsyncIterable +from dataclasses import dataclass +from typing import Literal, Protocol +from uuid import UUID + +from app.modules.artifacts.sources import ArtifactCommitment +from app.modules.authorization.runtime import AuthorizationContext + +__all__ = ( + "ArtifactAuditResourceType", + "ArtifactBindingResourceType", + "ArtifactBindingCreateRequest", + "ArtifactBindingPort", + "ArtifactMaterializationPort", + "ArtifactOperatorReadPort", + "ArtifactOperatorRecoveryPort", + "ArtifactRecoveryRequest", + "BindingMaterializationRequest", + "CheckerArtifactOutputPort", + "CheckerOutputArtifactRequest", + "ContributorArtifactUploadPort", + "GuideArtifactIngestPort", + "GuideArtifactIngestRequest", + "ReadyUploadSetRequest", +) + +ArtifactBindingResourceType = Literal[ + "project", + "project_guide", + "guide_source_snapshot", + "guide_source_snapshot_item", + "task", + "submission", + "checker_run", + "review", +] +ArtifactAuditResourceType = Literal[ + "artifact_binding", + "artifact_content", + "artifact_replica", + "artifact_receipt", + "artifact_verification_job", + "artifact_recovery_attempt", +] + + +@dataclass(frozen=True, slots=True) +class GuideArtifactIngestRequest: + """Authorized guide-source bytes and their canonical product ownership.""" + + authorization_context: AuthorizationContext + project_id: UUID + guide_source_snapshot_id: UUID + source_item_id: UUID + logical_role: str + byte_source: AsyncIterable[bytes] + client_commitment: ArtifactCommitment | None = None + + +@dataclass(frozen=True, slots=True) +class ArtifactBindingCreateRequest: + """Verified content and exact product facts for immutable binding.""" + + authorization_context: AuthorizationContext + project_id: UUID + task_id: UUID + submission_id: UUID | None + checker_run_id: UUID | None + logical_role: str + verified_content_ids: tuple[UUID, ...] + + +@dataclass(frozen=True, slots=True) +class ReadyUploadSetRequest: + """One sealed upload set and its locked task policy context.""" + + authorization_context: AuthorizationContext + task_id: UUID + sealed_upload_session_id: UUID + submission_artifact_policy_id: UUID + checker_policy_id: UUID + + +@dataclass(frozen=True, slots=True) +class BindingMaterializationRequest: + """Immutable bindings selected by exact execution context.""" + + authorization_context: AuthorizationContext + task_id: UUID + submission_id: UUID | None + checker_run_id: UUID + binding_ids: tuple[UUID, ...] + + +@dataclass(frozen=True, slots=True) +class CheckerOutputArtifactRequest: + """Generated checker bytes bound to one fixed service execution.""" + + service_actor_context: AuthorizationContext + task_id: UUID + submission_id: UUID + checker_run_id: UUID + logical_role: str + byte_source: AsyncIterable[bytes] + + +@dataclass(frozen=True, slots=True) +class ArtifactRecoveryRequest: + """Reason-bound Operator retry of one exact verification job.""" + + authorization_context: AuthorizationContext + project_id: UUID + task_id: UUID + submission_id: UUID | None + source_verification_job_id: UUID + reason: str + client_idempotency_key: str + expected_source_job_cas_version: int + + +class GuideArtifactIngestPort(Protocol): + """Ingest authorized guide bytes without exposing provider operations.""" + + async def ingest(self, request: GuideArtifactIngestRequest) -> object: + """Ingest one canonical guide source item.""" + + +class ContributorArtifactUploadPort(Protocol): + """Own the closed contributor upload-session lifecycle.""" + + async def create( + self, + *, + authorization_context: AuthorizationContext, + task_id: UUID, + ) -> object: + """Create one authorized task upload session.""" + + async def read( + self, + *, + authorization_context: AuthorizationContext, + task_id: UUID, + upload_session_id: UUID, + ) -> object: + """Read one authorized upload session.""" + + async def write( + self, + *, + authorization_context: AuthorizationContext, + task_id: UUID, + upload_session_id: UUID, + logical_role: str, + byte_source: AsyncIterable[bytes], + client_commitment: ArtifactCommitment | None = None, + ) -> object: + """Write one bounded item through artifact orchestration.""" + + async def seal( + self, + *, + authorization_context: AuthorizationContext, + task_id: UUID, + upload_session_id: UUID, + ) -> object: + """Seal one exact upload set.""" + + async def cancel( + self, + *, + authorization_context: AuthorizationContext, + task_id: UUID, + upload_session_id: UUID, + ) -> None: + """Cancel one unsealed upload session.""" + + +class ArtifactBindingPort(Protocol): + """Create bindings only from orchestrator-verified content.""" + + async def bind_verified(self, request: ArtifactBindingCreateRequest) -> object: + """Bind exact verified content to canonical product facts.""" + + +class ArtifactMaterializationPort(Protocol): + """Materialize only the two canonical immutable source forms.""" + + async def materialize_ready_upload_set( + self, + request: ReadyUploadSetRequest, + ) -> object: + """Materialize one sealed upload set whose items are ready.""" + + async def materialize_bindings( + self, + request: BindingMaterializationRequest, + ) -> object: + """Materialize exact immutable binding IDs.""" + + +class CheckerArtifactOutputPort(Protocol): + """Store generated output for one fixed checker execution.""" + + async def store(self, request: CheckerOutputArtifactRequest) -> object: + """Store one generated checker artifact.""" + + +class ArtifactOperatorReadPort(Protocol): + """Expose bounded Operator reads without provider references.""" + + async def list_bindings( + self, + *, + authorization_context: AuthorizationContext, + resource_type: ArtifactBindingResourceType, + resource_id: UUID, + cursor: str | None, + limit: int, + ) -> object: + """List bindings for one exact canonical product resource.""" + + async def list_replicas( + self, + *, + authorization_context: AuthorizationContext, + content_id: UUID, + cursor: str | None, + limit: int, + ) -> object: + """List replicas for one exact Workstream content identity.""" + + async def list_receipts( + self, + *, + authorization_context: AuthorizationContext, + replica_id: UUID, + cursor: str | None, + limit: int, + ) -> object: + """List receipts for one exact Workstream replica identity.""" + + async def get_verification_job( + self, + *, + authorization_context: AuthorizationContext, + verification_job_id: UUID, + ) -> object: + """Read one exact verification job.""" + + async def get_recovery_attempt( + self, + *, + authorization_context: AuthorizationContext, + recovery_attempt_id: UUID, + ) -> object: + """Read one exact recovery attempt.""" + + async def list_audit_events( + self, + *, + authorization_context: AuthorizationContext, + resource_type: ArtifactAuditResourceType, + resource_id: UUID, + cursor: str | None, + limit: int, + ) -> object: + """List bounded artifact audit events for one exact resource.""" + + async def admission_usage( + self, + *, + authorization_context: AuthorizationContext, + project_id: UUID | None, + task_id: UUID | None, + cursor: str | None, + limit: int, + ) -> object: + """Read bounded reserved/completed byte usage and configured limits.""" + + +class ArtifactOperatorRecoveryPort(Protocol): + """Expose only reason-bound verification retry to Operators.""" + + async def retry_verification(self, request: ArtifactRecoveryRequest) -> object: + """Retry one exact source verification job under CAS fencing.""" diff --git a/backend/app/interfaces/artifacts.py b/backend/app/interfaces/artifacts.py index bbb2f7ab7..9fbd6aa60 100644 --- a/backend/app/interfaces/artifacts.py +++ b/backend/app/interfaces/artifacts.py @@ -1,173 +1,188 @@ -"""Provider-neutral contracts for immutable artifact storage.""" +"""Provider-neutral contracts for immutable artifact byte storage.""" from __future__ import annotations -from collections.abc import AsyncIterable, AsyncIterator, Mapping +from collections.abc import AsyncIterator from dataclasses import dataclass -from datetime import datetime -from enum import StrEnum from typing import Protocol from app.core.hashing import canonical_json_hash - - -class ArtifactOperation(StrEnum): - """Storage operations with independent idempotency scopes.""" - - STORE = "store" - VERIFY = "verify" - RETAIN = "retain" - RELEASE = "release" - - -class VerificationState(StrEnum): - """Provider observation of full-object verification.""" - - PENDING = "pending" - VERIFIED = "verified" - FAILED = "failed" - - -class RetentionState(StrEnum): - """Provider observation of active retention references.""" - - UNRETAINED = "unretained" - RETAINED = "retained" - RELEASED = "released" - - -class AvailabilityState(StrEnum): - """Provider observation of object availability.""" - - AVAILABLE = "available" - UNAVAILABLE = "unavailable" - MISSING = "missing" - - -class IntegrityState(StrEnum): - """Provider observation of object integrity.""" - - UNKNOWN = "unknown" - VALID = "valid" - QUARANTINED = "quarantined" - - -class ReceiptOutcome(StrEnum): - """Successful immutable provider operation facts.""" - - STORED = "stored" - VERIFIED = "verified" - RETAINED = "retained" - RELEASED = "released" - - -@dataclass(frozen=True, slots=True) -class IdempotencyIdentity: - """Complete identity and commitment for one provider operation.""" - - service_principal: str - operation: ArtifactOperation - key: str - request_digest: str +from app.interfaces.external_services import ( + ExternalServiceAdapter, + ExternalServiceAdapterIdentity, +) +from app.modules.artifacts.sources import ArtifactCommitment, CommittedArtifactSource + + +ARTIFACT_STORE_CAPABILITY_KEY = "artifact_store" +_MAXIMUM_PROVIDER_OBJECT_REF_LENGTH = 1024 +_RESERVED_NAMESPACE_DESCRIPTOR_KEYS = frozenset( + {"adapter", "backend", "provider_profile"} +) +_NAMESPACE_DESCRIPTOR_KEYS_BY_PROFILE = { + "local-v2": frozenset({"private_prefix", "private_root_identity"}), +} + + +def _validate_provider_object_ref(provider_object_ref: str) -> None: + """Require one bounded opaque provider reference without control characters.""" + if ( + not isinstance(provider_object_ref, str) + or not provider_object_ref + or len(provider_object_ref) > _MAXIMUM_PROVIDER_OBJECT_REF_LENGTH + or any(ord(character) < 32 or ord(character) == 127 for character in provider_object_ref) + ): + raise ValueError("artifact provider object reference is invalid") @dataclass(frozen=True, slots=True) class ArtifactByteRange: - """Zero-based byte range with an exclusive end.""" + """Zero-based byte range whose optional length defines an exclusive end.""" offset: int = 0 length: int | None = None def __post_init__(self) -> None: - """Reject negative offsets and lengths.""" - if self.offset < 0 or (self.length is not None and self.length < 0): - raise ValueError("artifact byte range values must be nonnegative") + """Reject negative, boolean, and non-integer range values.""" + if type(self.offset) is not int or self.offset < 0: + raise ValueError("artifact byte range offset is invalid") + if self.length is not None and (type(self.length) is not int or self.length < 0): + raise ValueError("artifact byte range length is invalid") @dataclass(frozen=True, slots=True) -class StoreArtifactRequest: - """Immutable metadata for one streamed store operation.""" +class ArtifactPutResult: + """Provider-neutral result of immutable publication or an exact replay.""" - expected_sha256: str | None - expected_size: int | None - maximum_bytes: int - media_type: str - metadata: Mapping[str, str] - idempotency: IdempotencyIdentity + provider_object_ref: str + replayed: bool + + def __post_init__(self) -> None: + """Validate the bounded provider result.""" + _validate_provider_object_ref(self.provider_object_ref) + if type(self.replayed) is not bool: + raise ValueError("artifact put replay observation is invalid") @dataclass(frozen=True, slots=True) -class ReceiptDetail: - """One bounded provider receipt detail entry.""" - - name: str - value: str +class ArtifactPutObservation: + """Read-only exact observation of one commitment-derived provider object.""" + provider_object_ref: str + committed: bool -@dataclass(frozen=True, slots=True) -class OperationReceipt: - """Bounded provider evidence for one authoritative operation.""" - - receipt_id: str - provider_operation_reference: str - identity: IdempotencyIdentity - response_digest: str - outcome: ReceiptOutcome - provider_artifact_id: str | None - sha256: str | None - byte_count: int | None - recorded_at: datetime - retention_reference: str | None = None - retention_class: str | None = None - details: tuple[ReceiptDetail, ...] = () + def __post_init__(self) -> None: + """Validate the bounded provider observation.""" + _validate_provider_object_ref(self.provider_object_ref) + if type(self.committed) is not bool: + raise ValueError("artifact put observation is invalid") @dataclass(frozen=True, slots=True) -class RetentionSummary: - """One provider retention reference and its current state.""" +class ArtifactObjectHead: + """Safe provider observation for an existing or missing immutable object.""" - retention_reference: str - retention_class: str - state: str + provider_object_ref: str + exists: bool + byte_count: int | None = None + media_type: str | None = None - -@dataclass(frozen=True, slots=True) -class StoredArtifact: - """Provider-neutral result of a successful store operation.""" - - provider_artifact_id: str - sha256: str - byte_count: int - media_type: str - receipt: OperationReceipt - replayed: bool + def __post_init__(self) -> None: + """Keep missing and existing head observations unambiguous.""" + _validate_provider_object_ref(self.provider_object_ref) + if type(self.exists) is not bool: + raise ValueError("artifact head existence observation is invalid") + if self.exists: + if type(self.byte_count) is not int or self.byte_count < 0: + raise ValueError("existing artifact head requires an exact byte count") + elif self.byte_count is not None or self.media_type is not None: + raise ValueError("missing artifact head cannot contain object metadata") + if self.media_type is not None: + ArtifactCommitment.validate_media_type(self.media_type) @dataclass(frozen=True, slots=True) -class ArtifactOperationResult: - """One immutable receipt plus response-time replay information.""" +class ArtifactStoreNamespaceIdentity: + """Provider-owned, non-secret identity for one immutable byte namespace.""" - receipt: OperationReceipt - replayed: bool + provider_profile: str + descriptor_items: tuple[tuple[str, str], ...] + + def __post_init__(self) -> None: + """Require a canonical closed descriptor without duplicate keys.""" + if not self.provider_profile or len(self.provider_profile) > 100: + raise ValueError("artifact provider profile is invalid") + if type(self.descriptor_items) is not tuple: + raise ValueError("artifact namespace descriptor is not immutable") + expected_keys = _NAMESPACE_DESCRIPTOR_KEYS_BY_PROFILE.get(self.provider_profile) + if expected_keys is None: + raise ValueError("artifact provider profile is unsupported") + keys: list[str] = [] + for item in self.descriptor_items: + if ( + type(item) is not tuple + or len(item) != 2 + or not all(isinstance(value, str) and value for value in item) + ): + raise ValueError("artifact namespace descriptor is invalid") + keys.append(item[0]) + if ( + keys != sorted(keys) + or len(keys) != len(set(keys)) + or not _RESERVED_NAMESPACE_DESCRIPTOR_KEYS.isdisjoint(keys) + or set(keys) != expected_keys + ): + raise ValueError("artifact namespace descriptor is not canonical") + + def as_dict(self) -> dict[str, str]: + """Return a fresh JSON-compatible descriptor projection.""" + return dict(self.descriptor_items) + + +def artifact_store_namespace_material( + *, + backend: str, + adapter_identity: ExternalServiceAdapterIdentity, + namespace_identity: ArtifactStoreNamespaceIdentity, +) -> tuple[dict[str, object], str]: + """Build one canonical namespace descriptor and its matching fingerprint.""" + descriptor: dict[str, object] = { + "backend": backend, + "adapter": adapter_identity.provider_key, + "provider_profile": namespace_identity.provider_profile, + **namespace_identity.as_dict(), + } + return descriptor, canonical_json_hash(descriptor) @dataclass(frozen=True, slots=True) -class ArtifactStatus: - """Current provider observations for an immutable object.""" +class ArtifactStoreNamespaceClaim: + """Proof that PostgreSQL accepted one exact adapter namespace identity.""" - provider_artifact_id: str - sha256: str - byte_count: int - media_type: str - verification_state: VerificationState - retention_state: RetentionState - availability_state: AvailabilityState - integrity_state: IntegrityState - active_retentions: tuple[RetentionSummary, ...] = () + adapter_identity: ExternalServiceAdapterIdentity + namespace_identity: ArtifactStoreNamespaceIdentity + namespace_fingerprint: str + + def __post_init__(self) -> None: + """Reject malformed or mismatched startup proof values.""" + if type(self.adapter_identity) is not ExternalServiceAdapterIdentity: + raise ValueError("artifact namespace adapter identity is invalid") + if type(self.namespace_identity) is not ArtifactStoreNamespaceIdentity: + raise ValueError("artifact namespace identity is invalid") + if ( + not isinstance(self.namespace_fingerprint, str) + or not self.namespace_fingerprint.startswith("sha256:") + or len(self.namespace_fingerprint) != 71 + or any( + character not in "0123456789abcdef" + for character in self.namespace_fingerprint[7:] + ) + ): + raise ValueError("artifact namespace fingerprint is invalid") class ArtifactStoreError(Exception): - """Base class for stable artifact-store failures.""" + """Base class for stable, provider-secret-safe artifact failures.""" code = "artifact_storage_error" category = "terminal" @@ -175,184 +190,104 @@ class ArtifactStoreError(Exception): class ArtifactStoreUnavailableError(ArtifactStoreError): - """Raised when a retryable provider operation is unavailable.""" + """Raised when provider availability prevents an exact result.""" code = "artifact_storage_unavailable" category = "retryable" retryable = True -class ArtifactMalformedRequestError(ArtifactStoreError): - """Raised when a provider request is structurally invalid.""" +class ArtifactObjectMissingError(ArtifactStoreError): + """Raised when an operation requires an object that does not exist.""" - code = "artifact_malformed_request" + code = "artifact_object_missing" + category = "not_found" -class ArtifactInputMismatchError(ArtifactStoreError): - """Raised when supplied bytes violate their initial commitment.""" +class ArtifactPreconditionFailedError(ArtifactStoreError): + """Raised when immutable provider publication loses its condition.""" - code = "artifact_input_mismatch" - category = "input_mismatch" + code = "artifact_precondition_failed" + category = "conflict" -class ArtifactLimitExceededError(ArtifactStoreError): - """Raised when bounded artifact input exceeds a declared limit.""" +class ArtifactInputMismatchError(ArtifactStoreError): + """Raised when supplied bytes violate their server-owned commitment.""" - code = "artifact_limit_exceeded" - category = "terminal" + code = "artifact_input_mismatch" + category = "input_mismatch" class ArtifactIntegrityError(ArtifactStoreError): - """Raised when persisted or replayed provider facts are inconsistent.""" + """Raised when provider or prepared bytes violate immutable facts.""" code = "artifact_integrity_failure" category = "integrity" -class ArtifactIdempotencyMismatchError(ArtifactStoreError): - """Raised when one operation identity is reused with a changed request.""" - - code = "artifact_idempotency_mismatch" - - -class ArtifactNotFoundError(ArtifactStoreError): - """Raised when an opaque provider artifact is unknown or unavailable.""" +class ArtifactRangeInvalidError(ArtifactStoreError): + """Raised when a requested range starts past the object end.""" - code = "artifact_not_found" - category = "not_found" + code = "artifact_range_invalid" + category = "input_mismatch" -class ArtifactRetentionConflictError(ArtifactStoreError): - """Raised when an exact retention reference cannot be changed.""" +class ArtifactOperationConflictError(ArtifactStoreError): + """Raised when an opaque provider reference is invalid for an operation.""" - code = "artifact_retention_conflict" - category = "input_mismatch" + code = "artifact_operation_conflict" + category = "conflict" -class ArtifactThrottledError(ArtifactStoreError): - """Raised when a retryable provider throttle prevents an operation.""" +class ArtifactLimitExceededError(ArtifactStoreError): + """Raised when bounded artifact input exceeds its hard limit.""" - code = "artifact_throttled" - category = "retryable" - retryable = True + code = "artifact_limit_exceeded" + category = "terminal" class ArtifactConfigurationError(ArtifactStoreError): - """Raised when a configured artifact adapter cannot be resolved.""" + """Raised when artifact adapter construction or selection is invalid.""" code = "artifact_storage_configuration_invalid" + category = "configuration" -class ArtifactStore(Protocol): - """Port implemented by development and production artifact providers.""" +class ArtifactStore(ExternalServiceAdapter, Protocol): + """Immutable byte-provider capability used only by artifact orchestration.""" - async def store( - self, - stream: AsyncIterable[bytes], - request: StoreArtifactRequest, - ) -> StoredArtifact: - """Store one bounded stream or return its exact replay result.""" + async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: + """Publish one sealed committed source or prove an exact replay.""" - async def recover_committed_store( - self, request: StoreArtifactRequest - ) -> StoredArtifact: - """Recover one committed store effect from exact persisted commitments.""" + async def observe_put_result( + self, + commitment: ArtifactCommitment, + ) -> ArtifactPutObservation: + """Observe one commitment-derived object without replaying a write.""" def open( self, - provider_artifact_id: str, + provider_object_ref: str, byte_range: ArtifactByteRange | None = None, ) -> AsyncIterator[bytes]: - """Stream one available object or range in bounded chunks.""" + """Stream one object or bounded range in bounded chunks.""" - async def stat(self, provider_artifact_id: str) -> ArtifactStatus: - """Return current provider observations for one object.""" + async def head(self, provider_object_ref: str) -> ArtifactObjectHead: + """Return a safe existing or missing object observation.""" - async def verify( - self, - provider_artifact_id: str, - expected_sha256: str, - expected_size: int, - idempotency: IdempotencyIdentity, - ) -> ArtifactOperationResult: - """Independently hash and count one complete provider object.""" - - async def retain( - self, - provider_artifact_id: str, - retention_reference: str, - retention_class: str, - idempotency: IdempotencyIdentity, - ) -> ArtifactOperationResult: - """Add an exact reference-counted retention intent.""" - - async def release( - self, - provider_artifact_id: str, - retention_reference: str, - idempotency: IdempotencyIdentity, - ) -> ArtifactOperationResult: - """Release only the exact retention reference supplied.""" - async def get_operation_receipt( +class ArtifactStoreBootstrap(ExternalServiceAdapter, Protocol): + """Composition-only lifecycle that cannot publish bytes before DB claim.""" + + @property + def namespace_identity(self) -> ArtifactStoreNamespaceIdentity: + """Return the identity of the already-open provider namespace.""" + + def initialize_after_namespace_claim( self, - service_principal: str, - operation: ArtifactOperation, - idempotency_key: str, - ) -> OperationReceipt | None: - """Load the authoritative receipt for one complete operation identity.""" - - -def canonical_store_request_digest(request: StoreArtifactRequest) -> str: - """Return the canonical digest for all authoritative store request fields.""" - payload: dict[str, object] = { - "maximum_bytes": request.maximum_bytes, - "media_type": request.media_type, - "metadata": [ - {"name": name, "value": value} - for name, value in sorted(request.metadata.items()) - ], - } - if request.expected_sha256 is not None: - payload["expected_sha256"] = request.expected_sha256 - if request.expected_size is not None: - payload["expected_size"] = request.expected_size - return canonical_json_hash(payload) - - -def canonical_verify_request_digest( - provider_artifact_id: str, expected_sha256: str, expected_size: int -) -> str: - """Return the canonical digest for one verify request.""" - return canonical_json_hash( - { - "provider_artifact_id": provider_artifact_id, - "expected_sha256": expected_sha256, - "expected_size": expected_size, - } - ) - - -def canonical_retain_request_digest( - provider_artifact_id: str, retention_reference: str, retention_class: str -) -> str: - """Return the canonical digest for one retain request.""" - return canonical_json_hash( - { - "provider_artifact_id": provider_artifact_id, - "retention_reference": retention_reference, - "retention_class": retention_class, - } - ) - - -def canonical_release_request_digest( - provider_artifact_id: str, retention_reference: str -) -> str: - """Return the canonical digest for one release request.""" - return canonical_json_hash( - { - "provider_artifact_id": provider_artifact_id, - "retention_reference": retention_reference, - } - ) + claim: ArtifactStoreNamespaceClaim, + ) -> ArtifactStore: + """Initialize provider layout only after accepting the exact DB claim.""" + + def close(self) -> None: + """Release provider startup and operation resources.""" diff --git a/backend/app/main.py b/backend/app/main.py index a95c4f829..e031ab921 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -14,6 +14,10 @@ from starlette.exceptions import HTTPException as StarletteHTTPException from app.api.router import api_router +from app.adapters.artifacts import ( + cleanup_stale_artifact_scratch, + create_artifact_store_bootstrap, +) from app.core.api_controls import ( ApiErrorResponse, RequestContextMiddleware, @@ -23,6 +27,7 @@ ) from app.core.auth import build_auth_verifier, cache_auth_verifier, prepare_auth_verifier from app.core.config import Settings, get_settings +from app.interfaces.artifacts import ArtifactStoreBootstrap, ArtifactStoreNamespaceClaim PRODUCTION_LIKE_ENVIRONMENTS = {"staging", "preview", "prod", "production"} MAX_VALIDATION_ERRORS = 20 @@ -60,9 +65,38 @@ async def _application_lifespan(app: FastAPI) -> AsyncIterator[None]: and not app.state.auth_configuration_valid ): build_auth_verifier(settings) + if settings.artifact_store_backend != "disabled": + bootstrap = create_artifact_store_bootstrap(settings) + try: + claim = await _validate_artifact_storage_namespace_at_startup( + settings, + bootstrap, + ) + bootstrap.initialize_after_namespace_claim(claim) + await cleanup_stale_artifact_scratch(settings) + finally: + bootstrap.close() yield +async def _validate_artifact_storage_namespace_at_startup( + settings: Settings, + bootstrap: ArtifactStoreBootstrap, +) -> ArtifactStoreNamespaceClaim: + """Call the artifact owner's exact startup namespace fence.""" + try: + from app.modules.artifacts.service import ( + validate_artifact_storage_namespace_at_startup, + ) + except ImportError as exc: + raise RuntimeError("artifact namespace startup validation is unavailable") from exc + + return await validate_artifact_storage_namespace_at_startup( + bootstrap, + settings, + ) + + def _validation_error_detail(error: dict[str, Any]) -> dict[str, Any]: """Return one validation error without raw request input.""" safe_error: dict[str, Any] = {} diff --git a/backend/app/modules/artifacts/models.py b/backend/app/modules/artifacts/models.py index 5c15362e7..2f03ca045 100644 --- a/backend/app/modules/artifacts/models.py +++ b/backend/app/modules/artifacts/models.py @@ -95,8 +95,8 @@ class ArtifactUploadItem(Base): __table_args__ = ( UniqueConstraint("session_id", "idempotency_key", name="uq_artifact_item_operation"), CheckConstraint( - "state in ('reserved', 'uploading', 'provider_committed', " - "'replay_required', 'ready', 'failed', 'cancelled')", + "state in ('reserved', 'uploading', 'replay_required', " + "'stored_pending_verification', 'ready', 'failed', 'cancelled')", name="state", ), CheckConstraint( @@ -110,9 +110,11 @@ class ArtifactUploadItem(Base): name="expected_sha256_shape", ), CheckConstraint( - "(state = 'ready') = " - "(content_id is not null and provider_operation_reference is not null)", - name="ready_result_required", + "((state in ('stored_pending_verification', 'ready')) and " + "content_id is not null and provider_object_ref is not null) or " + "((state not in ('stored_pending_verification', 'ready')) and " + "content_id is null and provider_object_ref is null)", + name="stored_result_required", ), CheckConstraint( "state != 'failed' or error_code is not null", @@ -134,7 +136,7 @@ class ArtifactUploadItem(Base): request_digest: Mapped[str] = mapped_column(String(71), nullable=False) state: Mapped[str] = mapped_column(String(30), nullable=False, default="reserved", index=True) cas_version: Mapped[int] = mapped_column(Integer, nullable=False, default=0) - provider_operation_reference: Mapped[str | None] = mapped_column(String(200)) + provider_object_ref: Mapped[str | None] = mapped_column(String(1024)) content_id: Mapped[str | None] = mapped_column( ForeignKey("artifact_contents.id", ondelete="RESTRICT"), index=True ) @@ -200,37 +202,67 @@ class ArtifactBinding(Base): created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) +class ArtifactStorageNamespace(Base): + """Immutable singleton fencing one deployment to one storage namespace.""" + + __tablename__ = "artifact_storage_namespaces" + __table_args__ = ( + CheckConstraint("id = 'primary'", name="singleton_id"), + CheckConstraint(SHA256_CHECK.format(column="namespace_fingerprint"), name="fingerprint_shape"), + UniqueConstraint( + "namespace_fingerprint", + name="uq_artifact_storage_namespace_fingerprint", + ), + ) + + id: Mapped[str] = mapped_column(String(20), primary_key=True) + backend: Mapped[str] = mapped_column(String(50), nullable=False) + adapter: Mapped[str] = mapped_column(String(50), nullable=False) + provider_profile: Mapped[str] = mapped_column(String(100), nullable=False) + namespace_descriptor: Mapped[dict[str, object]] = mapped_column(JSON, nullable=False) + namespace_fingerprint: Mapped[str] = mapped_column(String(71), nullable=False) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) + + class ArtifactReplica(Base): """Provider observation record for one immutable content object.""" __tablename__ = "artifact_replicas" __table_args__ = ( - UniqueConstraint("adapter", "provider_artifact_id", name="uq_artifact_replica_provider"), - CheckConstraint( - "verification_state in ('pending', 'verified', 'failed')", name="verification_state" + UniqueConstraint( + "storage_namespace_id", + "provider_object_ref", + name="uq_artifact_replica_provider_object", ), CheckConstraint( - "retention_state in ('unretained', 'retained', 'released')", name="retention_state" + "verification_state in ('pending', 'verified', 'missing', 'integrity_mismatch')", + name="verification_state", ), CheckConstraint( - "availability_state in ('available', 'unavailable', 'missing')", + "availability_state in ('unknown', 'available', 'unavailable')", name="availability_state", ), CheckConstraint( - "integrity_state in ('unknown', 'valid', 'quarantined')", + "integrity_state in ('unknown', 'valid', 'invalid')", name="integrity_state", ), + CheckConstraint(SHA256_CHECK.format(column="namespace_fingerprint"), name="fingerprint_shape"), ) id: Mapped[str] = mapped_column(String(36), primary_key=True) content_id: Mapped[str] = mapped_column( ForeignKey("artifact_contents.id", ondelete="RESTRICT"), nullable=False, index=True ) + storage_namespace_id: Mapped[str] = mapped_column( + ForeignKey("artifact_storage_namespaces.id", ondelete="RESTRICT"), + nullable=False, + index=True, + ) + namespace_fingerprint: Mapped[str] = mapped_column(String(71), nullable=False) adapter: Mapped[str] = mapped_column(String(50), nullable=False) - provider_artifact_id: Mapped[str] = mapped_column(String(200), nullable=False) - provider_manifest_id: Mapped[str | None] = mapped_column(String(200)) + provider_profile: Mapped[str] = mapped_column(String(100), nullable=False) + provider_object_ref: Mapped[str] = mapped_column(String(1024), nullable=False) verification_state: Mapped[str] = mapped_column(String(30), nullable=False) - retention_state: Mapped[str] = mapped_column(String(30), nullable=False) availability_state: Mapped[str] = mapped_column(String(30), nullable=False) integrity_state: Mapped[str] = mapped_column(String(30), nullable=False) last_reconciled_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) @@ -241,70 +273,36 @@ class ArtifactReplica(Base): class ArtifactOperationReceipt(Base): - """Append-only Workstream copy of authoritative provider evidence.""" + """Append-only Workstream evidence for one immutable put acknowledgement.""" __tablename__ = "artifact_operation_receipts" __table_args__ = ( - UniqueConstraint( - "adapter", "service_principal", "operation", "idempotency_key", - name="uq_artifact_receipt_operation", - ), + UniqueConstraint("upload_item_id", name="uq_artifact_receipt_upload_item"), CheckConstraint(SHA256_CHECK.format(column="request_digest"), name="request_digest_shape"), - CheckConstraint(SHA256_CHECK.format(column="response_digest"), name="response_digest_shape"), - CheckConstraint("operation in ('store', 'verify', 'retain', 'release')", name="operation"), - CheckConstraint( - "outcome in ('stored', 'verified', 'retained', 'released')", name="outcome" - ), - CheckConstraint( - "(operation = 'store' and outcome = 'stored') or " - "(operation = 'verify' and outcome = 'verified') or " - "(operation = 'retain' and outcome = 'retained') or " - "(operation = 'release' and outcome = 'released')", - name="operation_outcome", - ), + CheckConstraint("operation = 'put'", name="operation"), + CheckConstraint("outcome = 'stored_pending_verification'", name="outcome"), CheckConstraint("attempt_number > 0", name="attempt_positive"), - CheckConstraint( - "operation != 'retain' or " - "(retention_reference is not null and retention_class is not null " - "and retention_owner is not null)", - name="retain_fields", - ), - CheckConstraint( - "operation != 'release' or " - "(retention_reference is not null and retention_class is not null " - "and retention_owner is not null)", - name="release_reference", - ), - CheckConstraint( - "operation in ('retain', 'release') or " - "(retention_reference is null and retention_class is null " - "and retention_owner is null)", - name="non_retention_fields_empty", - ), ) id: Mapped[str] = mapped_column(String(36), primary_key=True) - upload_item_id: Mapped[str | None] = mapped_column( - ForeignKey("artifact_upload_items.id", ondelete="RESTRICT"), index=True + upload_item_id: Mapped[str] = mapped_column( + ForeignKey("artifact_upload_items.id", ondelete="RESTRICT"), + nullable=False, + index=True, ) - replica_id: Mapped[str | None] = mapped_column( - ForeignKey("artifact_replicas.id", ondelete="RESTRICT"), index=True + replica_id: Mapped[str] = mapped_column( + ForeignKey("artifact_replicas.id", ondelete="RESTRICT"), + nullable=False, + index=True, ) - adapter: Mapped[str] = mapped_column(String(50), nullable=False) - service_principal: Mapped[str] = mapped_column(String(200), nullable=False) operation: Mapped[str] = mapped_column(String(30), nullable=False) idempotency_key: Mapped[str] = mapped_column(String(200), nullable=False) request_digest: Mapped[str] = mapped_column(String(71), nullable=False) - response_digest: Mapped[str] = mapped_column(String(71), nullable=False) - provider_receipt_id: Mapped[str] = mapped_column(String(200), nullable=False) - provider_operation_reference: Mapped[str] = mapped_column(String(200), nullable=False) + provider_object_ref: Mapped[str] = mapped_column(String(1024), nullable=False) + replayed: Mapped[bool] = mapped_column(nullable=False) outcome: Mapped[str] = mapped_column(String(30), nullable=False) attempt_number: Mapped[int] = mapped_column(Integer, nullable=False, default=1) correlation_id: Mapped[str] = mapped_column(String(100), nullable=False) - retention_reference: Mapped[str | None] = mapped_column(String(200)) - retention_class: Mapped[str | None] = mapped_column(String(100)) - retention_owner: Mapped[str | None] = mapped_column(String(200)) - provider_recorded_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False) details: Mapped[list[dict[str, str]]] = mapped_column(JSON, nullable=False, default=list) created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) diff --git a/backend/app/modules/artifacts/repository.py b/backend/app/modules/artifacts/repository.py index 698819198..a27103365 100644 --- a/backend/app/modules/artifacts/repository.py +++ b/backend/app/modules/artifacts/repository.py @@ -10,6 +10,7 @@ ArtifactContent, ArtifactOperationReceipt, ArtifactReplica, + ArtifactStorageNamespace, ArtifactUploadItem, ArtifactUploadSession, ) @@ -61,58 +62,68 @@ async def get_or_create_content(self, content: ArtifactContent) -> ArtifactConte ) return result.scalar_one() - async def add_replica(self, replica: ArtifactReplica) -> ArtifactReplica: - """Persist a provider replica observation.""" - self._session.add(replica) - await self._session.flush() - return replica + async def get_or_create_replica(self, replica: ArtifactReplica) -> ArtifactReplica: + """Atomically return one replica for a namespace and provider object.""" + await self._session.execute( + insert(ArtifactReplica) + .values( + id=replica.id, + content_id=replica.content_id, + storage_namespace_id=replica.storage_namespace_id, + namespace_fingerprint=replica.namespace_fingerprint, + adapter=replica.adapter, + provider_profile=replica.provider_profile, + provider_object_ref=replica.provider_object_ref, + verification_state=replica.verification_state, + availability_state=replica.availability_state, + integrity_state=replica.integrity_state, + ) + .on_conflict_do_nothing(constraint="uq_artifact_replica_provider_object") + ) + result = await self._session.execute( + select(ArtifactReplica).where( + ArtifactReplica.storage_namespace_id == replica.storage_namespace_id, + ArtifactReplica.provider_object_ref == replica.provider_object_ref, + ) + ) + return result.scalar_one() async def add_receipt(self, receipt: ArtifactOperationReceipt) -> ArtifactOperationReceipt: - """Persist one append-only provider operation receipt.""" + """Persist one append-only Workstream put receipt.""" self._session.add(receipt) await self._session.flush() return receipt - async def get_receipt( - self, - adapter: str, - service_principal: str, - operation: str, - idempotency_key: str, + async def get_receipt_for_item( + self, upload_item_id: str ) -> ArtifactOperationReceipt | None: - """Load the authoritative Workstream receipt for one operation identity.""" + """Load the Workstream put receipt for one upload item.""" result = await self._session.execute( select(ArtifactOperationReceipt).where( - ArtifactOperationReceipt.adapter == adapter, - ArtifactOperationReceipt.service_principal == service_principal, - ArtifactOperationReceipt.operation == operation, - ArtifactOperationReceipt.idempotency_key == idempotency_key, + ArtifactOperationReceipt.upload_item_id == upload_item_id, ) ) return result.scalar_one_or_none() - async def get_replica_by_provider_id( - self, adapter: str, provider_artifact_id: str - ) -> ArtifactReplica | None: - """Load one provider replica by its opaque provider identifier.""" - result = await self._session.execute( - select(ArtifactReplica).where( - ArtifactReplica.adapter == adapter, - ArtifactReplica.provider_artifact_id == provider_artifact_id, + async def claim_storage_namespace( + self, namespace: ArtifactStorageNamespace + ) -> ArtifactStorageNamespace: + """Atomically claim or load the immutable deployment namespace.""" + await self._session.execute( + insert(ArtifactStorageNamespace) + .values( + id=namespace.id, + backend=namespace.backend, + adapter=namespace.adapter, + provider_profile=namespace.provider_profile, + namespace_descriptor=namespace.namespace_descriptor, + namespace_fingerprint=namespace.namespace_fingerprint, ) + .on_conflict_do_nothing(index_elements=[ArtifactStorageNamespace.id]) ) - return result.scalar_one_or_none() - - async def lock_replica_by_provider_id( - self, adapter: str, provider_artifact_id: str - ) -> ArtifactReplica | None: - """Load one provider replica under a reconciliation row lock.""" result = await self._session.execute( - select(ArtifactReplica) - .where( - ArtifactReplica.adapter == adapter, - ArtifactReplica.provider_artifact_id == provider_artifact_id, + select(ArtifactStorageNamespace).where( + ArtifactStorageNamespace.id == namespace.id ) - .with_for_update() ) - return result.scalar_one_or_none() + return result.scalar_one() diff --git a/backend/app/modules/artifacts/service.py b/backend/app/modules/artifacts/service.py index 75e737d3c..72e1903da 100644 --- a/backend/app/modules/artifacts/service.py +++ b/backend/app/modules/artifacts/service.py @@ -1,206 +1,175 @@ -"""Transaction-safe artifact ingest and reconciliation orchestration.""" +"""Internal orchestration for namespace-fenced immutable artifact storage.""" from __future__ import annotations import asyncio -from collections.abc import AsyncIterable from dataclasses import dataclass -from datetime import UTC, datetime from uuid import uuid4 from sqlalchemy.ext.asyncio import AsyncSession +from app.core.config import Settings +from app.core.cancellation import await_cancellation_resistant +from app.db.session import get_session_factory from app.interfaces.artifacts import ( - ArtifactOperationResult, - ArtifactStatus, ArtifactInputMismatchError, ArtifactIntegrityError, - ArtifactNotFoundError, + ArtifactPutResult, ArtifactStore, + ArtifactStoreBootstrap, ArtifactStoreError, - IdempotencyIdentity, - OperationReceipt, - ReceiptOutcome, - StoreArtifactRequest, - StoredArtifact, + ArtifactStoreNamespaceClaim, + artifact_store_namespace_material, ) -from app.core.cancellation import await_cancellation_resistant +from app.interfaces.external_services import ExternalServiceAdapterIdentity from app.modules.artifacts.models import ( ArtifactContent, ArtifactOperationReceipt, ArtifactReplica, + ArtifactStorageNamespace, ArtifactUploadItem, ArtifactUploadSession, ) from app.modules.artifacts.repository import ArtifactRepository -from app.modules.audit.repository import AuditRepository -from app.modules.tasks.models import AuditEvent +from app.modules.artifacts.sources import ArtifactCommitment, CommittedArtifactSource -RECONCILER_ACTOR_ID = "workstream.artifact.reconciler" -RECONCILER_ISSUER = "https://workstream.internal" +ARTIFACT_STORAGE_NAMESPACE_ID = "primary" class ArtifactIngestStateError(Exception): - """Raised when an artifact item cannot perform the requested transition.""" + """Raised when persisted artifact state cannot perform an internal transition.""" + + +class ArtifactStorageNamespaceError(ArtifactIngestStateError): + """Raised before provider I/O when deployment storage identity has drifted.""" @dataclass(frozen=True, slots=True) -class ProviderAttemptFence: - """Committed item/session CAS values fencing one provider attempt.""" +class ArtifactStorageNamespaceSpec: + """Canonical non-secret identity for one configured artifact namespace.""" - item_cas: int - session_cas: int + backend: str + adapter: str + provider_profile: str + namespace_descriptor: dict[str, object] + namespace_fingerprint: str -class ArtifactIngestService: - """Coordinate Workstream transactions around provider artifact I/O.""" +@dataclass(frozen=True, slots=True) +class ProviderAttemptFence: + """Committed item CAS value fencing one provider call.""" + + item_cas: int - def __init__(self, session: AsyncSession, store: ArtifactStore, *, adapter_name: str) -> None: - """Create an ingest coordinator. - Args: - session: Database session owned by this ingest operation. - store: Provider-neutral artifact store. - adapter_name: Stable configured adapter name. - """ +def artifact_storage_namespace_spec( + settings: Settings, + store: ArtifactStoreBootstrap, +) -> ArtifactStorageNamespaceSpec: + """Build the canonical descriptor from one already-pinned provider root.""" + identity = store.identity + if type(identity) is not ExternalServiceAdapterIdentity: + raise ArtifactStorageNamespaceError("artifact adapter identity is invalid") + if settings.artifact_store_backend != identity.provider_key: + raise ArtifactStorageNamespaceError("artifact adapter identity does not match configuration") + namespace_identity = store.namespace_identity + descriptor, fingerprint = artifact_store_namespace_material( + backend=settings.artifact_store_backend, + adapter_identity=identity, + namespace_identity=namespace_identity, + ) + return ArtifactStorageNamespaceSpec( + backend=settings.artifact_store_backend, + adapter=identity.provider_key, + provider_profile=namespace_identity.provider_profile, + namespace_descriptor=descriptor, + namespace_fingerprint=fingerprint, + ) + + +class ArtifactStorageOrchestrator: + """Sole internal owner of the writable ArtifactStore capability.""" + + def __init__( + self, + session: AsyncSession, + store: ArtifactStore, + namespace: ArtifactStorageNamespaceSpec, + ) -> None: + """Bind one database session, byte store, and deployment namespace.""" self._session = session self._store = store - self._adapter_name = adapter_name + self._namespace = namespace self._repo = ArtifactRepository(session) - self._audit_repo = AuditRepository(session) - async def ingest_reserved_item( + async def ensure_storage_namespace(self) -> ArtifactStorageNamespace: + """Claim or validate the immutable singleton before provider access.""" + async with self._session.begin(): + return await self._claim_and_validate_namespace() + + async def put_reserved_item( self, item_id: str, - stream: AsyncIterable[bytes], - request: StoreArtifactRequest, - ) -> StoredArtifact: - """Commit intent, call provider outside a transaction, then persist facts. - - Args: - item_id: Reserved upload-item identifier. - stream: Exact bytes to store or replay. - request: Provider request matching the persisted item commitment. - - Returns: - Provider-neutral stored artifact result. - - Raises: - ArtifactIngestStateError: If the persisted intent is incompatible. - ArtifactStoreError: If provider ingest or verification fails. - """ - attempt_cas = await self._start_provider_operation(item_id, request) + source: CommittedArtifactSource, + *, + correlation_id: str | None = None, + ) -> ArtifactPutResult: + """Publish one already-reserved internal item without activating product ingest.""" + commitment = source.commitment + fence = await self._start_put(item_id, commitment) try: - stored = await self._store.store(stream, request) - except asyncio.CancelledError: - await await_cancellation_resistant( - self._mark_provider_recovery_required( - item_id, request, attempt_cas, provider_commit_confirmed=False + result = await self._store.put(source) + except asyncio.CancelledError as cancellation: + try: + await await_cancellation_resistant( + self._mark_replay_required(item_id, fence) ) - ) + except BaseException: + raise cancellation from None raise except ArtifactStoreError as exc: if exc.retryable: - await self._mark_provider_recovery_required( - item_id, request, attempt_cas, provider_commit_confirmed=False - ) + await self._mark_replay_required(item_id, fence) else: - await self._fail_pre_finalization(item_id, exc.code, attempt_cas) - raise - try: - await self._finalize_provider_operation(item_id, request, stored, attempt_cas) - except asyncio.CancelledError: - await await_cancellation_resistant( - self._mark_provider_recovery_required( - item_id, request, attempt_cas, provider_commit_confirmed=True - ) - ) - raise - except (ArtifactIntegrityError, ArtifactInputMismatchError) as exc: - await self._fail_pre_finalization(item_id, exc.code, attempt_cas) + await self._fail_put(item_id, exc.code, fence) raise - except Exception: - await self._mark_provider_recovery_required( - item_id, request, attempt_cas, provider_commit_confirmed=True - ) + except BaseException: + await self._mark_replay_required(item_id, fence) raise - return stored - async def reconcile_committed_item( - self, item_id: str, request: StoreArtifactRequest - ) -> StoredArtifact: - """Finalize a committed provider effect without replaying client bytes.""" - attempt_cas = await self._resume_provider_committed(item_id, request) try: - stored = await self._store.recover_committed_store(request) - observed_sha256, observed_size = await self._hash_provider_object( - stored.provider_artifact_id, request.maximum_bytes + await self._finalize_put( + item_id, + commitment, + result, + fence, + correlation_id=correlation_id or str(uuid4()), ) - if observed_sha256 != stored.sha256 or observed_size != stored.byte_count: - raise ArtifactIntegrityError("recovered provider bytes changed") - await self._finalize_provider_operation(item_id, request, stored, attempt_cas) - return stored except (ArtifactIntegrityError, ArtifactInputMismatchError) as exc: - await self._fail_pre_finalization(item_id, exc.code, attempt_cas) + await self._fail_put(item_id, exc.code, fence) raise - except ArtifactNotFoundError: - await self._mark_provider_recovery_required( - item_id, request, attempt_cas, provider_commit_confirmed=False - ) + except asyncio.CancelledError as cancellation: + try: + await await_cancellation_resistant( + self._mark_replay_required(item_id, fence) + ) + except BaseException: + raise cancellation from None raise except BaseException: - await await_cancellation_resistant( - self._mark_provider_recovery_required( - item_id, request, attempt_cas, provider_commit_confirmed=True - ) - ) + await self._mark_replay_required(item_id, fence) raise + return result - async def _hash_provider_object( - self, provider_artifact_id: str, maximum_bytes: int - ) -> tuple[str, int]: - """Independently hash and count complete provider bytes.""" - import hashlib - - digest = hashlib.sha256() - byte_count = 0 - async for chunk in self._store.open(provider_artifact_id): - digest.update(chunk) - byte_count += len(chunk) - if byte_count > maximum_bytes: - raise ArtifactIntegrityError("recovered provider bytes exceed the request limit") - return f"sha256:{digest.hexdigest()}", byte_count - - async def _resume_provider_committed( - self, item_id: str, request: StoreArtifactRequest - ) -> ProviderAttemptFence: - """Fence one receipt-based recovery attempt for a committed provider effect.""" - async with self._session.begin(): - item = await self._repo.lock_upload_item(item_id) - if item is None or item.state != "provider_committed": - raise ArtifactIngestStateError("artifact item is not provider committed") - if ( - item.request_digest != request.idempotency.request_digest - or item.idempotency_key != request.idempotency.key - or item.expected_sha256 != request.expected_sha256 - or item.expected_size != request.expected_size - ): - raise ArtifactIngestStateError("artifact recovery commitment changed") - item.state = "uploading" - item.error_code = None - item.cas_version += 1 - upload_session = await self._repo.lock_upload_session(item.session_id) - if upload_session is None or upload_session.state != "open": - raise ArtifactIngestStateError("artifact upload session is not open") - upload_session.cas_version += 1 - return ProviderAttemptFence(item.cas_version, upload_session.cas_version) - - async def _start_provider_operation( - self, item_id: str, request: StoreArtifactRequest + async def _start_put( + self, + item_id: str, + commitment: ArtifactCommitment, ) -> ProviderAttemptFence: - """Transaction A: lock and commit the exact provider intent.""" + """Transaction A validates namespace and exact reserved commitment.""" async with self._session.begin(): + await self._claim_and_validate_namespace() item = await self._repo.lock_upload_item(item_id) if item is None: raise ArtifactIngestStateError("artifact upload item does not exist") @@ -208,470 +177,222 @@ async def _start_provider_operation( if upload_session is None or upload_session.state != "open": raise ArtifactIngestStateError("artifact upload session is not open") if item.state not in {"reserved", "replay_required"}: - raise ArtifactIngestStateError("artifact upload item cannot be ingested") - if item.request_digest != request.idempotency.request_digest: - raise ArtifactIngestStateError("artifact upload request digest changed") - if item.idempotency_key != request.idempotency.key: - raise ArtifactIngestStateError("artifact upload idempotency key changed") - if item.expected_sha256 != request.expected_sha256 or item.expected_size != request.expected_size: - raise ArtifactIngestStateError("artifact upload commitment changed") - if request.maximum_bytes > item.reserved_bytes: - raise ArtifactIngestStateError("artifact upload maximum exceeds its reservation") - if request.expected_size is not None and request.expected_size > item.reserved_bytes: - raise ArtifactIngestStateError("artifact expected size exceeds its reservation") + raise ArtifactIngestStateError("artifact upload item cannot be stored") if ( - upload_session.current_bytes + upload_session.reserved_bytes - > upload_session.maximum_bytes - or upload_session.current_items + upload_session.reserved_items - > upload_session.maximum_items - or upload_session.reserved_bytes < item.reserved_bytes - or upload_session.reserved_items < 1 + item.expected_sha256 != commitment.sha256 + or item.expected_size != commitment.byte_count + or item.media_type != commitment.media_type + or commitment.byte_count > item.reserved_bytes ): - raise ArtifactIngestStateError("artifact upload reservation is inconsistent") + raise ArtifactIngestStateError("artifact upload commitment changed") item.state = "uploading" + item.error_code = None item.cas_version += 1 upload_session.cas_version += 1 - return ProviderAttemptFence(item.cas_version, upload_session.cas_version) + return ProviderAttemptFence(item.cas_version) - async def _finalize_provider_operation( + async def _finalize_put( self, item_id: str, - request: StoreArtifactRequest, - stored: StoredArtifact, - attempt_cas: ProviderAttemptFence, + commitment: ArtifactCommitment, + result: ArtifactPutResult, + fence: ProviderAttemptFence, + *, + correlation_id: str, ) -> None: - """Transaction B: verify provider facts and persist immutable records.""" - if stored.sha256 != stored.receipt.sha256 or stored.byte_count != stored.receipt.byte_count: - raise ArtifactIntegrityError("provider receipt disagrees with stored artifact") - if stored.receipt.identity != request.idempotency: - raise ArtifactIntegrityError("provider receipt identity disagrees with request") - if ( - stored.receipt.outcome != ReceiptOutcome.STORED - or stored.receipt.provider_artifact_id != stored.provider_artifact_id - or stored.byte_count > request.maximum_bytes - ): - raise ArtifactIntegrityError("provider result disagrees with the store operation") - if ( - request.expected_sha256 is not None - and stored.sha256 != request.expected_sha256 - ) or ( - request.expected_size is not None - and stored.byte_count != request.expected_size - ): - raise ArtifactIntegrityError("provider result violates the persisted commitment") + """Transaction B records acknowledgement without making bytes bindable.""" async with self._session.begin(): - item = await self._repo.lock_upload_item(item_id) - if ( - item is None - or item.state != "uploading" - or item.cas_version != attempt_cas.item_cas - ): - raise ArtifactIngestStateError("artifact upload item is not awaiting finalization") - upload_session = await self._repo.lock_upload_session(item.session_id) - if upload_session is None or upload_session.state != "open": - raise ArtifactIngestStateError("artifact upload session changed during ingest") - if upload_session.cas_version != attempt_cas.session_cas: - raise ArtifactIngestStateError("artifact upload session CAS changed during ingest") - if ( - item.request_digest != request.idempotency.request_digest - or item.idempotency_key != request.idempotency.key - or item.expected_sha256 != request.expected_sha256 - or item.expected_size != request.expected_size - or stored.byte_count > item.reserved_bytes - or upload_session.reserved_bytes < item.reserved_bytes - or upload_session.reserved_items < 1 - ): - raise ArtifactIngestStateError("artifact upload intent changed during ingest") - existing_receipt = await self._repo.get_receipt( - self._adapter_name, - request.idempotency.service_principal, - request.idempotency.operation.value, - request.idempotency.key, - ) + namespace = await self._claim_and_validate_namespace() + item, upload_session = await self._locked_attempt(item_id, fence) + existing_receipt = await self._repo.get_receipt_for_item(item.id) if existing_receipt is not None: - if existing_receipt.request_digest != request.idempotency.request_digest: - raise ArtifactIntegrityError("stored operation receipt request digest changed") - if ( - existing_receipt.response_digest != stored.receipt.response_digest - or existing_receipt.provider_receipt_id != stored.receipt.receipt_id - or existing_receipt.outcome != ReceiptOutcome.STORED.value - or existing_receipt.provider_operation_reference - != stored.receipt.provider_operation_reference - ): - raise ArtifactIntegrityError("stored operation receipt facts changed") - existing_replica = await self._repo.get_replica_by_provider_id( - self._adapter_name, stored.provider_artifact_id - ) - if existing_replica is None: - raise ArtifactIntegrityError("stored operation receipt has no replica") - item.state = "ready" - item.provider_operation_reference = stored.receipt.provider_operation_reference - item.content_id = existing_replica.content_id - item.cas_version += 1 - self._apply_ready_accounting(upload_session, item, stored.byte_count) - return + raise ArtifactIntegrityError("artifact upload item already has put evidence") + content = await self._repo.get_or_create_content( ArtifactContent( id=str(uuid4()), - sha256=stored.sha256, - byte_count=stored.byte_count, - media_type=stored.media_type, + sha256=commitment.sha256, + byte_count=commitment.byte_count, + media_type=commitment.media_type, normalized_display_name=item.display_name, ) ) - replica = await self._repo.add_replica( + replica = await self._repo.get_or_create_replica( ArtifactReplica( id=str(uuid4()), content_id=content.id, - adapter=self._adapter_name, - provider_artifact_id=stored.provider_artifact_id, + storage_namespace_id=namespace.id, + namespace_fingerprint=namespace.namespace_fingerprint, + adapter=namespace.adapter, + provider_profile=namespace.provider_profile, + provider_object_ref=result.provider_object_ref, verification_state="pending", - retention_state="unretained", - availability_state="available", - integrity_state="valid", + availability_state="unknown", + integrity_state="unknown", ) ) + if ( + replica.content_id != content.id + or replica.namespace_fingerprint != namespace.namespace_fingerprint + or replica.adapter != namespace.adapter + or replica.provider_profile != namespace.provider_profile + ): + raise ArtifactIntegrityError("provider object reference changed content identity") + await self._repo.add_receipt( - self._operation_receipt_model( - stored.receipt, - adapter=self._adapter_name, + ArtifactOperationReceipt( + id=str(uuid4()), upload_item_id=item.id, replica_id=replica.id, - retention_owner=( - request.idempotency.service_principal - if stored.receipt.retention_reference is not None - else None - ), + operation="put", + idempotency_key=item.idempotency_key, + request_digest=item.request_digest, + provider_object_ref=result.provider_object_ref, + replayed=result.replayed, + outcome="stored_pending_verification", + attempt_number=1, + correlation_id=correlation_id, + details=[ + { + "name": "namespace_fingerprint", + "value": namespace.namespace_fingerprint, + } + ], ) ) - item.state = "ready" + item.state = "stored_pending_verification" item.content_id = content.id - item.provider_operation_reference = stored.receipt.provider_operation_reference + item.provider_object_ref = result.provider_object_ref item.cas_version += 1 - self._apply_ready_accounting(upload_session, item, stored.byte_count) + self._apply_committed_accounting(upload_session, item, commitment.byte_count) + + async def _claim_and_validate_namespace(self) -> ArtifactStorageNamespace: + """Atomically claim the singleton or reject deployment identity drift.""" + return await _claim_and_validate_storage_namespace(self._repo, self._namespace) + + async def _locked_attempt( + self, + item_id: str, + fence: ProviderAttemptFence, + ) -> tuple[ArtifactUploadItem, ArtifactUploadSession]: + """Reload and fence the exact item/session pair after provider I/O.""" + item = await self._repo.lock_upload_item(item_id) + if item is None or item.state != "uploading" or item.cas_version != fence.item_cas: + raise ArtifactIngestStateError("artifact upload item is not awaiting acknowledgement") + upload_session = await self._repo.lock_upload_session(item.session_id) + if ( + upload_session is None + or upload_session.state != "open" + ): + raise ArtifactIngestStateError("artifact upload session is not open") + return item, upload_session @staticmethod - def _apply_ready_accounting( + def _apply_committed_accounting( upload_session: ArtifactUploadSession, item: ArtifactUploadItem, byte_count: int, ) -> None: - """Move one committed item reservation into exact session totals.""" + """Move one item from reserved capacity to acknowledged byte usage.""" + ArtifactStorageOrchestrator._validate_reserved_accounting(upload_session, item) upload_session.reserved_bytes -= item.reserved_bytes upload_session.reserved_items -= 1 upload_session.current_bytes += byte_count upload_session.current_items += 1 upload_session.cas_version += 1 - async def _mark_provider_recovery_required( + async def _mark_replay_required( self, item_id: str, - request: StoreArtifactRequest, - expected_fence: ProviderAttemptFence, - *, - provider_commit_confirmed: bool, + fence: ProviderAttemptFence, ) -> None: - """Record a provider effect that Workstream could not yet finalize.""" + """Record an ambiguous acknowledgement only when the attempt fence matches.""" await self._session.rollback() async with self._session.begin(): item = await self._repo.lock_upload_item(item_id) - if ( - item is None - or item.state != "uploading" - or item.cas_version != expected_fence.item_cas - ): + if item is None or item.state != "uploading" or item.cas_version != fence.item_cas: return - item.state = ( - "provider_committed" - if provider_commit_confirmed - and request.expected_sha256 is not None - and request.expected_size is not None - else "replay_required" - ) - item.error_code = "artifact_finalization_pending" + item.state = "replay_required" + item.error_code = "artifact_put_acknowledgement_unknown" item.cas_version += 1 - async def _fail_pre_finalization( + async def _fail_put( self, item_id: str, error_code: str, - expected_fence: ProviderAttemptFence, + fence: ProviderAttemptFence, ) -> None: - """Fail an unpromoted item without creating content, replica, or binding.""" + """Fail one terminal provider attempt and release its reservation once.""" await self._session.rollback() async with self._session.begin(): item = await self._repo.lock_upload_item(item_id) - if ( - item is None - or item.state != "uploading" - or item.cas_version != expected_fence.item_cas - ): + if item is None or item.state != "uploading" or item.cas_version != fence.item_cas: return upload_session = await self._repo.lock_upload_session(item.session_id) + if upload_session is None: + raise ArtifactIntegrityError("artifact upload session is missing") + self._validate_reserved_accounting(upload_session, item) item.state = "failed" item.error_code = error_code item.cas_version += 1 - if upload_session is not None: - upload_session.reserved_bytes = max( - 0, upload_session.reserved_bytes - item.reserved_bytes - ) - upload_session.reserved_items = max(0, upload_session.reserved_items - 1) - upload_session.cas_version += 1 - - async def quarantine_existing_replica( - self, - adapter: str, - provider_artifact_id: str, - *, - reason: str, - ) -> None: - """Record provider quarantine and audit it without mutating content facts.""" - async with self._session.begin(): - current = await self._repo.lock_replica_by_provider_id( - adapter, provider_artifact_id - ) - if current is None or current.integrity_state == "quarantined": - return - previous_integrity_state = current.integrity_state - current.integrity_state = "quarantined" - current.availability_state = "unavailable" - current.last_reconciled_at = datetime.now(UTC) - await self._audit_repo.add_audit_event( - self._service_audit_event( - current.id, - "artifact_replica_quarantined", - previous_integrity_state, - "quarantined", - RECONCILER_ACTOR_ID, - {"adapter": current.adapter}, - reason=reason, - ) - ) - - async def retain_replica( - self, - adapter: str, - provider_artifact_id: str, - retention_reference: str, - retention_class: str, - identity: IdempotencyIdentity, - ) -> ArtifactOperationResult: - """Retain one replica and copy provider evidence into Workstream.""" - result = await self._store.retain( - provider_artifact_id, - retention_reference, - retention_class, - identity, - ) - await self._record_retention_result(adapter, result, identity, retained=True) - return result - - async def release_replica( - self, - adapter: str, - provider_artifact_id: str, - retention_reference: str, - identity: IdempotencyIdentity, - ) -> ArtifactOperationResult: - """Release one owned reference and audit the Workstream transition.""" - result = await self._store.release( - provider_artifact_id, - retention_reference, - identity, - ) - status = await self._store.stat(provider_artifact_id) - await self._record_retention_result( - adapter, - result, - identity, - retained=bool(status.active_retentions), - audit_release=True, - ) - return result - - async def reconcile_replica( - self, adapter: str, provider_artifact_id: str - ) -> ArtifactStatus: - """Copy current provider observations and audit any state correction.""" - status = await self._store.stat(provider_artifact_id) - async with self._session.begin(): - replica = await self._repo.lock_replica_by_provider_id( - adapter, provider_artifact_id - ) - if replica is None: - raise ArtifactIngestStateError("artifact replica does not exist") - before = { - "verification_state": replica.verification_state, - "retention_state": replica.retention_state, - "availability_state": replica.availability_state, - "integrity_state": replica.integrity_state, - } - after = { - "verification_state": status.verification_state.value, - "retention_state": status.retention_state.value, - "availability_state": status.availability_state.value, - "integrity_state": status.integrity_state.value, - } - provider_observation = dict(after) - if before["integrity_state"] == "quarantined": - after["integrity_state"] = "quarantined" - after["availability_state"] = "unavailable" - for field_name, value in after.items(): - setattr(replica, field_name, value) - replica.last_reconciled_at = datetime.now(UTC) - if before != after: - await self._audit_repo.add_audit_event( - self._service_audit_event( - replica.id, - "artifact_replica_reconciled", - before["integrity_state"], - after["integrity_state"], - RECONCILER_ACTOR_ID, - { - "adapter": adapter, - "before": before, - "after": after, - "provider_observation": provider_observation, - }, - ) - ) - return status + upload_session.reserved_bytes -= item.reserved_bytes + upload_session.reserved_items -= 1 + upload_session.cas_version += 1 - async def _record_retention_result( - self, - adapter: str, - result: ArtifactOperationResult, - identity: IdempotencyIdentity, - *, - retained: bool, - audit_release: bool = False, + @staticmethod + def _validate_reserved_accounting( + upload_session: ArtifactUploadSession, + item: ArtifactUploadItem, ) -> None: - """Persist one provider retention receipt and matching replica state.""" - receipt = result.receipt + """Reject aggregate drift before consuming one item's reservation.""" if ( - receipt.identity != identity - or receipt.provider_artifact_id is None - or receipt.retention_reference is None - or ( - identity.operation.value == "retain" - and receipt.outcome != ReceiptOutcome.RETAINED - ) - or ( - identity.operation.value == "release" - and receipt.outcome != ReceiptOutcome.RELEASED - ) + upload_session.reserved_bytes < item.reserved_bytes + or upload_session.reserved_items < 1 ): - raise ArtifactIntegrityError("provider retention receipt is inconsistent") - async with self._session.begin(): - replica = await self._repo.lock_replica_by_provider_id( - adapter, receipt.provider_artifact_id + raise ArtifactIntegrityError( + "artifact upload session reservation accounting is invalid" ) - if replica is None: - raise ArtifactIngestStateError("artifact replica does not exist") - existing = await self._repo.get_receipt( - adapter, - identity.service_principal, - identity.operation.value, - identity.key, - ) - receipt_created = existing is None - if existing is None: - await self._repo.add_receipt( - self._operation_receipt_model( - receipt, - adapter=adapter, - replica_id=replica.id, - retention_owner=identity.service_principal, - ) - ) - elif ( - existing.request_digest != identity.request_digest - or existing.response_digest != receipt.response_digest - or existing.provider_receipt_id != receipt.receipt_id - or existing.provider_operation_reference - != receipt.provider_operation_reference - or existing.outcome != receipt.outcome.value - or existing.retention_reference != receipt.retention_reference - or existing.retention_class != receipt.retention_class - or existing.retention_owner != identity.service_principal - ): - raise ArtifactIntegrityError("Workstream retention receipt changed") - previous = replica.retention_state - replica.retention_state = "retained" if retained else "released" - replica.last_reconciled_at = datetime.now(UTC) - if audit_release and receipt_created: - await self._audit_repo.add_audit_event( - self._service_audit_event( - replica.id, - "artifact_retention_released", - previous, - replica.retention_state, - identity.service_principal, - {"retention_reference": receipt.retention_reference}, - ) - ) - @staticmethod - def _operation_receipt_model( - receipt: OperationReceipt, - *, - adapter: str, - replica_id: str, - upload_item_id: str | None = None, - retention_owner: str | None = None, - ) -> ArtifactOperationReceipt: - """Project one provider receipt into the append-only Workstream record.""" - return ArtifactOperationReceipt( - id=str(uuid4()), - upload_item_id=upload_item_id, - replica_id=replica_id, - adapter=adapter, - service_principal=receipt.identity.service_principal, - operation=receipt.identity.operation.value, - idempotency_key=receipt.identity.key, - request_digest=receipt.identity.request_digest, - response_digest=receipt.response_digest, - provider_receipt_id=receipt.receipt_id, - provider_operation_reference=receipt.provider_operation_reference, - outcome=receipt.outcome.value, - attempt_number=1, - correlation_id=str(uuid4()), - retention_reference=receipt.retention_reference, - retention_class=receipt.retention_class, - retention_owner=retention_owner, - provider_recorded_at=receipt.recorded_at, - details=[ - {"name": detail.name, "value": detail.value} - for detail in receipt.details - ], - ) - @staticmethod - def _service_audit_event( - replica_id: str, - event_type: str, - from_status: str, - to_status: str, - service_principal: str, - payload: dict[str, object], - *, - reason: str | None = None, - ) -> AuditEvent: - """Build audit evidence attributed to an internal service principal.""" - return AuditEvent( - id=str(uuid4()), - entity_type="artifact_replica", - entity_id=replica_id, - event_type=event_type, - from_status=from_status, - to_status=to_status, - actor_id=service_principal, - external_subject=service_principal, - external_issuer=RECONCILER_ISSUER, - actor_roles=["service"], - claim_snapshot={"service_principal": service_principal}, - auth_source="service", - is_dev_auth=False, - reason=reason, - event_payload=payload, - ) +async def validate_artifact_storage_namespace_at_startup( + store: ArtifactStoreBootstrap, + settings: Settings, +) -> ArtifactStoreNamespaceClaim: + """Claim one pinned namespace and return its exact initialization proof.""" + namespace = artifact_storage_namespace_spec(settings, store) + async with get_session_factory()() as session: + async with session.begin(): + await _claim_and_validate_storage_namespace( + ArtifactRepository(session), + namespace, + ) + return ArtifactStoreNamespaceClaim( + adapter_identity=store.identity, + namespace_identity=store.namespace_identity, + namespace_fingerprint=namespace.namespace_fingerprint, + ) + + +async def _claim_and_validate_storage_namespace( + repository: ArtifactRepository, + namespace: ArtifactStorageNamespaceSpec, +) -> ArtifactStorageNamespace: + """Atomically claim the singleton or reject deployment identity drift.""" + candidate = ArtifactStorageNamespace( + id=ARTIFACT_STORAGE_NAMESPACE_ID, + backend=namespace.backend, + adapter=namespace.adapter, + provider_profile=namespace.provider_profile, + namespace_descriptor=namespace.namespace_descriptor, + namespace_fingerprint=namespace.namespace_fingerprint, + ) + persisted = await repository.claim_storage_namespace(candidate) + if ( + persisted.backend != candidate.backend + or persisted.adapter != candidate.adapter + or persisted.provider_profile != candidate.provider_profile + or persisted.namespace_descriptor != candidate.namespace_descriptor + or persisted.namespace_fingerprint != candidate.namespace_fingerprint + ): + raise ArtifactStorageNamespaceError("artifact storage namespace does not match") + return persisted diff --git a/backend/app/workers/artifacts.py b/backend/app/workers/artifacts.py new file mode 100644 index 000000000..0e3764207 --- /dev/null +++ b/backend/app/workers/artifacts.py @@ -0,0 +1,17 @@ +"""Database-independent artifact scratch maintenance tasks.""" + +from __future__ import annotations + +from app.adapters.artifacts import cleanup_stale_artifact_scratch +from app.core.config import get_settings +from app.workers.async_runner import run_async_task +from app.workers.celery_app import ARTIFACT_SCRATCH_CLEANUP_TASK, celery_app + + +@celery_app.task(name=ARTIFACT_SCRATCH_CLEANUP_TASK) +def cleanup_stale_scratch() -> int: + """Remove only expired scratch reservations for enabled artifact storage.""" + settings = get_settings() + if settings.artifact_store_backend == "disabled": + return 0 + return run_async_task(lambda: cleanup_stale_artifact_scratch(settings)) diff --git a/backend/app/workers/celery_app.py b/backend/app/workers/celery_app.py index 65b2aef8c..d1f9b4317 100644 --- a/backend/app/workers/celery_app.py +++ b/backend/app/workers/celery_app.py @@ -7,6 +7,9 @@ from app.core.config import get_settings from app.workers.errors import CeleryConfigurationError +ARTIFACT_SCRATCH_CLEANUP_TASK = "workstream.artifacts.cleanup_stale_scratch" +ARTIFACT_SCRATCH_CLEANUP_SCHEDULE = "artifact-scratch-cleanup" + def create_celery_app() -> Celery: """Create the Celery application from Workstream settings. @@ -27,7 +30,11 @@ def create_celery_app() -> Celery: "workstream", broker=broker_url, backend=settings.celery_result_backend_url, - include=["app.workers.checkers", "app.workers.project_setup"], + include=[ + "app.workers.artifacts", + "app.workers.checkers", + "app.workers.project_setup", + ], ) celery_app.conf.update( accept_content=["json"], @@ -37,6 +44,12 @@ def create_celery_app() -> Celery: task_ignore_result=True, task_serializer="json", timezone="UTC", + beat_schedule={ + ARTIFACT_SCRATCH_CLEANUP_SCHEDULE: { + "task": ARTIFACT_SCRATCH_CLEANUP_TASK, + "schedule": settings.artifact_scratch_cleanup_interval_seconds, + } + }, ) return celery_app diff --git a/backend/tests/artifact_store_helpers.py b/backend/tests/artifact_store_helpers.py new file mode 100644 index 000000000..bdc149a35 --- /dev/null +++ b/backend/tests/artifact_store_helpers.py @@ -0,0 +1,101 @@ +"""Test-only activation helpers for the namespace-fenced local artifact store.""" + +from __future__ import annotations + +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager +from pathlib import Path + +from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap +from app.interfaces.artifacts import ( + ArtifactStoreNamespaceClaim, + artifact_store_namespace_material, +) +from app.modules.artifacts.preparation import ( + HARD_MAXIMUM_ARTIFACT_BYTES, + ArtifactPreparationLimits, + ArtifactPreparationService, + ArtifactScratchManager, +) +from app.modules.artifacts.sources import CommittedArtifactSource + + +async def artifact_byte_stream(*chunks: bytes) -> AsyncIterator[bytes]: + """Yield exact test bytes through the public preparation boundary.""" + for chunk in chunks: + yield chunk + + +def artifact_preparation_limits( + *, + maximum_files: int = 8, + maximum_concurrency: int = 8, +) -> ArtifactPreparationLimits: + """Return bounded test limits with the production aggregate ceiling.""" + return ArtifactPreparationLimits( + aggregate_reserved_bytes=HARD_MAXIMUM_ARTIFACT_BYTES, + maximum_files=maximum_files, + maximum_concurrency=maximum_concurrency, + minimum_free_bytes=0, + reservation_ttl_seconds=30, + total_deadline_seconds=10, + cleanup_margin_seconds=1, + stream_buffer_bytes=2, + maximum_source_bytes=1024, + ) + + +@asynccontextmanager +async def minted_source( + scratch_root: Path, + *chunks: bytes, + media_type: str = "application/octet-stream", + limits: ArtifactPreparationLimits | None = None, +) -> AsyncIterator[CommittedArtifactSource]: + """Mint and release one real sealed source without exposing scratch internals.""" + manager = ArtifactScratchManager( + root=scratch_root, + limits=limits or artifact_preparation_limits(), + ) + prepared = await ArtifactPreparationService(manager).prepare( + artifact_byte_stream(*chunks), + media_type=media_type, + ) + try: + async with prepared as source: + yield source + finally: + manager.close() + + +def initialize_local_store( + root: Path, + *, + buffer_bytes: int = 1024 * 1024, + lock_timeout_seconds: float = 1800.0, +) -> LocalStorageAdapter: + """Pre-provision, pin, claim, and initialize one test-only local store.""" + root.mkdir(parents=True, exist_ok=True, mode=0o700) + bootstrap = LocalStorageBootstrap( + LocalStorageAdapter( + root=root, + buffer_bytes=buffer_bytes, + lock_timeout_seconds=lock_timeout_seconds, + ) + ) + return bootstrap.initialize_after_namespace_claim(local_namespace_claim(bootstrap)) + + +def local_namespace_claim(bootstrap: LocalStorageBootstrap) -> ArtifactStoreNamespaceClaim: + """Mint the exact test-only namespace claim for one pinned adapter.""" + namespace_identity = bootstrap.namespace_identity + _, fingerprint = artifact_store_namespace_material( + backend="local", + adapter_identity=bootstrap.identity, + namespace_identity=namespace_identity, + ) + return ArtifactStoreNamespaceClaim( + adapter_identity=bootstrap.identity, + namespace_identity=namespace_identity, + namespace_fingerprint=fingerprint, + ) diff --git a/backend/tests/test_alembic.py b/backend/tests/test_alembic.py index 14b5a2548..e048e20f8 100644 --- a/backend/tests/test_alembic.py +++ b/backend/tests/test_alembic.py @@ -159,7 +159,9 @@ def test_current_schema_uses_project_policy_contract( "artifact_upload_items.id", "artifact_contents.sha256", "artifact_bindings.scope_version", - "artifact_replicas.provider_artifact_id", + "artifact_storage_namespaces.namespace_fingerprint", + "artifact_replicas.provider_object_ref", + "artifact_replicas.storage_namespace_id", "artifact_operation_receipts.request_digest", }.issubset(columns) discarded_columns = { @@ -184,6 +186,11 @@ def test_current_schema_uses_project_policy_contract( "workstream_tasks.locked_checker_policy_version", "submissions.locked_checker_policy_version", "checker_runs.locked_checker_policy_version", + "artifact_replicas.provider_artifact_id", + "artifact_replicas.provider_manifest_id", + "artifact_replicas.retention_state", + "artifact_operation_receipts.provider_receipt_id", + "artifact_operation_receipts.retention_reference", } assert columns.isdisjoint(discarded_columns) @@ -1153,6 +1160,169 @@ def test_artifact_foundation_enforces_immutable_facts_and_guarded_downgrade( command.downgrade(config, "base") +def test_artifact_store_v2_empty_clean_cut_and_reversible_shape( + isolated_database_env: str, + migration_lock, +) -> None: + """Migrate only empty v1 tables and restore their empty shape on downgrade.""" + config = _alembic_config() + with migration_lock(): + try: + command.downgrade(config, "base") + command.upgrade(config, "0023_service_actor_identity") + command.upgrade(config, "0025_artifact_store_v2") + v2_columns = asyncio.run(_fetch_columns(isolated_database_env)) + command.downgrade(config, "0023_service_actor_identity") + v1_columns = asyncio.run(_fetch_columns(isolated_database_env)) + command.upgrade(config, "0025_artifact_store_v2") + finally: + command.downgrade(config, "base") + + assert { + "artifact_storage_namespaces.namespace_fingerprint", + "artifact_upload_items.provider_object_ref", + "artifact_replicas.storage_namespace_id", + "artifact_replicas.namespace_fingerprint", + "artifact_replicas.provider_profile", + "artifact_replicas.provider_object_ref", + "artifact_operation_receipts.replayed", + }.issubset(v2_columns) + assert { + "artifact_upload_items.provider_operation_reference", + "artifact_replicas.provider_artifact_id", + "artifact_replicas.provider_manifest_id", + "artifact_replicas.retention_state", + "artifact_operation_receipts.provider_receipt_id", + "artifact_operation_receipts.retention_reference", + }.issubset(v1_columns) + assert { + "artifact_upload_items.provider_object_ref", + "artifact_replicas.storage_namespace_id", + "artifact_replicas.namespace_fingerprint", + "artifact_replicas.provider_profile", + "artifact_replicas.provider_object_ref", + "artifact_operation_receipts.provider_object_ref", + "artifact_operation_receipts.replayed", + }.isdisjoint(v1_columns) + assert "artifact_storage_namespaces.id" not in v1_columns + + +def test_artifact_store_v2_refuses_populated_v1_before_ddl( + isolated_database_env: str, + migration_lock, +) -> None: + """Never fabricate namespace or verification provenance for a v1 row.""" + config = _alembic_config() + with migration_lock(): + try: + command.downgrade(config, "base") + command.upgrade(config, "0023_service_actor_identity") + asyncio.run(_seed_artifact_content(isolated_database_env)) + with pytest.raises( + RuntimeError, + match="artifact storage clean cut requires empty pre-production tables", + ): + command.upgrade(config, "0025_artifact_store_v2") + refused = asyncio.run(_artifact_v2_refusal_state(isolated_database_env)) + asyncio.run(_truncate_artifact_foundation(isolated_database_env)) + command.upgrade(config, "0025_artifact_store_v2") + finally: + command.downgrade(config, "base") + + assert refused == { + "revision": "0023_service_actor_identity", + "namespace_table_exists": False, + "v1_content_count": 1, + } + + +def test_artifact_store_v2_refuses_populated_v2_downgrade_before_ddl( + isolated_database_env: str, + migration_lock, +) -> None: + """Never drop a namespace-only v2 deployment fence during downgrade.""" + config = _alembic_config() + with migration_lock(): + try: + command.downgrade(config, "base") + command.upgrade(config, "0025_artifact_store_v2") + asyncio.run(_seed_v2_artifact_namespace(isolated_database_env)) + with pytest.raises( + RuntimeError, + match="artifact storage clean cut requires empty pre-production tables", + ): + command.downgrade(config, "0023_service_actor_identity") + refused_revision = asyncio.run(_current_revision(isolated_database_env)) + refused_columns = asyncio.run(_fetch_columns(isolated_database_env)) + refused_namespace_count = asyncio.run( + _artifact_namespace_count(isolated_database_env) + ) + asyncio.run(_truncate_v2_artifact_namespace(isolated_database_env)) + command.downgrade(config, "0023_service_actor_identity") + finally: + asyncio.run(_truncate_v2_artifact_namespace(isolated_database_env)) + command.downgrade(config, "base") + + assert refused_revision == "0025_artifact_store_v2" + assert refused_namespace_count == 1 + assert "artifact_replicas.provider_object_ref" in refused_columns + + +def test_artifact_store_v2_waits_for_concurrent_v1_writer_and_refuses( + isolated_database_env: str, + migration_lock, +) -> None: + """Serialize the clean-cut emptiness check against every v1 writer.""" + config = _alembic_config() + inserted = threading.Event() + release_insert = threading.Event() + upgrade_started = threading.Event() + + def guarded_upgrade() -> None: + upgrade_started.set() + command.upgrade(config, "0025_artifact_store_v2") + + with migration_lock(): + try: + command.downgrade(config, "base") + command.upgrade(config, "0023_service_actor_identity") + with ThreadPoolExecutor(max_workers=2) as pool: + insert_future = pool.submit( + asyncio.run, + _insert_v1_artifact_content_until_released( + isolated_database_env, + inserted, + release_insert, + ), + ) + assert inserted.wait(timeout=5) + upgrade_future = pool.submit(guarded_upgrade) + assert upgrade_started.wait(timeout=5) + time.sleep(0.2) + assert upgrade_future.done() is False + release_insert.set() + insert_future.result(timeout=5) + with pytest.raises( + RuntimeError, + match="artifact storage clean cut requires empty pre-production tables", + ): + upgrade_future.result(timeout=5) + + refused = asyncio.run(_artifact_v2_refusal_state(isolated_database_env)) + asyncio.run(_truncate_artifact_foundation(isolated_database_env)) + command.upgrade(config, "0025_artifact_store_v2") + finally: + release_insert.set() + asyncio.run(_truncate_artifact_foundation(isolated_database_env)) + command.downgrade(config, "base") + + assert refused == { + "revision": "0023_service_actor_identity", + "namespace_table_exists": False, + "v1_content_count": 1, + } + + def test_api_rate_control_schema_preserves_domain_and_guards_downgrade( isolated_database_env: str, migration_lock, @@ -2946,6 +3116,114 @@ async def _artifact_table_counts(database_url: str) -> dict[str, int]: await engine.dispose() +async def _seed_artifact_content(database_url: str) -> None: + """Insert one content fact that a clean-cut migration must refuse.""" + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + await connection.execute( + text( + "insert into artifact_contents " + "(id, sha256, byte_count, media_type, normalized_display_name) " + "values (:id, :sha256, 1, 'application/octet-stream', 'legacy.bin')" + ), + {"id": str(uuid4()), "sha256": "sha256:" + "1" * 64}, + ) + finally: + await engine.dispose() + + +async def _seed_v2_artifact_namespace(database_url: str) -> None: + """Insert the v2-only namespace fact that downgrade must preserve.""" + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + await connection.execute( + text( + "insert into artifact_storage_namespaces " + "(id, backend, adapter, provider_profile, namespace_descriptor, " + "namespace_fingerprint) values " + "('primary', 'local', 'local', 'local-v2', '{}'::json, :fingerprint)" + ), + {"fingerprint": "sha256:" + "3" * 64}, + ) + finally: + await engine.dispose() + + +async def _artifact_namespace_count(database_url: str) -> int: + """Return the current v2 deployment-namespace fact count.""" + engine = create_async_engine(database_url) + try: + async with engine.connect() as connection: + count = await connection.scalar( + text("select count(*) from artifact_storage_namespaces") + ) + return int(count or 0) + finally: + await engine.dispose() + + +async def _truncate_v2_artifact_namespace(database_url: str) -> None: + """Clear the v2 namespace when its table exists during test cleanup.""" + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + exists = await connection.scalar( + text("select to_regclass('public.artifact_storage_namespaces') is not null") + ) + if exists: + await connection.execute( + text("truncate table artifact_storage_namespaces cascade") + ) + finally: + await engine.dispose() + + +async def _insert_v1_artifact_content_until_released( + database_url: str, + inserted: threading.Event, + release: threading.Event, +) -> None: + """Hold one uncommitted v1 writer while the clean-cut upgrade requests locks.""" + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + await connection.execute( + text( + "insert into artifact_contents " + "(id, sha256, byte_count, media_type, normalized_display_name) " + "values (:id, :sha256, 1, 'application/octet-stream', 'raced.bin')" + ), + {"id": str(uuid4()), "sha256": "sha256:" + "2" * 64}, + ) + inserted.set() + assert await asyncio.to_thread(release.wait, 5) + finally: + await engine.dispose() + + +async def _artifact_v2_refusal_state(database_url: str) -> dict[str, object]: + """Return transactional proof that refusal changed no v1 schema or data.""" + engine = create_async_engine(database_url) + try: + async with engine.connect() as connection: + revision = await connection.scalar(text("select version_num from alembic_version")) + namespace_table = await connection.scalar( + text("select to_regclass('public.artifact_storage_namespaces') is not null") + ) + content_count = await connection.scalar( + text("select count(*) from artifact_contents") + ) + return { + "revision": revision, + "namespace_table_exists": namespace_table, + "v1_content_count": content_count, + } + finally: + await engine.dispose() + + async def _assert_artifact_fact_guards(database_url: str, ids: dict[str, str]) -> None: """Exercise digest and immutable-row guards directly in PostgreSQL.""" engine = create_async_engine(database_url) diff --git a/backend/tests/test_artifact_architecture.py b/backend/tests/test_artifact_architecture.py new file mode 100644 index 000000000..2ad35c0d6 --- /dev/null +++ b/backend/tests/test_artifact_architecture.py @@ -0,0 +1,268 @@ +"""Static enforcement for the closed artifact capability boundary.""" + +from __future__ import annotations + +import ast +from pathlib import Path + +from app.interfaces import artifact_operations + + +BACKEND_ROOT = Path(__file__).parents[1] +APP_ROOT = BACKEND_ROOT / "app" +ARTIFACT_OPERATIONS = APP_ROOT / "interfaces" / "artifact_operations.py" +COMPOSITION_ROOT = APP_ROOT / "adapters" / "artifacts" / "__init__.py" +CLOSED_PORTS = { + "GuideArtifactIngestPort", + "ContributorArtifactUploadPort", + "ArtifactBindingPort", + "ArtifactMaterializationPort", + "CheckerArtifactOutputPort", + "ArtifactOperatorReadPort", + "ArtifactOperatorRecoveryPort", +} +CANONICAL_REQUESTS = { + "GuideArtifactIngestRequest", + "ArtifactBindingCreateRequest", + "ReadyUploadSetRequest", + "BindingMaterializationRequest", + "CheckerOutputArtifactRequest", + "ArtifactRecoveryRequest", +} +CANONICAL_TYPE_ALIASES = { + "ArtifactAuditResourceType", + "ArtifactBindingResourceType", +} +RAW_TYPES = {"ArtifactStore", "ArtifactStorageOrchestrator"} +PROVIDER_METHODS = {"put", "observe_put_result", "open", "head"} +CONCRETE_ADAPTER_MODULES = { + "app.adapters.artifacts.local", + "app.adapters.artifacts.s3_compatible", +} + + +def _python_files(*roots: Path) -> tuple[Path, ...]: + return tuple(sorted(path for root in roots for path in root.rglob("*.py"))) + + +def _tree(path: Path) -> ast.Module: + return ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + + +def _annotation_names(annotation: ast.expr | None) -> set[str]: + if annotation is None: + return set() + return { + node.id + for node in ast.walk(annotation) + if isinstance(node, ast.Name) + } | { + node.attr + for node in ast.walk(annotation) + if isinstance(node, ast.Attribute) + } + + +def test_product_api_and_workers_cannot_import_or_inject_raw_artifact_types() -> None: + product_modules = [ + path + for path in _python_files(APP_ROOT / "modules", APP_ROOT / "api", APP_ROOT / "workers") + if APP_ROOT / "modules" / "artifacts" not in path.parents + ] + violations: list[str] = [] + for path in product_modules: + tree = _tree(path) + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom): + imported = {alias.name for alias in node.names} + forbidden = imported & RAW_TYPES + if forbidden: + violations.append(f"{path.relative_to(BACKEND_ROOT)} imports {sorted(forbidden)}") + if ( + node.module == "app.modules.artifacts.service" + and any(alias.name == "*" for alias in node.names) + ): + violations.append( + f"{path.relative_to(BACKEND_ROOT)} imports broad artifact services" + ) + if isinstance(node, ast.Import): + if any(alias.name == "app.interfaces.artifacts" for alias in node.names): + violations.append( + f"{path.relative_to(BACKEND_ROOT)} imports the raw artifact module" + ) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + annotations = [node.returns] + annotations.extend(argument.annotation for argument in node.args.args) + annotations.extend(argument.annotation for argument in node.args.kwonlyargs) + for annotation in annotations: + forbidden = _annotation_names(annotation) & RAW_TYPES + if forbidden: + violations.append( + f"{path.relative_to(BACKEND_ROOT)} injects {sorted(forbidden)}" + ) + assert violations == [] + + +def test_concrete_adapter_construction_has_one_composition_path() -> None: + factory_calls: list[Path] = [] + adapter_calls: list[Path] = [] + concrete_imports: list[Path] = [] + for path in _python_files(APP_ROOT): + tree = _tree(path) + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom) and node.module in CONCRETE_ADAPTER_MODULES: + concrete_imports.append(path) + if isinstance(node, ast.Import) and any( + alias.name in CONCRETE_ADAPTER_MODULES for alias in node.names + ): + concrete_imports.append(path) + if not isinstance(node, ast.Call): + continue + called = node.func + if isinstance(called, ast.Subscript): + called = called.value + if isinstance(called, ast.Name): + name = called.id + elif isinstance(called, ast.Attribute): + name = called.attr + else: + name = None + if name == "ExternalServiceAdapterFactory": + factory_calls.append(path) + if name in {"LocalStorageAdapter", "S3CompatibleArtifactStore"}: + adapter_calls.append(path) + + assert factory_calls == [COMPOSITION_ROOT] + assert adapter_calls == [COMPOSITION_ROOT] + assert concrete_imports == [COMPOSITION_ROOT] + + +def test_provider_methods_stay_inside_artifact_orchestration_and_adapters() -> None: + violations: list[str] = [] + allowed_roots = { + APP_ROOT / "adapters" / "artifacts", + APP_ROOT / "modules" / "artifacts", + } + for path in _python_files(APP_ROOT): + if any(root in path.parents for root in allowed_roots): + continue + tree = _tree(path) + artifact_store_receivers = { + argument.arg + for node in ast.walk(tree) + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + for argument in (*node.args.args, *node.args.kwonlyargs) + if "ArtifactStore" in _annotation_names(argument.annotation) + } + for node in ast.walk(tree): + if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute): + continue + if node.func.attr not in PROVIDER_METHODS: + continue + if not isinstance(node.func.value, ast.Name): + continue + if node.func.value.id not in artifact_store_receivers: + continue + violations.append( + f"{path.relative_to(BACKEND_ROOT)} calls provider method {node.func.attr}" + ) + assert violations == [] + + +def test_artifact_operations_exports_only_canonical_closed_contracts() -> None: + assert set(artifact_operations.__all__) == ( + CLOSED_PORTS | CANONICAL_REQUESTS | CANONICAL_TYPE_ALIASES + ) + tree = _tree(ARTIFACT_OPERATIONS) + protocol_names = { + node.name + for node in tree.body + if isinstance(node, ast.ClassDef) + and any(isinstance(base, ast.Name) and base.id == "Protocol" for base in node.bases) + } + request_names = { + node.name + for node in tree.body + if isinstance(node, ast.ClassDef) and node.name.endswith("Request") + } + assert protocol_names == CLOSED_PORTS + assert request_names == CANONICAL_REQUESTS + + exported_names = { + element.value + for node in tree.body + if isinstance(node, ast.Assign) + and any(isinstance(target, ast.Name) and target.id == "__all__" for target in node.targets) + and isinstance(node.value, (ast.Tuple, ast.List)) + for element in node.value.elts + if isinstance(element, ast.Constant) and isinstance(element.value, str) + } + assert exported_names == CLOSED_PORTS | CANONICAL_REQUESTS | CANONICAL_TYPE_ALIASES + + forbidden_fields = { + "adapter", + "provider_object_ref", + "storage_namespace", + "scope_map", + "server_content_id", + } + fields = { + node.target.id + for class_node in tree.body + if isinstance(class_node, ast.ClassDef) and class_node.name in CANONICAL_REQUESTS + for node in class_node.body + if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name) + } + assert fields.isdisjoint(forbidden_fields) + + operator_read = next( + node + for node in tree.body + if isinstance(node, ast.ClassDef) and node.name == "ArtifactOperatorReadPort" + ) + operator_methods = { + node.name + for node in operator_read.body + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + } + assert operator_methods == { + "list_bindings", + "list_replicas", + "list_receipts", + "get_verification_job", + "get_recovery_attempt", + "list_audit_events", + "admission_usage", + } + resource_type_annotations = { + annotation.id + for method in operator_read.body + if isinstance(method, (ast.FunctionDef, ast.AsyncFunctionDef)) + for argument in method.args.kwonlyargs + if argument.arg == "resource_type" + for annotation in [argument.annotation] + if isinstance(annotation, ast.Name) + } + assert resource_type_annotations == { + "ArtifactAuditResourceType", + "ArtifactBindingResourceType", + } + + +def test_scratch_cleanup_worker_has_no_product_or_database_state() -> None: + path = APP_ROOT / "workers" / "artifacts.py" + forbidden_import_prefixes = ( + "app.db", + "app.modules.actors", + "app.modules.audit", + "app.modules.authorization", + ) + imported_modules: set[str] = set() + for node in ast.walk(_tree(path)): + if isinstance(node, ast.ImportFrom) and node.module is not None: + imported_modules.add(node.module) + elif isinstance(node, ast.Import): + imported_modules.update(alias.name for alias in node.names) + assert not any( + module.startswith(forbidden_import_prefixes) for module in imported_modules + ) diff --git a/backend/tests/test_artifact_cleanup_wiring.py b/backend/tests/test_artifact_cleanup_wiring.py new file mode 100644 index 000000000..e13646a94 --- /dev/null +++ b/backend/tests/test_artifact_cleanup_wiring.py @@ -0,0 +1,325 @@ +"""Composition tests for artifact startup and scratch cleanup ownership.""" + +from __future__ import annotations + +import importlib +from pathlib import Path +from types import ModuleType + +import pytest + +from app.adapters.artifacts import artifact_preparation_limits +from app.core.config import Settings, get_settings +from app.main import create_app +from app.interfaces.external_services import UnknownExternalServiceProviderError + + +def _enabled_settings(tmp_path: Path, **changes: object) -> Settings: + values: dict[str, object] = { + "environment": "test", + "artifact_store_backend": "local", + "artifact_local_root": tmp_path / "store", + "artifact_scratch_root": tmp_path / "scratch", + } + values.update(changes) + return Settings(**values) + + +class _Store: + def __init__(self, events: list[str]) -> None: + self.events = events + + def close(self) -> None: + self.events.append("close") + + def initialize_after_namespace_claim(self, claim: object) -> _Store: + assert claim == "claim" + self.events.append("initialize") + return self + + +async def test_enabled_startup_validates_namespace_then_cleans_before_yield( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + import app.main as main_module + + events: list[str] = [] + store = _Store(events) + + async def validate(settings: Settings, candidate: object) -> str: + assert settings.artifact_store_backend == "local" + assert candidate is store + events.append("namespace") + return "claim" + + def create(_settings: Settings) -> _Store: + events.append("construct") + return store + + async def cleanup(settings: Settings) -> int: + assert settings.artifact_store_backend == "local" + events.append("cleanup") + return 0 + + monkeypatch.setattr(main_module, "create_artifact_store_bootstrap", create) + monkeypatch.setattr(main_module, "_validate_artifact_storage_namespace_at_startup", validate) + monkeypatch.setattr(main_module, "cleanup_stale_artifact_scratch", cleanup) + app = create_app(_enabled_settings(tmp_path)) + + async with app.router.lifespan_context(app): + events.append("yield") + + assert events == ["construct", "namespace", "initialize", "cleanup", "close", "yield"] + + +async def test_production_auth_validation_precedes_artifact_startup( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + import app.main as main_module + + events: list[str] = [] + settings = Settings( + environment="production", + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", + ) + app = create_app(settings) + store = _Store(events) + + def validate_auth(_settings: Settings) -> None: + events.append("auth") + + async def validate_namespace(candidate_settings: Settings, candidate: object) -> str: + assert candidate_settings is settings + assert candidate is store + events.append("namespace") + return "claim" + + def create(_settings: Settings) -> _Store: + events.append("construct") + return store + + async def cleanup(_settings: Settings) -> int: + events.append("cleanup") + return 0 + + monkeypatch.setattr(main_module, "build_auth_verifier", validate_auth) + monkeypatch.setattr(main_module, "create_artifact_store_bootstrap", create) + monkeypatch.setattr( + main_module, + "_validate_artifact_storage_namespace_at_startup", + validate_namespace, + ) + monkeypatch.setattr(main_module, "cleanup_stale_artifact_scratch", cleanup) + + async with app.router.lifespan_context(app): + events.append("yield") + + assert events == [ + "auth", + "construct", + "namespace", + "initialize", + "cleanup", + "close", + "yield", + ] + + +async def test_disabled_startup_does_not_construct_or_clean_artifacts( + monkeypatch: pytest.MonkeyPatch, +) -> None: + import app.main as main_module + + def unexpected(*_args: object, **_kwargs: object) -> None: + raise AssertionError("disabled artifact storage activated") + + monkeypatch.setattr(main_module, "create_artifact_store_bootstrap", unexpected) + monkeypatch.setattr(main_module, "cleanup_stale_artifact_scratch", unexpected) + app = create_app(Settings(environment="test")) + + async with app.router.lifespan_context(app): + pass + + +async def test_unimplemented_s3_startup_fails_through_typed_factory( + tmp_path: Path, +) -> None: + """Keep real startup on the one ADR 0014 unavailable-provider path.""" + app = create_app( + Settings( + environment="test", + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", + ) + ) + + with pytest.raises(UnknownExternalServiceProviderError) as caught: + async with app.router.lifespan_context(app): + pass + + assert caught.value.identity is not None + assert caught.value.identity.provider_key == "s3_compatible" + + +async def test_namespace_failure_is_fail_closed_and_skips_cleanup( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + import app.main as main_module + + events: list[str] = [] + store = _Store(events) + + async def fail_namespace(_settings: Settings, candidate: object) -> None: + assert candidate is store + events.append("namespace") + raise RuntimeError("namespace mismatch") + + def create(_settings: Settings) -> _Store: + events.append("construct") + return store + + async def unexpected_cleanup(_settings: Settings) -> int: + raise AssertionError("cleanup ran after namespace failure") + + monkeypatch.setattr(main_module, "create_artifact_store_bootstrap", create) + monkeypatch.setattr( + main_module, + "_validate_artifact_storage_namespace_at_startup", + fail_namespace, + ) + monkeypatch.setattr(main_module, "cleanup_stale_artifact_scratch", unexpected_cleanup) + app = create_app(_enabled_settings(tmp_path)) + + with pytest.raises(RuntimeError, match="namespace mismatch"): + async with app.router.lifespan_context(app): + pass + + assert events == ["construct", "namespace", "close"] + assert not (tmp_path / "store").exists() + + +async def test_cleanup_failure_prevents_startup_and_closes_store( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + import app.main as main_module + + events: list[str] = [] + store = _Store(events) + + async def validate(_settings: Settings, candidate: object) -> str: + assert candidate is store + events.append("namespace") + return "claim" + + def create(_settings: Settings) -> _Store: + events.append("construct") + return store + + async def fail_cleanup(_settings: Settings) -> int: + events.append("cleanup") + raise RuntimeError("cleanup failed") + + monkeypatch.setattr(main_module, "create_artifact_store_bootstrap", create) + monkeypatch.setattr(main_module, "_validate_artifact_storage_namespace_at_startup", validate) + monkeypatch.setattr(main_module, "cleanup_stale_artifact_scratch", fail_cleanup) + app = create_app(_enabled_settings(tmp_path)) + + with pytest.raises(RuntimeError, match="cleanup failed"): + async with app.router.lifespan_context(app): + pass + + assert events == ["construct", "namespace", "initialize", "cleanup", "close"] + + +def test_preparation_limit_mapping_uses_every_canonical_setting(tmp_path: Path) -> None: + settings = _enabled_settings( + tmp_path, + artifact_scratch_aggregate_reserved_bytes=3 * 512 * 1024 * 1024, + artifact_scratch_maximum_files=7, + artifact_scratch_maximum_concurrency=3, + artifact_scratch_minimum_free_bytes=123, + artifact_scratch_reservation_ttl_seconds=3000, + artifact_preparation_total_deadline_seconds=2000, + artifact_scratch_cleanup_margin_seconds=400, + artifact_stream_buffer_bytes=8192, + artifact_maximum_bytes=64 * 1024 * 1024, + ) + + limits = artifact_preparation_limits(settings) + + assert limits.aggregate_reserved_bytes == settings.artifact_scratch_aggregate_reserved_bytes + assert limits.maximum_files == settings.artifact_scratch_maximum_files + assert limits.maximum_concurrency == settings.artifact_scratch_maximum_concurrency + assert limits.minimum_free_bytes == settings.artifact_scratch_minimum_free_bytes + assert limits.reservation_ttl_seconds == settings.artifact_scratch_reservation_ttl_seconds + assert limits.total_deadline_seconds == settings.artifact_preparation_total_deadline_seconds + assert limits.cleanup_margin_seconds == settings.artifact_scratch_cleanup_margin_seconds + assert limits.stream_buffer_bytes == settings.artifact_stream_buffer_bytes + assert limits.maximum_source_bytes == settings.artifact_maximum_bytes + + +def _load_worker_modules( + monkeypatch: pytest.MonkeyPatch, + request: pytest.FixtureRequest, +) -> tuple[ModuleType, ModuleType]: + monkeypatch.setenv("WORKSTREAM_CELERY_TASK_ALWAYS_EAGER", "true") + get_settings.cache_clear() + request.addfinalizer(get_settings.cache_clear) + celery_module = importlib.import_module("app.workers.celery_app") + celery_module = importlib.reload(celery_module) + artifacts_module = importlib.import_module("app.workers.artifacts") + artifacts_module = importlib.reload(artifacts_module) + return celery_module, artifacts_module + + +def test_celery_registers_named_task_include_and_exact_beat_cadence( + monkeypatch: pytest.MonkeyPatch, + request: pytest.FixtureRequest, +) -> None: + celery_module, artifacts_module = _load_worker_modules(monkeypatch, request) + settings = Settings( + environment="test", + celery_task_always_eager=True, + artifact_scratch_cleanup_interval_seconds=137, + ) + monkeypatch.setattr(celery_module, "get_settings", lambda: settings) + + app = celery_module.create_celery_app() + schedule = app.conf.beat_schedule[celery_module.ARTIFACT_SCRATCH_CLEANUP_SCHEDULE] + + assert "app.workers.artifacts" in app.conf.include + assert schedule["task"] == "workstream.artifacts.cleanup_stale_scratch" + assert schedule["schedule"] == 137 + assert artifacts_module.cleanup_stale_scratch.name == schedule["task"] + assert schedule["task"] in celery_module.celery_app.tasks + + +def test_cleanup_task_runs_shared_helper_and_propagates_failure( + monkeypatch: pytest.MonkeyPatch, + request: pytest.FixtureRequest, + tmp_path: Path, +) -> None: + _celery_module, artifacts_module = _load_worker_modules(monkeypatch, request) + settings = _enabled_settings(tmp_path) + calls: list[Settings] = [] + + async def cleanup(candidate: Settings) -> int: + calls.append(candidate) + return 4 + + monkeypatch.setattr(artifacts_module, "get_settings", lambda: settings) + monkeypatch.setattr(artifacts_module, "cleanup_stale_artifact_scratch", cleanup) + assert artifacts_module.cleanup_stale_scratch.run() == 4 + assert calls == [settings] + + async def fail_cleanup(_candidate: Settings) -> int: + raise RuntimeError("scratch cleanup failed") + + monkeypatch.setattr(artifacts_module, "cleanup_stale_artifact_scratch", fail_cleanup) + with pytest.raises(RuntimeError, match="scratch cleanup failed"): + artifacts_module.cleanup_stale_scratch.run() diff --git a/backend/tests/test_artifact_store_conformance.py b/backend/tests/test_artifact_store_conformance.py new file mode 100644 index 000000000..27acb8c85 --- /dev/null +++ b/backend/tests/test_artifact_store_conformance.py @@ -0,0 +1,336 @@ +"""Provider-neutral ArtifactStore v2 conformance vectors.""" + +from __future__ import annotations + +import asyncio +from dataclasses import FrozenInstanceError +import hashlib +import inspect +from pathlib import Path + +import pytest + +from app.interfaces.artifacts import ( + ArtifactByteRange, + ArtifactObjectHead, + ArtifactObjectMissingError, + ArtifactPutObservation, + ArtifactPutResult, + ArtifactRangeInvalidError, + ArtifactStore, + ArtifactStoreNamespaceClaim, + ArtifactStoreNamespaceIdentity, +) +from app.interfaces.external_services import ExternalServiceAdapterIdentity +from tests.artifact_store_helpers import ( + initialize_local_store, + minted_source, +) + + +class ArtifactStoreConformanceTests: + """Reusable v2 vectors for LocalStorage and later provider adapters.""" + + expected_identity: ExternalServiceAdapterIdentity + + def make_store(self, root: Path) -> ArtifactStore: + """Construct the provider under test.""" + raise NotImplementedError + + @staticmethod + def close_store(store: ArtifactStore) -> None: + """Release optional provider resources after one vector.""" + close = getattr(store, "close", None) + if close is not None: + close() + + @pytest.mark.asyncio + async def test_identity_and_exact_public_surface(self, tmp_path: Path) -> None: + """Expose only shared identity and the four ArtifactStore v2 operations.""" + store = self.make_store(tmp_path / "store") + try: + assert store.identity == self.expected_identity + public = { + name + for name in dir(store) + if not name.startswith("_") and name not in {"close"} + } + assert public == {"head", "identity", "observe_put_result", "open", "put"} + for removed in ( + "store", + "recover_committed_store", + "stat", + "verify", + "retain", + "release", + "get_operation_receipt", + ): + assert not hasattr(store, removed) + finally: + self.close_store(store) + + @pytest.mark.asyncio + async def test_put_head_observe_and_full_open(self, tmp_path: Path) -> None: + """Publish exact bytes and expose bounded provider-neutral facts.""" + store = self.make_store(tmp_path / "store") + content = b"hello" + expected_digest = hashlib.sha256(content).hexdigest() + try: + async with minted_source( + tmp_path / "scratch", + b"hel", + b"lo", + media_type="text/plain", + ) as source: + commitment = source.commitment + result = await store.put(source) + assert result == ArtifactPutResult( + f"sha256/{expected_digest[:2]}/{expected_digest[2:]}", + replayed=False, + ) + head = await store.head(result.provider_object_ref) + assert head.exists is True + assert head.byte_count == len(content) + assert head.media_type is None + assert await store.observe_put_result(commitment) == ArtifactPutObservation( + result.provider_object_ref, + committed=True, + ) + chunks = [chunk async for chunk in store.open(result.provider_object_ref)] + assert b"".join(chunks) == content + assert all(0 < len(chunk) <= 2 for chunk in chunks) + finally: + self.close_store(store) + + @pytest.mark.asyncio + async def test_exact_replay_returns_same_reference_without_overwrite( + self, + tmp_path: Path, + ) -> None: + """Verify all existing bytes before accepting a commitment replay.""" + store = self.make_store(tmp_path / "store") + try: + async with minted_source(tmp_path / "scratch-a", b"replay") as first_source: + first = await store.put(first_source) + async with minted_source(tmp_path / "scratch-b", b"re", b"play") as replay_source: + replay = await store.put(replay_source) + assert replay.provider_object_ref == first.provider_object_ref + assert first.replayed is False + assert replay.replayed is True + assert b"".join( + [chunk async for chunk in store.open(first.provider_object_ref)] + ) == b"replay" + finally: + self.close_store(store) + + @pytest.mark.asyncio + async def test_concurrent_identical_puts_publish_once(self, tmp_path: Path) -> None: + """Allow one immutable publication and return one exact replay.""" + first_store = self.make_store(tmp_path / "store") + second_store = self.make_store(tmp_path / "store") + try: + async with minted_source( + tmp_path / "scratch-a", + b"same bytes", + media_type="text/plain", + ) as first_source: + async with minted_source( + tmp_path / "scratch-b", + b"same ", + b"bytes", + media_type="text/plain", + ) as second_source: + first, second = await asyncio.gather( + first_store.put(first_source), + second_store.put(second_source), + ) + assert first.provider_object_ref == second.provider_object_ref + assert sorted((first.replayed, second.replayed)) == [False, True] + finally: + self.close_store(first_store) + self.close_store(second_store) + + @pytest.mark.asyncio + async def test_missing_head_observation_and_open(self, tmp_path: Path) -> None: + """Distinguish safe absence from operations that require existing bytes.""" + store = self.make_store(tmp_path / "store") + try: + async with minted_source(tmp_path / "scratch", b"absent") as source: + commitment = source.commitment + provider_object_ref = ( + f"sha256/{commitment.sha256[7:9]}/{commitment.sha256[9:]}" + ) + assert (await store.head(provider_object_ref)).exists is False + assert await store.observe_put_result(commitment) == ArtifactPutObservation( + provider_object_ref, + committed=False, + ) + with pytest.raises(ArtifactObjectMissingError): + _ = [chunk async for chunk in store.open(provider_object_ref)] + finally: + self.close_store(store) + + @pytest.mark.asyncio + async def test_ranges_cover_zero_eof_and_invalid_offsets(self, tmp_path: Path) -> None: + """Apply zero-based exclusive-end range semantics exactly.""" + store = self.make_store(tmp_path / "store") + try: + async with minted_source(tmp_path / "scratch", b"abcdef") as source: + result = await store.put(source) + + async def read(selected: ArtifactByteRange) -> bytes: + return b"".join( + [chunk async for chunk in store.open(result.provider_object_ref, selected)] + ) + + assert await read(ArtifactByteRange(offset=2, length=3)) == b"cde" + assert await read(ArtifactByteRange(offset=4)) == b"ef" + assert await read(ArtifactByteRange(offset=1, length=99)) == b"bcdef" + assert await read(ArtifactByteRange(offset=0, length=0)) == b"" + assert await read(ArtifactByteRange(offset=6)) == b"" + assert await read(ArtifactByteRange(offset=6, length=1)) == b"" + with pytest.raises(ArtifactRangeInvalidError): + await read(ArtifactByteRange(offset=7)) + finally: + self.close_store(store) + + @pytest.mark.asyncio + async def test_empty_object_is_present_and_readable(self, tmp_path: Path) -> None: + """Keep a committed zero-byte object distinct from a missing object.""" + store = self.make_store(tmp_path / "store") + try: + async with minted_source(tmp_path / "scratch", b"") as source: + result = await store.put(source) + commitment = source.commitment + assert (await store.head(result.provider_object_ref)).byte_count == 0 + assert await store.observe_put_result(commitment) == ArtifactPutObservation( + result.provider_object_ref, + committed=True, + ) + assert [chunk async for chunk in store.open(result.provider_object_ref)] == [] + finally: + self.close_store(store) + + +class TestLocalArtifactStoreConformance(ArtifactStoreConformanceTests): + """Run the shared v2 contract against LocalStorage.""" + + expected_identity = ExternalServiceAdapterIdentity("artifact_store", "local") + + def make_store(self, root: Path) -> ArtifactStore: + """Construct one small-buffer local provider.""" + return initialize_local_store(root, buffer_bytes=2) + + +def test_v2_value_types_are_immutable_and_range_values_are_exact() -> None: + """Reject mutation, booleans, negative values, and ambiguous missing heads.""" + result = ArtifactPutResult("sha256/00/" + "0" * 62, replayed=False) + with pytest.raises(FrozenInstanceError): + result.replayed = True # type: ignore[misc] + for kwargs in ({"offset": True}, {"offset": -1}, {"length": True}, {"length": -1}): + with pytest.raises(ValueError): + ArtifactByteRange(**kwargs) # type: ignore[arg-type] + for provider_ref in ("", "x" * 1025, "sha256/00/" + "0" * 61 + "\n", 7): + with pytest.raises(ValueError, match="provider object reference"): + ArtifactPutResult(provider_ref, replayed=False) # type: ignore[arg-type] + with pytest.raises(ValueError, match="replay observation"): + ArtifactPutResult("sha256/00/" + "0" * 62, replayed=1) # type: ignore[arg-type] + with pytest.raises(ValueError, match="put observation"): + ArtifactPutObservation("sha256/00/" + "0" * 62, committed=1) # type: ignore[arg-type] + + +def test_namespace_startup_values_are_closed_and_immutable() -> None: + """Reject malformed provider descriptors and namespace admission proofs.""" + identity = ArtifactStoreNamespaceIdentity( + provider_profile="local-v2", + descriptor_items=( + ("private_prefix", "objects/sha256"), + ("private_root_identity", "sha256:" + "0" * 64), + ), + ) + claim = ArtifactStoreNamespaceClaim( + adapter_identity=ExternalServiceAdapterIdentity("artifact_store", "local"), + namespace_identity=identity, + namespace_fingerprint="sha256:" + "0" * 64, + ) + with pytest.raises(FrozenInstanceError): + claim.namespace_fingerprint = "sha256:" + "1" * 64 # type: ignore[misc] + assert identity.as_dict() == { + "private_prefix": "objects/sha256", + "private_root_identity": "sha256:" + "0" * 64, + } + + for profile, items in ( + ("", (("private_prefix", "objects/sha256"),)), + ("local-v2", (("z", "1"), ("a", "2"))), + ("local-v2", (("a", "1"), ("a", "2"))), + ("local-v2", (("", "1"),)), + ("local-v2", (("backend", "local"),)), + ("local-v2", (("private_prefix", "objects/sha256"),)), + ( + "local-v2", + ( + ("private_prefix", "objects/sha256"), + ("private_root_identity", "sha256:" + "0" * 64), + ("unknown", "value"), + ), + ), + ( + "unknown-v1", + ( + ("private_prefix", "objects/sha256"), + ("private_root_identity", "sha256:" + "0" * 64), + ), + ), + ): + with pytest.raises(ValueError): + ArtifactStoreNamespaceIdentity(profile, items) + with pytest.raises(ValueError, match="not immutable"): + ArtifactStoreNamespaceIdentity( + "local-v2", + [ # type: ignore[arg-type] + ("private_prefix", "objects/sha256"), + ("private_root_identity", "sha256:" + "0" * 64), + ], + ) + for kwargs in ( + {"adapter_identity": object()}, + {"namespace_identity": object()}, + {"namespace_fingerprint": "sha256:bad"}, + {"namespace_fingerprint": "sha256:" + "G" * 64}, + ): + values = { + "adapter_identity": claim.adapter_identity, + "namespace_identity": identity, + "namespace_fingerprint": claim.namespace_fingerprint, + **kwargs, + } + with pytest.raises(ValueError): + ArtifactStoreNamespaceClaim(**values) # type: ignore[arg-type] + with pytest.raises(ValueError, match="existence observation"): + ArtifactObjectHead("sha256/00/" + "0" * 62, exists=1) # type: ignore[arg-type] + with pytest.raises(ValueError, match="exact byte count"): + ArtifactObjectHead("sha256/00/" + "0" * 62, exists=True) + with pytest.raises(ValueError, match="cannot contain"): + ArtifactObjectHead( + "sha256/00/" + "0" * 62, + exists=False, + media_type="text/plain", + ) + with pytest.raises(ValueError, match="media type"): + ArtifactObjectHead( + "sha256/00/" + "0" * 62, + exists=True, + byte_count=1, + media_type="bad\nmedia", + ) + + +def test_artifact_store_protocol_declares_only_v2_operations() -> None: + """Prevent v1 operation or receipt methods from returning to the port.""" + methods = { + name + for name, value in inspect.getmembers(ArtifactStore) + if not name.startswith("_") and (inspect.isfunction(value) or isinstance(value, property)) + } + assert methods == {"head", "identity", "observe_put_result", "open", "put"} diff --git a/backend/tests/test_artifacts.py b/backend/tests/test_artifacts.py index b663acd48..47a2086af 100644 --- a/backend/tests/test_artifacts.py +++ b/backend/tests/test_artifacts.py @@ -1,1926 +1,1107 @@ +"""PostgreSQL integration tests for the ArtifactStore v2 orchestration boundary.""" + from __future__ import annotations import asyncio -from collections.abc import AsyncIterator, Iterator -from dataclasses import replace +from collections.abc import AsyncIterator from datetime import UTC, datetime, timedelta -import hashlib -import json -import os from pathlib import Path -import stat as stat_module -import threading +from unittest.mock import AsyncMock from uuid import uuid4 from alembic import command from alembic.config import Config -from jsonschema import Draft202012Validator import pytest -from sqlalchemy import func, select -from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine +from sqlalchemy import select, text +from sqlalchemy.exc import IntegrityError +from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine -from app.adapters.artifacts.local import LocalStorageAdapter +from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap +from app.core.config import Settings from app.core.hashing import canonical_json_hash from app.interfaces.artifacts import ( ArtifactByteRange, - ArtifactIdempotencyMismatchError, + ArtifactConfigurationError, ArtifactInputMismatchError, ArtifactIntegrityError, - ArtifactLimitExceededError, - ArtifactNotFoundError, - ArtifactRetentionConflictError, - ArtifactOperation, - ArtifactMalformedRequestError, + ArtifactObjectHead, + ArtifactPutObservation, + ArtifactPutResult, ArtifactStoreUnavailableError, - ArtifactThrottledError, - IdempotencyIdentity, - IntegrityState, - StoreArtifactRequest, - canonical_release_request_digest, - canonical_retain_request_digest, - canonical_store_request_digest, - canonical_verify_request_digest, + ArtifactStoreNamespaceIdentity, ) +from app.interfaces.external_services import ExternalServiceAdapterIdentity from app.modules.artifacts.models import ( - ArtifactBinding, ArtifactContent, ArtifactOperationReceipt, ArtifactReplica, + ArtifactStorageNamespace, ArtifactUploadItem, ArtifactUploadSession, ) -from app.modules.artifacts.service import ArtifactIngestService +from app.modules.artifacts.service import ( + ArtifactIngestStateError, + ArtifactStorageNamespaceError, + ArtifactStorageNamespaceSpec, + ArtifactStorageOrchestrator, + ProviderAttemptFence, + artifact_storage_namespace_spec, + validate_artifact_storage_namespace_at_startup, +) +import app.modules.artifacts.service as artifact_service_module +from app.modules.artifacts.sources import ArtifactCommitment, CommittedArtifactSource from app.modules.projects.models import Project -from app.modules.tasks.models import AuditEvent - - -async def byte_stream(*chunks: bytes) -> AsyncIterator[bytes]: - """Yield exact byte chunks for adapter tests.""" - for chunk in chunks: - yield chunk - - -def store_request( - *, - key: str = "store-1", - request_value: str = "same", - expected_sha256: str | None = None, - expected_size: int | None = None, - maximum_bytes: int = 64, -) -> StoreArtifactRequest: - """Build one canonical store request.""" - identity = IdempotencyIdentity( - service_principal="workstream.artifact.ingest", - operation=ArtifactOperation.STORE, - key=key, - request_digest="sha256:" + "0" * 64, - ) - request = StoreArtifactRequest( - expected_sha256=expected_sha256, - expected_size=expected_size, - maximum_bytes=maximum_bytes, - media_type="application/octet-stream", - metadata={"request": request_value}, - idempotency=identity, - ) - return replace( - request, - idempotency=replace(identity, request_digest=canonical_store_request_digest(request)), - ) +from tests.artifact_store_helpers import ( + local_namespace_claim, + minted_source as _minted_source, +) -def verify_identity( - provider_artifact_id: str, - expected_sha256: str, - expected_size: int, - *, - key: str = "verify-1", - principal: str = "workstream.artifact.ingest", -) -> IdempotencyIdentity: - """Build one canonical verify identity.""" - return IdempotencyIdentity( - principal, - ArtifactOperation.VERIFY, - key, - canonical_verify_request_digest(provider_artifact_id, expected_sha256, expected_size), - ) +def _alembic_config() -> Config: + root = Path(__file__).resolve().parents[1] + config = Config(str(root / "alembic.ini")) + config.set_main_option("script_location", str(root / "alembic")) + return config -def retain_identity( - provider_artifact_id: str, - reference: str, - retention_class: str, - *, - key: str = "retain-1", - principal: str = "workstream.artifact.ingest", -) -> IdempotencyIdentity: - """Build one canonical retain identity.""" - return IdempotencyIdentity( - principal, - ArtifactOperation.RETAIN, - key, - canonical_retain_request_digest(provider_artifact_id, reference, retention_class), - ) +@pytest.fixture +def artifact_database_env(isolated_database_env: str, migration_lock) -> str: + """Provide one fully migrated empty PostgreSQL database per test.""" + config = _alembic_config() + with migration_lock(): + asyncio.run(_truncate_artifacts_if_present(isolated_database_env)) + command.downgrade(config, "base") + command.upgrade(config, "head") + yield isolated_database_env + with migration_lock(): + asyncio.run(_truncate_artifacts(isolated_database_env)) + command.downgrade(config, "base") -def release_identity( - provider_artifact_id: str, - reference: str, - *, - key: str = "release-1", - principal: str = "workstream.artifact.ingest", -) -> IdempotencyIdentity: - """Build one canonical release identity.""" - return IdempotencyIdentity( - principal, - ArtifactOperation.RELEASE, - key, - canonical_release_request_digest(provider_artifact_id, reference), - ) +async def _truncate_artifacts(database_url: str) -> None: + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + await connection.execute( + text( + "truncate table artifact_storage_namespaces, artifact_upload_sessions, " + "artifact_contents cascade" + ) + ) + finally: + await engine.dispose() -def test_versioned_contract_fixtures_validate() -> None: - """Validate every positive and negative JSON Schema vector.""" - root = Path(__file__).resolve().parents[2] / "contracts" / "artifact-store" / "version_1" - schema = json.loads((root / "schema" / "contract.schema.json").read_text()) - valid = json.loads((root / "fixtures" / "valid.json").read_text()) - invalid = json.loads((root / "fixtures" / "invalid.json").read_text()) - for fixture in valid["fixtures"]: - definition = schema["$defs"][fixture["schema_ref"].rsplit("/", 1)[-1]] - Draft202012Validator({**schema, **definition}).validate(fixture["instance"]) - for fixture in invalid["fixtures"]: - definition = schema["$defs"][fixture["schema_ref"].rsplit("/", 1)[-1]] - errors = list(Draft202012Validator({**schema, **definition}).iter_errors(fixture["instance"])) - assert errors, fixture["name"] - assert any(error.validator == fixture["expected_keyword"] for error in errors) - - error_types = { - "malformed": ArtifactMalformedRequestError, - "oversized": ArtifactLimitExceededError, - "input_mismatch": ArtifactInputMismatchError, - "integrity": ArtifactIntegrityError, - "not_found": ArtifactNotFoundError, - "replay_conflict": ArtifactIdempotencyMismatchError, - "throttled": ArtifactThrottledError, - "provider_unavailable": ArtifactStoreUnavailableError, - } - for vector in invalid["error_vectors"]: - error_type = error_types[vector["name"]] - assert error_type.code == vector["code"] - assert error_type.category == vector["category"] - assert error_type.retryable is vector["retryable"] - - -def test_canonical_digest_vectors_use_shared_hashing() -> None: - """Prove contract digests use Workstream's shared canonical helper.""" - fixture = ( - Path(__file__).resolve().parents[2] - / "contracts" - / "artifact-store" - / "version_1" - / "fixtures" - / "canonical-digests.json" +async def _truncate_artifacts_if_present(database_url: str) -> None: + """Clear only artifact tables that exist at the database's current revision.""" + engine = create_async_engine(database_url) + candidates = ( + "artifact_storage_namespaces", + "artifact_upload_sessions", + "artifact_contents", ) - for vector in json.loads(fixture.read_text())["vectors"]: - assert canonical_json_hash(vector["input"]) == vector["digest"] + try: + async with engine.begin() as connection: + existing = [ + table_name + for table_name in candidates + if await connection.scalar( + text("select to_regclass(:table_name) is not null"), + {"table_name": f"public.{table_name}"}, + ) + ] + if existing: + await connection.execute(text(f"truncate table {', '.join(existing)} cascade")) + finally: + await engine.dispose() -@pytest.mark.asyncio -async def test_executable_conformance_vectors(tmp_path: Path) -> None: - """Execute every named provider conformance vector against the local adapter.""" - fixture_path = ( - Path(__file__).resolve().parents[2] - / "contracts" - / "artifact-store" - / "version_1" - / "fixtures" - / "conformance.json" +def _settings(tmp_path: Path) -> Settings: + (tmp_path / "store").mkdir(mode=0o700, exist_ok=True) + return Settings( + environment="test", + artifact_store_backend="local", + artifact_local_root=tmp_path / "store", + artifact_scratch_root=tmp_path / "scratch", + artifact_scratch_minimum_free_bytes=0, ) - vectors = {vector["name"]: vector for vector in json.loads(fixture_path.read_text())["vectors"]} - executed: set[str] = set() - - adapter = LocalStorageAdapter(root=tmp_path / "store", buffer_bytes=2) - request = store_request(key="conformance-store", maximum_bytes=4) - stored = await adapter.store(byte_stream(b""), request) - replay = await adapter.store(byte_stream(b""), request) - assert replay.replayed and replay.receipt.receipt_id == stored.receipt.receipt_id - executed.add("store_exact_replay") - - consumed = False - - async def untouched() -> AsyncIterator[bytes]: - nonlocal consumed - consumed = True - yield b"" - - with pytest.raises(ArtifactIdempotencyMismatchError) as mismatch: - await adapter.store( - untouched(), store_request(key="conformance-store", request_value="changed") + + +async def _seed_reserved_item( + session, + commitment: ArtifactCommitment, +) -> str: + project_id = str(uuid4()) + session_id = str(uuid4()) + item_id = str(uuid4()) + session.add(Project(id=project_id, name="Artifact project", slug=f"artifact-{project_id}")) + await session.flush() + session.add( + ArtifactUploadSession( + id=session_id, + actor_id="actor-1", + project_id=project_id, + permitted_roles=["submission"], + state="open", + maximum_bytes=1024, + current_bytes=0, + reserved_bytes=commitment.byte_count, + maximum_items=1, + current_items=0, + reserved_items=1, + expires_at=datetime.now(UTC) + timedelta(minutes=5), + cas_version=0, ) - assert mismatch.value.code == vectors["store_request_digest_mismatch"]["expected_error"] - assert consumed is vectors["store_request_digest_mismatch"]["consume_replay_stream"] - executed.add("store_request_digest_mismatch") - - with pytest.raises(ArtifactIntegrityError) as replay_mismatch: - await adapter.store(byte_stream(b"x"), request) - assert replay_mismatch.value.code == vectors["store_replay_byte_mismatch"]["expected_error"] - executed.add("store_replay_byte_mismatch") - - recovery_adapter = LocalStorageAdapter(root=tmp_path / "conformance-recovery", buffer_bytes=2) - recovery_request = store_request( - key="conformance-recovery", - expected_sha256="sha256:" + hashlib.sha256(b"data").hexdigest(), - expected_size=4, - maximum_bytes=4, ) - recovery_stored = await recovery_adapter.store(byte_stream(b"data"), recovery_request) - next((tmp_path / "conformance-recovery" / "metadata").iterdir()).unlink() - next((tmp_path / "conformance-recovery" / "receipts").iterdir()).unlink() - recovered = await recovery_adapter.recover_committed_store(recovery_request) - assert recovered.provider_artifact_id == recovery_stored.provider_artifact_id - assert recovered.replayed is True - executed.add("recover_confirmed_store") - - range_adapter = LocalStorageAdapter(root=tmp_path / "ranges", buffer_bytes=2) - ranged = await range_adapter.store( - byte_stream(b"data"), store_request(key="ranges", maximum_bytes=4) - ) - assert b"".join( - [chunk async for chunk in range_adapter.open(ranged.provider_artifact_id, ArtifactByteRange(4))] - ) == b"" - executed.add("open_offset_at_eof") - with pytest.raises(ArtifactMalformedRequestError) as malformed: - b"".join( - [ - chunk - async for chunk in range_adapter.open( - ranged.provider_artifact_id, ArtifactByteRange(5) - ) - ] + await session.flush() + session.add( + ArtifactUploadItem( + id=item_id, + session_id=session_id, + logical_role="submission", + display_name="result.bin", + media_type=commitment.media_type, + reserved_bytes=commitment.byte_count, + expected_sha256=commitment.sha256, + expected_size=commitment.byte_count, + idempotency_key=f"put-{item_id}", + request_digest=canonical_json_hash( + { + "sha256": commitment.sha256, + "byte_count": commitment.byte_count, + "media_type": commitment.media_type, + } + ), + state="reserved", + cas_version=0, ) - assert malformed.value.code == vectors["open_offset_past_eof"]["expected_error"] - executed.add("open_offset_past_eof") - assert b"".join( - [ - chunk - async for chunk in range_adapter.open( - ranged.provider_artifact_id, ArtifactByteRange(1, 2) - ) - ] - ) == b"at" - executed.add("open_exclusive_end") - - release_request_identity = release_identity( - ranged.provider_artifact_id, - "unknown", - key="conformance-release", ) - with pytest.raises(ArtifactRetentionConflictError) as retention_conflict: - await range_adapter.release( - ranged.provider_artifact_id, "unknown", release_request_identity + await session.commit() + return item_id + + +async def _seed_reserved_items_in_one_session( + session, + commitments: tuple[ArtifactCommitment, ...], +) -> tuple[str, ...]: + """Create multiple independently fenced items under one aggregate ledger.""" + project_id = str(uuid4()) + session_id = str(uuid4()) + item_ids = tuple(str(uuid4()) for _ in commitments) + total_bytes = sum(commitment.byte_count for commitment in commitments) + session.add(Project(id=project_id, name="Artifact project", slug=f"artifact-{project_id}")) + await session.flush() + session.add( + ArtifactUploadSession( + id=session_id, + actor_id="actor-1", + project_id=project_id, + permitted_roles=["submission"], + state="open", + maximum_bytes=1024, + current_bytes=0, + reserved_bytes=total_bytes, + maximum_items=len(commitments), + current_items=0, + reserved_items=len(commitments), + expires_at=datetime.now(UTC) + timedelta(minutes=5), + cas_version=0, ) - assert retention_conflict.value.code == vectors["release_requires_exact_reference"][ - "expected_error" - ] - executed.add("release_requires_exact_reference") - - receipt = await adapter.get_operation_receipt( - request.idempotency.service_principal, - ArtifactOperation.STORE, - request.idempotency.key, - ) - assert receipt is not None and receipt.receipt_id == stored.receipt.receipt_id - executed.add("receipt_lookup_full_identity") - - truncated = store_request( - key="truncated", expected_size=4, maximum_bytes=4 ) - with pytest.raises(ArtifactInputMismatchError) as truncated_store: - await adapter.store(byte_stream(b"abc"), truncated) - assert truncated_store.value.code == vectors["truncated_store_stream"]["expected_error"] - executed.add("truncated_store_stream") - - object_path = next((tmp_path / "ranges" / "objects").iterdir()) - object_path.write_bytes(b"abc") - with pytest.raises(ArtifactIntegrityError) as truncated_open: - b"".join([chunk async for chunk in range_adapter.open(ranged.provider_artifact_id)]) - assert truncated_open.value.code == vectors["truncated_open_stream"]["expected_error"] - executed.add("truncated_open_stream") - - assert executed == set(vectors) - - -@pytest.mark.asyncio -async def test_local_adapter_store_open_range_replay_and_restart(tmp_path: Path) -> None: - """Store once, stream ranges, replay exactly, and survive adapter restart.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - request = store_request(maximum_bytes=5) - stored = await adapter.store(byte_stream(b"hello"), request) - - assert stored.byte_count == 5 - assert stored.sha256 == "sha256:" + hashlib.sha256(b"hello").hexdigest() - assert stored.receipt.response_digest == canonical_json_hash( - { - "byte_count": stored.byte_count, - "outcome": "stored", - "provider_artifact_id": stored.provider_artifact_id, - "provider_operation_id": stored.receipt.provider_operation_reference, - "receipt_id": stored.receipt.receipt_id, - "sha256": stored.sha256, - } - ) - read_chunks = [chunk async for chunk in adapter.open(stored.provider_artifact_id)] - assert b"".join(read_chunks) == b"hello" - assert max(map(len, read_chunks)) <= 2 - assert b"".join( - [ - chunk - async for chunk in adapter.open( - stored.provider_artifact_id, ArtifactByteRange(offset=1, length=2) - ) - ] - ) == b"el" - assert b"".join( - [ - chunk - async for chunk in adapter.open( - stored.provider_artifact_id, ArtifactByteRange(offset=5) + await session.flush() + for item_id, commitment in zip(item_ids, commitments, strict=True): + session.add( + ArtifactUploadItem( + id=item_id, + session_id=session_id, + logical_role="submission", + display_name=f"{item_id}.bin", + media_type=commitment.media_type, + reserved_bytes=commitment.byte_count, + expected_sha256=commitment.sha256, + expected_size=commitment.byte_count, + idempotency_key=f"put-{item_id}", + request_digest=canonical_json_hash( + { + "sha256": commitment.sha256, + "byte_count": commitment.byte_count, + "media_type": commitment.media_type, + } + ), + state="reserved", + cas_version=0, ) - ] - ) == b"" - with pytest.raises(ArtifactMalformedRequestError): - b"".join( - [ - chunk - async for chunk in adapter.open( - stored.provider_artifact_id, ArtifactByteRange(offset=6) - ) - ] ) - - replay = await adapter.store(byte_stream(b"hello"), request) - restarted = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - restarted_replay = await restarted.store(byte_stream(b"hello"), request) - assert replay.provider_artifact_id == stored.provider_artifact_id - assert restarted_replay.receipt.provider_operation_reference == ( - stored.receipt.provider_operation_reference - ) - assert len(list((tmp_path / "artifacts" / "objects").iterdir())) == 1 + await session.commit() + return item_ids @pytest.mark.asyncio -async def test_no_commitment_replay_is_fully_consumed_and_mismatch_quarantines( +async def test_put_acknowledgement_stops_at_pending_verification( + artifact_database_env: str, tmp_path: Path, ) -> None: - """Require complete no-commitment replay and quarantine changed bytes.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - request = store_request(maximum_bytes=8) - stored = await adapter.store(byte_stream(b"abcd"), request) - consumed: list[bytes] = [] - - async def changed_replay() -> AsyncIterator[bytes]: - for chunk in (b"ab", b"XY"): - consumed.append(chunk) - yield chunk - - with pytest.raises(ArtifactIntegrityError): - await adapter.store(changed_replay(), request) - assert consumed == [b"ab", b"XY"] - status = await adapter.stat(stored.provider_artifact_id) - assert status.integrity_state == IntegrityState.QUARANTINED - with pytest.raises(ArtifactNotFoundError, match="unavailable"): - b"".join([chunk async for chunk in adapter.open(stored.provider_artifact_id)]) - - -@pytest.mark.asyncio -async def test_request_digest_conflict_does_not_consume_or_duplicate(tmp_path: Path) -> None: - """Reject changed request metadata before reading replay bytes.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - await adapter.store(byte_stream(b"same"), store_request(key="one")) - consumed = False - - async def should_not_run() -> AsyncIterator[bytes]: - nonlocal consumed - consumed = True - yield b"same" - - with pytest.raises(ArtifactIdempotencyMismatchError): - await adapter.store( - should_not_run(), store_request(key="one", request_value="changed") - ) - assert consumed is False - assert len(list((tmp_path / "artifacts" / "objects").iterdir())) == 1 - - -@pytest.mark.asyncio -@pytest.mark.parametrize( - ("payload", "maximum", "fails"), - [(b"", 0, False), (b"abcd", 4, False), (b"abcde", 4, True)], -) -async def test_local_adapter_empty_boundary_and_oversize_cleanup( - tmp_path: Path, payload: bytes, maximum: int, fails: bool -) -> None: - """Apply exact byte limits and remove every oversized temporary file.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - request = store_request(key=f"limit-{len(payload)}", maximum_bytes=maximum) - if fails: - with pytest.raises(ArtifactLimitExceededError): - await adapter.store(byte_stream(payload), request) - assert list((tmp_path / "artifacts" / "objects").iterdir()) == [] - else: - stored = await adapter.store(byte_stream(payload), request) - assert stored.byte_count == len(payload) - assert list((tmp_path / "artifacts" / "tmp").iterdir()) == [] + content = b"artifact-v2" + engine = create_async_engine(artifact_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + settings = _settings(tmp_path) + bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=tmp_path / "store", buffer_bytes=2)) + namespace = artifact_storage_namespace_spec(settings, bootstrap) + store = bootstrap.initialize_after_namespace_claim(local_namespace_claim(bootstrap)) + try: + async with _minted_source(tmp_path / "scratch", content) as source: + async with factory() as session: + item_id = await _seed_reserved_item(session, source.commitment) + result = await ArtifactStorageOrchestrator( + session, store, namespace + ).put_reserved_item(item_id, source) + async with _minted_source(tmp_path / "scratch-replay", content) as replay_source: + async with factory() as session: + replay_item_id = await _seed_reserved_item(session, replay_source.commitment) + replay = await ArtifactStorageOrchestrator( + session, store, namespace + ).put_reserved_item(replay_item_id, replay_source) + assert replay.replayed is True + async with factory() as session: + item = await session.get(ArtifactUploadItem, item_id) + replica = (await session.execute(select(ArtifactReplica))).scalar_one() + receipts = (await session.execute(select(ArtifactOperationReceipt))).scalars().all() + receipt = next(value for value in receipts if value.upload_item_id == item_id) + replay_receipt = next( + value for value in receipts if value.upload_item_id == replay_item_id + ) + namespace_row = (await session.execute(select(ArtifactStorageNamespace))).scalar_one() + assert item is not None + assert item.state == "stored_pending_verification" + assert item.provider_object_ref == result.provider_object_ref + assert replica.verification_state == "pending" + assert replica.availability_state == "unknown" + assert replica.integrity_state == "unknown" + assert replica.namespace_fingerprint == namespace_row.namespace_fingerprint + assert receipt.operation == "put" + assert receipt.outcome == "stored_pending_verification" + assert receipt.provider_object_ref == result.provider_object_ref + assert receipt.replayed is False + assert len(receipts) == 2 + assert replay_receipt.replayed is True + assert await store.head(result.provider_object_ref) == ArtifactObjectHead( + result.provider_object_ref, + exists=True, + byte_count=len(content), + media_type=None, + ) + finally: + store.close() + await engine.dispose() -@pytest.mark.asyncio -async def test_local_adapter_disk_failure_cleans_temp_and_redacts_path( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Convert disk failure to a stable path-free error and remove partial bytes.""" - adapter = LocalStorageAdapter(root=tmp_path / "private-secret", buffer_bytes=2) - def fail_write(descriptor: int, chunk: memoryview) -> None: - raise OSError("disk full at /private-secret") +class _UnavailableStore: + """Count provider calls and fail with one sanitized retryable error.""" - monkeypatch.setattr(adapter, "_write_all", fail_write) - with pytest.raises(ArtifactStoreUnavailableError) as raised: - await adapter.store(byte_stream(b"data"), store_request()) - assert "/private-secret" not in str(raised.value) - assert list((tmp_path / "private-secret" / "tmp").iterdir()) == [] - assert list((tmp_path / "private-secret" / "objects").iterdir()) == [] + identity = ExternalServiceAdapterIdentity("artifact_store", "local") + namespace_identity = ArtifactStoreNamespaceIdentity( + provider_profile="local-v2", + descriptor_items=( + ("private_prefix", "objects/sha256"), + ("private_root_identity", "sha256:" + "0" * 64), + ), + ) + def __init__(self) -> None: + self.put_calls = 0 -@pytest.mark.asyncio -async def test_local_adapter_cancellation_waits_for_io_then_cleans_temp( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Wait for an in-flight thread write before cancellation cleanup.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - started = threading.Event() - release = threading.Event() - original_write = adapter._write_all - - def delayed_write(descriptor: int, chunk: memoryview) -> None: - if bytes(chunk) == b"da": - started.set() - release.wait(timeout=5) - original_write(descriptor, chunk) - - monkeypatch.setattr(adapter, "_write_all", delayed_write) - task = asyncio.create_task( - adapter.store(byte_stream(b"data"), store_request(key="cancelled")) - ) - assert await asyncio.to_thread(started.wait, 5) - task.cancel() - release.set() - with pytest.raises(asyncio.CancelledError): - await task - assert list((tmp_path / "artifacts" / "tmp").iterdir()) == [] - assert list((tmp_path / "artifacts" / "objects").iterdir()) == [] - monkeypatch.setattr(adapter, "_write_all", original_write) - recovered = await adapter.store( - byte_stream(b"data"), store_request(key="cancelled") - ) - assert recovered.replayed is True - assert recovered.byte_count == 4 + async def put(self, _source: CommittedArtifactSource) -> ArtifactPutResult: + self.put_calls += 1 + raise ArtifactStoreUnavailableError() + async def observe_put_result(self, _commitment: ArtifactCommitment) -> ArtifactPutObservation: + raise NotImplementedError -@pytest.mark.asyncio -async def test_cancellation_while_waiting_for_lock_does_not_clean_another_attempt( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Do not clean provider state when cancellation happened before lock ownership.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - request = store_request(key="waiting-cancellation") - scope = adapter._scope_key(request.idempotency) - held_lock = adapter._acquire_lock(scope) - cleanup_calls = 0 - original_cleanup = adapter._cleanup_unpublished - - async def track_cleanup(cleanup_scope: str) -> None: - nonlocal cleanup_calls - cleanup_calls += 1 - await original_cleanup(cleanup_scope) - - monkeypatch.setattr(adapter, "_cleanup_unpublished", track_cleanup) - task = asyncio.create_task(adapter.store(byte_stream(b"data"), request)) - await asyncio.sleep(0.05) - task.cancel() - await asyncio.to_thread(adapter._release_lock, held_lock) - with pytest.raises(asyncio.CancelledError): - await task - assert cleanup_calls == 0 + def open( + self, _provider_object_ref: str, _byte_range: ArtifactByteRange | None = None + ) -> AsyncIterator[bytes]: + raise NotImplementedError + async def head(self, _provider_object_ref: str) -> ArtifactObjectHead: + raise NotImplementedError -@pytest.mark.asyncio -async def test_local_adapter_publish_failure_leaves_no_partial_object( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Remove the completed temporary object if exclusive publication fails.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - def fail_publish(temporary: Path, final: Path) -> None: - raise OSError("publish failed at a private path") +class _TerminalStore(_UnavailableStore): + async def put(self, _source: CommittedArtifactSource) -> ArtifactPutResult: + self.put_calls += 1 + raise ArtifactInputMismatchError() - monkeypatch.setattr(adapter, "_publish_exclusive", fail_publish) - with pytest.raises(ArtifactStoreUnavailableError, match="local artifact operation failed"): - await adapter.store(byte_stream(b"data"), store_request(key="publish-failure")) - assert list((tmp_path / "artifacts" / "tmp").iterdir()) == [] - assert list((tmp_path / "artifacts" / "objects").iterdir()) == [] +class _AcknowledgementUnknownThenReplayStore(_UnavailableStore): + """Lose the first acknowledgement and return an exact replay next.""" -@pytest.mark.asyncio -async def test_local_adapter_fsync_and_publication_fault_recovery( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Recover deterministically from file, directory, metadata, and receipt faults.""" - file_root = tmp_path / "file-fsync" - file_adapter = LocalStorageAdapter(root=file_root, buffer_bytes=2) - original_fsync = os.fsync - - create_root = tmp_path / "create-fault" - create_adapter = LocalStorageAdapter(root=create_root, buffer_bytes=2) - - def fail_create(path: Path) -> int: - del path - raise OSError("temporary create failed") - - monkeypatch.setattr(create_adapter, "_open_exclusive", fail_create) - with pytest.raises(OSError, match="temporary create"): - await create_adapter._write_stream( - byte_stream(b"data"), "art_" + "2" * 32, store_request(key="create-fault") + async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: + self.put_calls += 1 + if self.put_calls == 1: + raise RuntimeError("provider acknowledgement lost") + digest_hex = source.commitment.sha256[7:] + return ArtifactPutResult( + f"sha256/{digest_hex[:2]}/{digest_hex[2:]}", + replayed=True, ) - assert list((create_root / "tmp").iterdir()) == [] - write_root = tmp_path / "bounded-write" - write_adapter = LocalStorageAdapter(root=write_root, buffer_bytes=2) - write_sizes: list[int] = [] - original_write = write_adapter._write_all - def track_write(descriptor: int, chunk: memoryview) -> None: - write_sizes.append(len(chunk)) - original_write(descriptor, chunk) +class _CancelledStore(_UnavailableStore): + async def put(self, _source: CommittedArtifactSource) -> ArtifactPutResult: + self.put_calls += 1 + raise asyncio.CancelledError("provider put cancelled") - monkeypatch.setattr(write_adapter, "_write_all", track_write) - await write_adapter._write_stream( - byte_stream(b"hello"), "art_" + "3" * 32, store_request(key="bounded-write") - ) - assert write_sizes == [2, 2, 1] - def fail_file_fsync(descriptor: int) -> None: - if stat_module.S_ISREG(os.fstat(descriptor).st_mode): - raise OSError("file fsync failed") - original_fsync(descriptor) - - monkeypatch.setattr(os, "fsync", fail_file_fsync) - with pytest.raises(OSError, match="file fsync"): - await file_adapter._write_stream( - byte_stream(b"data"), "art_" + "1" * 32, store_request(key="file-fsync") +class _AcknowledgingStore(_UnavailableStore): + async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: + self.put_calls += 1 + digest_hex = source.commitment.sha256[7:] + return ArtifactPutResult( + f"sha256/{digest_hex[:2]}/{digest_hex[2:]}", + replayed=False, ) - assert list((file_root / "tmp").iterdir()) == [] - assert list((file_root / "objects").iterdir()) == [] - monkeypatch.setattr(os, "fsync", original_fsync) - - directory_root = tmp_path / "directory-fsync" - directory_adapter = LocalStorageAdapter(root=directory_root, buffer_bytes=2) - directory_failed = False - - def fail_published_directory_fsync(descriptor: int) -> None: - nonlocal directory_failed - if ( - not directory_failed - and stat_module.S_ISDIR(os.fstat(descriptor).st_mode) - and any((directory_root / "objects").iterdir()) - ): - directory_failed = True - raise OSError("directory fsync failed") - original_fsync(descriptor) - - monkeypatch.setattr(os, "fsync", fail_published_directory_fsync) - directory_request = store_request(key="directory-fsync") - with pytest.raises(ArtifactStoreUnavailableError): - await directory_adapter.store(byte_stream(b"data"), directory_request) - monkeypatch.setattr(os, "fsync", original_fsync) - directory_recovered = await directory_adapter.store( - byte_stream(b"data"), directory_request - ) - assert directory_recovered.replayed is True - - for fault_directory in ("metadata", "receipts"): - root = tmp_path / f"publish-{fault_directory}" - adapter = LocalStorageAdapter(root=root, buffer_bytes=2) - request = store_request(key=f"publish-{fault_directory}") - original_publish = adapter._write_json_exclusive - failed = False - - async def fail_publish_once(path: Path, value: dict) -> None: - nonlocal failed - if path.parent.name == fault_directory and not failed: - failed = True - raise OSError(f"{fault_directory} publication failed") - await original_publish(path, value) - - monkeypatch.setattr(adapter, "_write_json_exclusive", fail_publish_once) - with pytest.raises(ArtifactStoreUnavailableError): - await adapter.store(byte_stream(b"data"), request) - monkeypatch.setattr(adapter, "_write_json_exclusive", original_publish) - recovered = await adapter.store(byte_stream(b"data"), request) - assert recovered.replayed is True -@pytest.mark.asyncio -async def test_local_adapter_verify_read_and_quarantine_faults_are_typed( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Return stable errors for verify-read and quarantine storage failures.""" - verify_root = tmp_path / "verify-read" - verify_adapter = LocalStorageAdapter(root=verify_root, buffer_bytes=2) - stored = await verify_adapter.store(byte_stream(b"data"), store_request()) - - def fail_open(provider_id: str): - del provider_id - raise OSError("verify read failed") - - monkeypatch.setattr(verify_adapter, "_open_regular_object", fail_open) - with pytest.raises(ArtifactStoreUnavailableError): - await verify_adapter.verify( - stored.provider_artifact_id, - stored.sha256, - stored.byte_count, - verify_identity(stored.provider_artifact_id, stored.sha256, stored.byte_count), +class _NamespaceObservingAcknowledgingStore(_AcknowledgingStore): + """Assert the committed namespace exists before the provider is invoked.""" + + def __init__(self, factory: async_sessionmaker[AsyncSession]) -> None: + super().__init__() + self._factory = factory + self.observed_namespace_fingerprint: str | None = None + + async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: + async with self._factory() as session: + namespace = await session.scalar(select(ArtifactStorageNamespace)) + assert namespace is not None + self.observed_namespace_fingerprint = namespace.namespace_fingerprint + return await super().put(source) + + +class _ConcurrentAcknowledgingStore(_AcknowledgingStore): + """Release two same-object acknowledgements into finalization together.""" + + def __init__(self) -> None: + super().__init__() + self._arrivals = 0 + self._release = asyncio.Event() + + async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: + self.put_calls += 1 + self._arrivals += 1 + arrival = self._arrivals + if self._arrivals == 2: + self._release.set() + await asyncio.wait_for(self._release.wait(), timeout=2) + digest_hex = source.commitment.sha256[7:] + return ArtifactPutResult( + f"sha256/{digest_hex[:2]}/{digest_hex[2:]}", + replayed=arrival > 1, ) - quarantine_root = tmp_path / "quarantine-fault" - quarantine_adapter = LocalStorageAdapter(root=quarantine_root, buffer_bytes=2) - request = store_request( - key="quarantine-fault", - expected_sha256="sha256:" + hashlib.sha256(b"data").hexdigest(), - ) - await quarantine_adapter.store(byte_stream(b"data"), request) - next((quarantine_root / "objects").iterdir()).write_bytes(b"evil") - original_move = quarantine_adapter._move_to_quarantine - def fail_quarantine(source: Path, target: Path) -> None: - del source, target - raise OSError("quarantine failed") +class _ConcurrentDistinctAcknowledgingStore(_ConcurrentAcknowledgingStore): + """Release two distinct initial publications into finalization together.""" - monkeypatch.setattr(quarantine_adapter, "_move_to_quarantine", fail_quarantine) - with pytest.raises(ArtifactStoreUnavailableError): - await quarantine_adapter.store(byte_stream(b"data"), request) - monkeypatch.setattr(quarantine_adapter, "_move_to_quarantine", original_move) - with pytest.raises(ArtifactIntegrityError): - await quarantine_adapter.store(byte_stream(b"data"), request) + async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: + result = await super().put(source) + return ArtifactPutResult(result.provider_object_ref, replayed=False) @pytest.mark.asyncio -async def test_reconciliation_hash_enforces_request_byte_ceiling(tmp_path: Path) -> None: - """Stop provider reconciliation as soon as bytes exceed the persisted ceiling.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - stored = await adapter.store(byte_stream(b"data"), store_request()) - service = ArtifactIngestService(None, adapter, adapter_name="local") # type: ignore[arg-type] - observed_sha256, observed_size = await service._hash_provider_object( - stored.provider_artifact_id, 4 - ) - assert (observed_sha256, observed_size) == (stored.sha256, stored.byte_count) - with pytest.raises(ArtifactIntegrityError, match="exceed"): - await service._hash_provider_object(stored.provider_artifact_id, 3) - - with pytest.raises(ArtifactInputMismatchError, match="exact commitment"): - await adapter.recover_committed_store(store_request(key="recovery-without-commitment")) - exact_request = store_request( - key="recovery-without-intent", - expected_sha256=stored.sha256, - expected_size=stored.byte_count, - ) - with pytest.raises(ArtifactIntegrityError, match="intent"): - await adapter.recover_committed_store(exact_request) - +async def test_concurrent_same_object_finalization_reuses_one_replica( + artifact_database_env: str, + tmp_path: Path, +) -> None: + """Finalize two exact replays with one replica and two receipts.""" + content = b"concurrent-artifact-v2" + engine = create_async_engine(artifact_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + store = _ConcurrentAcknowledgingStore() + namespace = artifact_storage_namespace_spec(_settings(tmp_path), store) + try: + async with _minted_source(tmp_path / "scratch-first", content) as first_source: + async with _minted_source(tmp_path / "scratch-second", content) as second_source: + async with factory() as session: + first_item_id = await _seed_reserved_item(session, first_source.commitment) + second_item_id = await _seed_reserved_item(session, second_source.commitment) + await ArtifactStorageOrchestrator( + session, store, namespace + ).ensure_storage_namespace() + async with session.begin(): + session.add( + ArtifactContent( + id=str(uuid4()), + sha256=first_source.commitment.sha256, + byte_count=first_source.commitment.byte_count, + media_type=first_source.commitment.media_type, + normalized_display_name="preexisting.bin", + ) + ) + + async def finalize(item_id: str, source: CommittedArtifactSource) -> None: + async with factory() as session: + await ArtifactStorageOrchestrator( + session, store, namespace + ).put_reserved_item(item_id, source) + + await asyncio.gather( + finalize(first_item_id, first_source), + finalize(second_item_id, second_source), + ) -@pytest.mark.asyncio -async def test_local_adapter_retain_release_verify_and_permissions(tmp_path: Path) -> None: - """Verify bytes, reference-count retention, and preserve private modes.""" - root = tmp_path / "artifacts" - adapter = LocalStorageAdapter(root=root, buffer_bytes=2) - stored = await adapter.store(byte_stream(b"data"), store_request()) - verify_request_identity = verify_identity( - stored.provider_artifact_id, stored.sha256, stored.byte_count - ) - await adapter.verify( - stored.provider_artifact_id, stored.sha256, stored.byte_count, verify_request_identity - ) - retain_request_identity = retain_identity( - stored.provider_artifact_id, "ref-1", "evaluation" - ) - await adapter.retain( - stored.provider_artifact_id, "ref-1", "evaluation", retain_request_identity - ) - replayed_retain = await adapter.retain( - stored.provider_artifact_id, "ref-1", "evaluation", retain_request_identity - ) - assert replayed_retain.receipt.provider_operation_reference is not None - assert replayed_retain.replayed is True - active = (await adapter.stat(stored.provider_artifact_id)).active_retentions - assert [(item.retention_reference, item.retention_class, item.state) for item in active] == [ - ("ref-1", "evaluation", "active") - ] - release_request_identity = release_identity(stored.provider_artifact_id, "ref-1") - await adapter.release(stored.provider_artifact_id, "ref-1", release_request_identity) - assert (await adapter.stat(stored.provider_artifact_id)).active_retentions == () - assert os.stat(root).st_mode & 0o777 == 0o700 - assert all(path.stat().st_mode & 0o777 == 0o600 for path in (root / "objects").iterdir()) + async with factory() as session: + items = ( + ( + await session.execute( + select(ArtifactUploadItem).where( + ArtifactUploadItem.id.in_((first_item_id, second_item_id)) + ) + ) + ) + .scalars() + .all() + ) + replicas = (await session.execute(select(ArtifactReplica))).scalars().all() + receipts = (await session.execute(select(ArtifactOperationReceipt))).scalars().all() + + assert {item.state for item in items} == {"stored_pending_verification"} + assert len(items) == 2 + assert len(replicas) == 1 + assert replicas[0].verification_state == "pending" + assert replicas[0].availability_state == "unknown" + assert replicas[0].integrity_state == "unknown" + assert len(receipts) == 2 + assert {receipt.replica_id for receipt in receipts} == {replicas[0].id} + finally: + await engine.dispose() @pytest.mark.asyncio -async def test_concurrent_retention_preserves_ownership_and_all_references( +async def test_independent_items_in_one_session_finalize_without_shared_cas_replay( + artifact_database_env: str, tmp_path: Path, ) -> None: - """Serialize retention changes per artifact and enforce the creating principal.""" - root = tmp_path / "artifacts" - first = LocalStorageAdapter(root=root, buffer_bytes=2) - second = LocalStorageAdapter(root=root, buffer_bytes=2) - stored = await first.store(byte_stream(b"data"), store_request()) - - def identity(principal: str, key: str, reference: str) -> IdempotencyIdentity: - return retain_identity( - stored.provider_artifact_id, - reference, - "evaluation" if reference == "ref-a" else "review", - key=key, - principal=principal, - ) - - await asyncio.gather( - first.retain( - stored.provider_artifact_id, - "ref-a", - "evaluation", - identity("workstream.artifact.a", "retain-a", "ref-a"), - ), - second.retain( - stored.provider_artifact_id, - "ref-b", - "review", - identity("workstream.artifact.b", "retain-b", "ref-b"), - ), - ) - assert [item.retention_reference for item in (await first.stat(stored.provider_artifact_id)).active_retentions] == [ - "ref-a", - "ref-b", - ] - wrong_owner = release_identity( - stored.provider_artifact_id, - "ref-a", - key="release-a", - principal="workstream.artifact.b", - ) - with pytest.raises(ArtifactRetentionConflictError, match="owner"): - await second.release(stored.provider_artifact_id, "ref-a", wrong_owner) - + """Fence provider acknowledgements per item while accounting one session.""" + engine = create_async_engine(artifact_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + store = _ConcurrentDistinctAcknowledgingStore() + namespace = artifact_storage_namespace_spec(_settings(tmp_path), store) + try: + async with _minted_source(tmp_path / "scratch-first", b"first-item") as first: + async with _minted_source(tmp_path / "scratch-second", b"second-item") as second: + async with factory() as session: + item_ids = await _seed_reserved_items_in_one_session( + session, + (first.commitment, second.commitment), + ) + await ArtifactStorageOrchestrator( + session, store, namespace + ).ensure_storage_namespace() + + async def finalize(item_id: str, source: CommittedArtifactSource) -> None: + async with factory() as candidate_session: + await ArtifactStorageOrchestrator( + candidate_session, store, namespace + ).put_reserved_item(item_id, source) + + await asyncio.gather( + finalize(item_ids[0], first), + finalize(item_ids[1], second), + ) -@pytest.mark.asyncio -async def test_verify_and_retain_serialize_without_losing_metadata(tmp_path: Path) -> None: - """Preserve both verification and retention facts under concurrent mutation.""" - root = tmp_path / "artifacts" - first = LocalStorageAdapter(root=root, buffer_bytes=2) - second = LocalStorageAdapter(root=root, buffer_bytes=2) - stored = await first.store(byte_stream(b"data"), store_request()) - await asyncio.gather( - first.verify( - stored.provider_artifact_id, - stored.sha256, - stored.byte_count, - verify_identity(stored.provider_artifact_id, stored.sha256, stored.byte_count), - ), - second.retain( - stored.provider_artifact_id, - "concurrent-review", - "review", - retain_identity(stored.provider_artifact_id, "concurrent-review", "review"), - ), - ) - status = await first.stat(stored.provider_artifact_id) - assert status.verification_state.value == "verified" - assert [retention.retention_reference for retention in status.active_retentions] == [ - "concurrent-review" - ] + async with factory() as session: + items = ( + ( + await session.execute( + select(ArtifactUploadItem).where(ArtifactUploadItem.id.in_(item_ids)) + ) + ) + .scalars() + .all() + ) + upload_session = await session.get(ArtifactUploadSession, items[0].session_id) + receipts = (await session.execute(select(ArtifactOperationReceipt))).scalars().all() + + assert {item.state for item in items} == {"stored_pending_verification"} + assert upload_session is not None + assert upload_session.reserved_bytes == 0 + assert upload_session.reserved_items == 0 + assert upload_session.current_bytes == len(b"first-item") + len(b"second-item") + assert upload_session.current_items == 2 + assert len(receipts) == 2 + finally: + await engine.dispose() @pytest.mark.asyncio -async def test_retention_retry_recovers_after_receipt_publication_failure( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch +async def test_database_rejects_partial_upload_result_metadata( + artifact_database_env: str, + tmp_path: Path, ) -> None: - """Recover the same retention effect after metadata commits before its receipt.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - stored = await adapter.store(byte_stream(b"data"), store_request()) - identity = retain_identity( - stored.provider_artifact_id, - "ref-crash", - "evaluation", - key="retain-crash", - ) - original = adapter._write_json_exclusive - failed = False - - async def fail_receipt_once(path: Path, value: dict) -> None: - nonlocal failed - if path.parent.name == "receipts" and not failed: - failed = True - raise OSError("simulated receipt publication failure") - await original(path, value) - - monkeypatch.setattr(adapter, "_write_json_exclusive", fail_receipt_once) - with pytest.raises(ArtifactStoreUnavailableError): - await adapter.retain( - stored.provider_artifact_id, "ref-crash", "evaluation", identity - ) - recovered = await adapter.retain( - stored.provider_artifact_id, "ref-crash", "evaluation", identity - ) - assert recovered.receipt.retention_reference == "ref-crash" - assert len((await adapter.stat(stored.provider_artifact_id)).active_retentions) == 1 - + """Require both stored-result references or neither in every item state.""" + engine = create_async_engine(artifact_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with _minted_source(tmp_path / "scratch", b"constraint") as source: + async with factory() as session: + item_id = await _seed_reserved_item(session, source.commitment) + content_id = str(uuid4()) + session.add( + ArtifactContent( + id=content_id, + sha256=source.commitment.sha256, + byte_count=source.commitment.byte_count, + media_type=source.commitment.media_type, + normalized_display_name="constraint.bin", + ) + ) + await session.commit() + + with pytest.raises(IntegrityError): + async with engine.begin() as connection: + await connection.execute( + text( + "update artifact_upload_items set content_id = :content_id " + "where id = :item_id" + ), + {"content_id": content_id, "item_id": item_id}, + ) + with pytest.raises(IntegrityError): + async with engine.begin() as connection: + await connection.execute( + text( + "update artifact_upload_items " + "set provider_object_ref = 'sha256/00/object' where id = :item_id" + ), + {"item_id": item_id}, + ) -@pytest.mark.asyncio -async def test_expected_commitment_recovers_object_only_provider_state(tmp_path: Path) -> None: - """Rebuild metadata and receipt only after hashing committed provider bytes.""" - root = tmp_path / "artifacts" - adapter = LocalStorageAdapter(root=root, buffer_bytes=2) - expected = "sha256:" + hashlib.sha256(b"data").hexdigest() - request = store_request(expected_sha256=expected, expected_size=4) - stored = await adapter.store(byte_stream(b"data"), request) - next((root / "metadata").iterdir()).unlink() - next((root / "receipts").iterdir()).unlink() - - restarted = LocalStorageAdapter(root=root, buffer_bytes=2) - recovered = await restarted.recover_committed_store(request) - assert recovered.provider_artifact_id == stored.provider_artifact_id - assert recovered.sha256 == expected - assert recovered.receipt.outcome.value == "stored" - assert recovered.replayed is True + async with factory() as session: + item = await session.get(ArtifactUploadItem, item_id) + assert item is not None + assert item.content_id is None + assert item.provider_object_ref is None + finally: + await engine.dispose() -@pytest.mark.asyncio -async def test_altered_object_and_receipt_are_quarantined(tmp_path: Path) -> None: - """Deny promotion and access when persisted provider facts are altered.""" - root = tmp_path / "artifacts" - adapter = LocalStorageAdapter(root=root, buffer_bytes=2) - request = store_request(expected_sha256="sha256:" + hashlib.sha256(b"data").hexdigest()) - await adapter.store(byte_stream(b"data"), request) - object_path = next((root / "objects").iterdir()) - object_path.write_bytes(b"evil") - with pytest.raises(ArtifactIntegrityError): - await adapter.store(byte_stream(b"data"), request) - assert (root / "quarantine" / object_path.name).exists() - - second_root = tmp_path / "receipt-artifacts" - second = LocalStorageAdapter(root=second_root, buffer_bytes=2) - second_request = store_request(key="receipt", expected_sha256=request.expected_sha256) - second_stored = await second.store(byte_stream(b"data"), second_request) - receipt_path = next((second_root / "receipts").iterdir()) - receipt_payload = json.loads(receipt_path.read_text()) - receipt_payload["response_digest"] = "sha256:" + "f" * 64 - receipt_path.write_text(json.dumps(receipt_payload)) - with pytest.raises(ArtifactIntegrityError): - await second.store(byte_stream(b"data"), second_request) - assert (await second.stat(second_stored.provider_artifact_id)).integrity_state == ( - IntegrityState.QUARANTINED - ) +def test_reservation_accounting_rejects_drift_without_clamping() -> None: + """Never consume capacity belonging to a different upload item.""" + upload_session = ArtifactUploadSession(reserved_bytes=3, reserved_items=1) + item = ArtifactUploadItem(reserved_bytes=4) - third_root = tmp_path / "receipt-identity-artifacts" - third = LocalStorageAdapter(root=third_root, buffer_bytes=2) - third_request = store_request(key="receipt-identity", expected_sha256=request.expected_sha256) - third_stored = await third.store(byte_stream(b"data"), third_request) - third_receipt_path = next((third_root / "receipts").iterdir()) - third_receipt = json.loads(third_receipt_path.read_text()) - third_receipt["receipt_id"] = "receipt_" + "a" * 32 - third_receipt_path.write_text(json.dumps(third_receipt)) - with pytest.raises(ArtifactIntegrityError, match="receipt is invalid"): - await third.store(byte_stream(b"data"), third_request) - assert (await third.stat(third_stored.provider_artifact_id)).integrity_state == ( - IntegrityState.QUARANTINED - ) + with pytest.raises(ArtifactIntegrityError, match="reservation accounting"): + ArtifactStorageOrchestrator._apply_committed_accounting(upload_session, item, 4) - fourth_root = tmp_path / "receipt-principal-artifacts" - fourth = LocalStorageAdapter(root=fourth_root, buffer_bytes=2) - fourth_request = store_request(key="receipt-principal") - fourth_stored = await fourth.store(byte_stream(b"data"), fourth_request) - fourth_receipt_path = next((fourth_root / "receipts").iterdir()) - fourth_receipt = json.loads(fourth_receipt_path.read_text()) - fourth_receipt["service_principal"] = "INVALID PRINCIPAL" - fourth_receipt_path.write_text(json.dumps(fourth_receipt)) - with pytest.raises(ArtifactIntegrityError, match="receipt is invalid"): - await fourth.store(byte_stream(b"data"), fourth_request) - assert (await fourth.stat(fourth_stored.provider_artifact_id)).integrity_state == ( - IntegrityState.QUARANTINED - ) + assert upload_session.reserved_bytes == 3 + assert upload_session.reserved_items == 1 -@pytest.mark.asyncio -async def test_verify_mismatch_and_invalid_retention_reference_fail_closed(tmp_path: Path) -> None: - """Quarantine failed full verification and reject unknown release references.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - stored = await adapter.store(byte_stream(b"data"), store_request()) - missing_release_identity = release_identity( - stored.provider_artifact_id, - "missing", - key="missing-release", - ) - with pytest.raises(ArtifactRetentionConflictError, match="not active"): - await adapter.release(stored.provider_artifact_id, "missing", missing_release_identity) - - bad_verify_identity = verify_identity( - stored.provider_artifact_id, - "sha256:" + "0" * 64, - 4, - key="bad-verify", - ) - with pytest.raises(ArtifactIntegrityError): - await adapter.verify( - stored.provider_artifact_id, "sha256:" + "0" * 64, 4, bad_verify_identity - ) - assert (await adapter.stat(stored.provider_artifact_id)).integrity_state == ( - IntegrityState.QUARANTINED - ) +class _FinalizationCancelledOrchestrator(ArtifactStorageOrchestrator): + async def _finalize_put( + self, + item_id: str, + commitment: ArtifactCommitment, + result: ArtifactPutResult, + fence: ProviderAttemptFence, + *, + correlation_id: str, + ) -> None: + del item_id, commitment, result, fence, correlation_id + raise asyncio.CancelledError("put finalization cancelled") -@pytest.mark.asyncio -async def test_receipt_lookup_and_request_validation(tmp_path: Path) -> None: - """Scope receipt lookup fully and reject malformed requests before I/O.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - request = store_request() - stored = await adapter.store(byte_stream(b"data"), request) - receipt = await adapter.get_operation_receipt( - request.idempotency.service_principal, - ArtifactOperation.STORE, - request.idempotency.key, - ) - assert receipt is not None - assert receipt.provider_operation_reference == stored.receipt.provider_operation_reference - assert ( - await adapter.get_operation_receipt( - request.idempotency.service_principal, ArtifactOperation.STORE, "unknown" - ) - is None - ) - with pytest.raises(ArtifactMalformedRequestError): - await adapter.store(byte_stream(b"x"), store_request(maximum_bytes=-1)) - with pytest.raises(ArtifactMalformedRequestError): - await adapter.store( - byte_stream(b"x"), - store_request(expected_sha256="SHA256:" + "0" * 64), - ) - malformed = store_request(key="bad-digest") - malformed = replace( - malformed, - idempotency=replace(malformed.idempotency, request_digest="sha256:" + "f" * 64), - ) - with pytest.raises(ArtifactIdempotencyMismatchError, match="not canonical"): - await adapter.store(byte_stream(b"x"), malformed) - with pytest.raises(ArtifactMalformedRequestError, match="media type"): - await adapter.store( - byte_stream(b"x"), replace(store_request(key="bad-media"), media_type="") - ) - with pytest.raises(ArtifactMalformedRequestError, match="metadata"): - await adapter.store( - byte_stream(b"x"), - replace(store_request(key="bad-metadata"), metadata={"bad name": "value"}), - ) - with pytest.raises(ArtifactNotFoundError): - await adapter.stat("../private") - missing_provider_id = "art_" + "0" * 32 - missing_verify = verify_identity( - missing_provider_id, - "sha256:" + "0" * 64, - 0, - key="missing-verify", - ) - with pytest.raises(ArtifactNotFoundError) as missing_error: - await adapter.verify(missing_provider_id, "sha256:" + "0" * 64, 0, missing_verify) - assert str(tmp_path) not in str(missing_error.value) +async def _assert_replay_required_without_durable_facts( + factory: async_sessionmaker, + item_id: str, +) -> None: + async with factory() as session: + item = await session.get(ArtifactUploadItem, item_id) + assert item is not None + upload_session = await session.get(ArtifactUploadSession, item.session_id) + assert upload_session is not None + assert item.state == "replay_required" + assert item.error_code == "artifact_put_acknowledgement_unknown" + assert upload_session.reserved_bytes == item.reserved_bytes + assert upload_session.reserved_items == 1 + assert await session.scalar(select(ArtifactContent)) is None + assert await session.scalar(select(ArtifactReplica)) is None + assert await session.scalar(select(ArtifactOperationReceipt)) is None @pytest.mark.asyncio -async def test_recovery_missing_and_changed_provider_state_fails_closed(tmp_path: Path) -> None: - """Reject receipt-only and changed object-only recovery states.""" - expected = "sha256:" + hashlib.sha256(b"data").hexdigest() - - missing_root = tmp_path / "missing" - missing = LocalStorageAdapter(root=missing_root, buffer_bytes=2) - missing_request = store_request(key="missing", expected_sha256=expected, expected_size=4) - await missing.store(byte_stream(b"data"), missing_request) - next((missing_root / "objects").iterdir()).unlink() - next((missing_root / "metadata").iterdir()).unlink() - with pytest.raises(ArtifactIntegrityError, match="receipt has no committed object"): - await missing.store(byte_stream(b"data"), missing_request) - - status_root = tmp_path / "missing-status" - status_adapter = LocalStorageAdapter(root=status_root, buffer_bytes=2) - status_stored = await status_adapter.store( - byte_stream(b"data"), store_request(key="missing-status") - ) - next((status_root / "objects").iterdir()).unlink() - status = await status_adapter.stat(status_stored.provider_artifact_id) - assert status.availability_state.value == "missing" - assert status.integrity_state.value == "unknown" - - replay_root = tmp_path / "replay-metadata" - replay = LocalStorageAdapter(root=replay_root, buffer_bytes=2) - replay_request = store_request(key="no-metadata") - await replay.store(byte_stream(b"data"), replay_request) - next((replay_root / "metadata").iterdir()).unlink() - recovered = await replay.store(byte_stream(b"data"), replay_request) - assert recovered.replayed is True - assert recovered.sha256 == expected - - changed_root = tmp_path / "changed" - changed = LocalStorageAdapter(root=changed_root, buffer_bytes=2) - changed_request = store_request(key="changed", expected_sha256=expected, expected_size=4) - stored = await changed.store(byte_stream(b"data"), changed_request) - next((changed_root / "metadata").iterdir()).unlink() - next((changed_root / "receipts").iterdir()).unlink() - next((changed_root / "objects").iterdir()).write_bytes(b"evil") - with pytest.raises(ArtifactIntegrityError, match="commitment"): - await changed.store(byte_stream(b"data"), changed_request) - assert (changed_root / "quarantine" / f"{stored.provider_artifact_id}.blob").exists() - - size_root = tmp_path / "changed-size" - changed_size = LocalStorageAdapter(root=size_root, buffer_bytes=2) - size_request = store_request(key="size", expected_sha256=expected, expected_size=4) - size_stored = await changed_size.store(byte_stream(b"data"), size_request) - next((size_root / "metadata").iterdir()).unlink() - next((size_root / "receipts").iterdir()).unlink() - intent_path = next((size_root / "intents").iterdir()) - intent = json.loads(intent_path.read_text()) - intent["expected_size"] = 5 - intent_path.write_text(json.dumps(intent)) - with pytest.raises(ArtifactIntegrityError, match="size does not match"): - await changed_size.store(byte_stream(b"data"), size_request) - assert (size_root / "quarantine" / f"{size_stored.provider_artifact_id}.blob").exists() - +async def test_retryable_put_marks_existing_item_replay_required( + artifact_database_env: str, + tmp_path: Path, +) -> None: + engine = create_async_engine(artifact_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + store = _UnavailableStore() + try: + async with _minted_source(tmp_path / "scratch", b"retry") as source: + async with factory() as session: + item_id = await _seed_reserved_item(session, source.commitment) + orchestrator = ArtifactStorageOrchestrator( + session, + store, + artifact_storage_namespace_spec(_settings(tmp_path), store), + ) + with pytest.raises(ArtifactStoreUnavailableError): + await orchestrator.put_reserved_item(item_id, source) -@pytest.mark.asyncio -async def test_retention_class_conflict_and_verify_replay(tmp_path: Path) -> None: - """Reject changed retention ownership and replay one verify receipt.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - stored = await adapter.store(byte_stream(b"data"), store_request()) - first = retain_identity( - stored.provider_artifact_id, - "same-ref", - "one", - key="retain-a", - ) - await adapter.retain(stored.provider_artifact_id, "same-ref", "one", first) - conflicting = retain_identity( - stored.provider_artifact_id, - "same-ref", - "two", - key="retain-b", - ) - with pytest.raises(ArtifactIdempotencyMismatchError, match="class changed"): - await adapter.retain(stored.provider_artifact_id, "same-ref", "two", conflicting) - - verify_request_identity = verify_identity( - stored.provider_artifact_id, - stored.sha256, - stored.byte_count, - key="verify-replay", - ) - first_result = await adapter.verify( - stored.provider_artifact_id, stored.sha256, stored.byte_count, verify_request_identity - ) - replay = await adapter.verify( - stored.provider_artifact_id, stored.sha256, stored.byte_count, verify_request_identity - ) - assert replay.receipt.provider_operation_reference == ( - first_result.receipt.provider_operation_reference - ) - assert replay.replayed is True + async with factory() as session: + item = await session.get(ArtifactUploadItem, item_id) + assert item is not None + assert item.state == "replay_required" + assert item.error_code == "artifact_put_acknowledgement_unknown" + assert await session.scalar(select(ArtifactReplica)) is None + assert await session.scalar(select(ArtifactOperationReceipt)) is None + assert store.put_calls == 1 + finally: + await engine.dispose() @pytest.mark.asyncio -async def test_truncated_object_and_malformed_metadata_are_integrity_failures( +async def test_same_item_replay_finalizes_once_without_double_accounting( + artifact_database_env: str, tmp_path: Path, ) -> None: - """Detect short object streams and non-object provider metadata.""" - root = tmp_path / "artifacts" - adapter = LocalStorageAdapter(root=root, buffer_bytes=2) - stored = await adapter.store(byte_stream(b"data"), store_request()) - next((root / "objects").iterdir()).write_bytes(b"d") - with pytest.raises(ArtifactIntegrityError, match="ended"): - b"".join([chunk async for chunk in adapter.open(stored.provider_artifact_id)]) - - metadata_path = next((root / "metadata").iterdir()) - metadata_path.write_text("[]") - with pytest.raises(ArtifactIntegrityError, match="metadata"): - await adapter.stat(stored.provider_artifact_id) - malformed_quarantine = await adapter.stat(stored.provider_artifact_id) - assert malformed_quarantine.integrity_state == IntegrityState.QUARANTINED - assert malformed_quarantine.availability_state.value == "unavailable" - - state_root = tmp_path / "invalid-state" - state_adapter = LocalStorageAdapter(root=state_root, buffer_bytes=2) - state_stored = await state_adapter.store( - byte_stream(b"data"), store_request(key="invalid-state") - ) - state_metadata_path = next((state_root / "metadata").iterdir()) - state_metadata = json.loads(state_metadata_path.read_text()) - state_metadata["verification_state"] = "corrupt" - state_metadata_path.write_text(json.dumps(state_metadata)) - with pytest.raises(ArtifactIntegrityError, match="metadata"): - await state_adapter.stat(state_stored.provider_artifact_id) - quarantined = await state_adapter.stat(state_stored.provider_artifact_id) - assert quarantined.integrity_state == IntegrityState.QUARANTINED + engine = create_async_engine(artifact_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + store = _AcknowledgementUnknownThenReplayStore() + content = b"same-item-replay" + try: + async with _minted_source(tmp_path / "scratch-first", content) as source: + async with factory() as session: + item_id = await _seed_reserved_item(session, source.commitment) + orchestrator = ArtifactStorageOrchestrator( + session, + store, + artifact_storage_namespace_spec(_settings(tmp_path), store), + ) + with pytest.raises(RuntimeError, match="acknowledgement lost"): + await orchestrator.put_reserved_item(item_id, source) + + async with _minted_source(tmp_path / "scratch-replay", content) as replay: + async with factory() as session: + result = await ArtifactStorageOrchestrator( + session, + store, + artifact_storage_namespace_spec(_settings(tmp_path), store), + ).put_reserved_item(item_id, replay) + + assert result.replayed is True + async with factory() as session: + item = await session.get(ArtifactUploadItem, item_id) + assert item is not None + upload_session = await session.get(ArtifactUploadSession, item.session_id) + assert upload_session is not None + receipts = (await session.execute(select(ArtifactOperationReceipt))).scalars().all() + replicas = (await session.execute(select(ArtifactReplica))).scalars().all() + contents = (await session.execute(select(ArtifactContent))).scalars().all() + + assert item.state == "stored_pending_verification" + assert item.error_code is None + assert upload_session.reserved_bytes == 0 + assert upload_session.reserved_items == 0 + assert upload_session.current_bytes == len(content) + assert upload_session.current_items == 1 + assert len(contents) == 1 + assert len(replicas) == 1 + assert len(receipts) == 1 + assert receipts[0].upload_item_id == item_id + assert receipts[0].replayed is True + assert store.put_calls == 2 + finally: + await engine.dispose() @pytest.mark.asyncio -async def test_stream_and_operation_validation_rejects_invalid_inputs(tmp_path: Path) -> None: - """Reject invalid stream values, sizes, and operation identities.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - - async def invalid_stream() -> AsyncIterator[bytes]: - yield "not-bytes" # type: ignore[misc] - - with pytest.raises(ArtifactInputMismatchError, match="yield bytes"): - await adapter.store(invalid_stream(), store_request(key="invalid-stream")) - with pytest.raises(ArtifactMalformedRequestError, match="expected artifact size"): - await adapter.store( - byte_stream(b"x"), store_request(key="negative-size", expected_size=-1) - ) - control_media_request = replace( - store_request(key="control-media"), media_type="text/plain\ninvalid" - ) - with pytest.raises(ArtifactMalformedRequestError, match="media type"): - await adapter.store(byte_stream(b"x"), control_media_request) - wrong_operation = IdempotencyIdentity( - "workstream.artifact.ingest", - ArtifactOperation.RETAIN, - "wrong-operation", - canonical_json_hash({"wrong": True}), - ) - invalid_request = StoreArtifactRequest( - None, None, 4, "application/octet-stream", {}, wrong_operation - ) - with pytest.raises(ArtifactMalformedRequestError, match="operation identity"): - await adapter.store(byte_stream(b"x"), invalid_request) - with pytest.raises(ValueError, match="nonnegative"): - ArtifactByteRange(offset=-1) - - stored = await adapter.store(byte_stream(b"x"), store_request(key="validation-object")) - negative_verify_identity = verify_identity( - stored.provider_artifact_id, - stored.sha256, - -1, - key="negative-verify", - ) - with pytest.raises(ArtifactMalformedRequestError, match="size must be nonnegative"): - await adapter.verify( - stored.provider_artifact_id, stored.sha256, -1, negative_verify_identity - ) - empty_retain_identity = retain_identity( - stored.provider_artifact_id, - "empty-placeholder", - "evaluation", - key="empty-reference", - ) - with pytest.raises(ArtifactMalformedRequestError, match="reference is invalid"): - await adapter.retain( - stored.provider_artifact_id, "", "evaluation", empty_retain_identity - ) - long_principal_identity = retain_identity( - stored.provider_artifact_id, - "long-principal", - "evaluation", - principal="a" * 101, - ) - with pytest.raises(ArtifactMalformedRequestError, match="service principal"): - await adapter.retain( - stored.provider_artifact_id, - "long-principal", - "evaluation", - long_principal_identity, - ) - assert (await adapter.stat(stored.provider_artifact_id)).active_retentions == () - - -def test_local_adapter_rejects_invalid_buffer(tmp_path: Path) -> None: - """Keep configured local I/O inside the one MiB ceiling.""" - with pytest.raises(ValueError, match="1 MiB"): - LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=0) - - -def test_local_adapter_rejects_symlink_root(tmp_path: Path) -> None: - """Reject a configured root that redirects through a symlink.""" - real = tmp_path / "real" - real.mkdir() - linked = tmp_path / "linked" - linked.symlink_to(real, target_is_directory=True) - with pytest.raises(ValueError, match="symlink"): - LocalStorageAdapter(root=linked) - +async def test_provider_cancellation_persists_replay_required_without_facts( + artifact_database_env: str, + tmp_path: Path, +) -> None: + engine = create_async_engine(artifact_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + store = _CancelledStore() + try: + async with _minted_source(tmp_path / "scratch", b"provider cancellation") as source: + async with factory() as session: + item_id = await _seed_reserved_item(session, source.commitment) + orchestrator = ArtifactStorageOrchestrator( + session, + store, + artifact_storage_namespace_spec(_settings(tmp_path), store), + ) + with pytest.raises(asyncio.CancelledError, match="provider put cancelled"): + await orchestrator.put_reserved_item(item_id, source) -@pytest.mark.asyncio -async def test_local_adapter_rejects_runtime_subdirectory_replacement(tmp_path: Path) -> None: - """Keep operations under the pinned root after a subdirectory replacement.""" - root = tmp_path / "artifacts" - adapter = LocalStorageAdapter(root=root, buffer_bytes=2) - stored = await adapter.store(byte_stream(b"data"), store_request()) - metadata = root / "metadata" - original = root / "metadata-original" - metadata.rename(original) - outside = tmp_path / "outside" - outside.mkdir() - metadata.symlink_to(outside, target_is_directory=True) - with pytest.raises(ArtifactIntegrityError): - await adapter.stat(stored.provider_artifact_id) - assert list(outside.iterdir()) == [] + await _assert_replay_required_without_durable_facts(factory, item_id) + assert store.put_calls == 1 + finally: + await engine.dispose() @pytest.mark.asyncio -async def test_local_adapter_startup_cleans_locked_abandoned_object_temp(tmp_path: Path) -> None: - """Remove only stale object temporary files associated with durable intent.""" - root = tmp_path / "artifacts" - adapter = LocalStorageAdapter(root=root, buffer_bytes=2) - stored = await adapter.store(byte_stream(b"data"), store_request()) - temporary = root / "tmp" / f"{stored.provider_artifact_id}_{'a' * 32}.part" - temporary.write_bytes(b"partial") - old = datetime.now(UTC).timestamp() - 7200 - os.utime(temporary, (old, old)) - adapter.close() - restarted = LocalStorageAdapter(root=root, buffer_bytes=2) - assert not temporary.exists() - restarted.close() - +async def test_finalization_cancellation_persists_replay_required_without_facts( + artifact_database_env: str, + tmp_path: Path, +) -> None: + engine = create_async_engine(artifact_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + store = _AcknowledgingStore() + try: + async with _minted_source(tmp_path / "scratch", b"finalization cancellation") as source: + async with factory() as session: + item_id = await _seed_reserved_item(session, source.commitment) + orchestrator = _FinalizationCancelledOrchestrator( + session, + store, + artifact_storage_namespace_spec(_settings(tmp_path), store), + ) + with pytest.raises(asyncio.CancelledError, match="finalization cancelled"): + await orchestrator.put_reserved_item(item_id, source) -@pytest.fixture -def artifact_database_env( - isolated_database_env: str, - migration_lock, -) -> Iterator[str]: - """Reset PostgreSQL around artifact coordinator integration tests.""" - project_root = Path(__file__).resolve().parents[1] - config = Config(str(project_root / "alembic.ini")) - config.set_main_option("script_location", str(project_root / "alembic")) - with migration_lock(): - command.downgrade(config, "base") - command.upgrade(config, "head") - yield isolated_database_env - asyncio.run(_truncate_artifact_tables(isolated_database_env)) - command.downgrade(config, "base") + await _assert_replay_required_without_durable_facts(factory, item_id) + assert store.put_calls == 1 + finally: + await engine.dispose() @pytest.mark.asyncio -async def test_ingest_service_promotes_only_verified_provider_result( - artifact_database_env: str, tmp_path: Path +async def test_terminal_put_failure_releases_reservation_and_fails_item( + artifact_database_env: str, + tmp_path: Path, ) -> None: - """Prove transaction B creates one content/replica/receipt and no binding.""" engine = create_async_engine(artifact_database_env) factory = async_sessionmaker(engine, expire_on_commit=False) + store = _TerminalStore() try: - request = store_request( - expected_sha256="sha256:" + hashlib.sha256(b"data").hexdigest(), - expected_size=4, - ) - ids = await _seed_reserved_item(factory, request) - async with factory() as session: - service = ArtifactIngestService( - session, - LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2), - adapter_name="local", - ) - await service.ingest_reserved_item(ids["item"], byte_stream(b"data"), request) - async with factory() as session: - assert await session.scalar(select(func.count()).select_from(ArtifactContent)) == 1 - assert await session.scalar(select(func.count()).select_from(ArtifactReplica)) == 1 - assert ( - await session.scalar(select(func.count()).select_from(ArtifactOperationReceipt)) - == 1 - ) - assert await session.scalar(select(func.count()).select_from(ArtifactBinding)) == 0 - item = await session.get(ArtifactUploadItem, ids["item"]) - assert item is not None and item.state == "ready" - replica = (await session.scalars(select(ArtifactReplica))).one() - replica_adapter = replica.adapter - replica_provider_id = replica.provider_artifact_id - replica_content_id = replica.content_id - binding_id = str(uuid4()) - secondary_replica_id = str(uuid4()) - await session.rollback() - async with session.begin(): - session.add( - ArtifactBinding( - id=binding_id, - content_id=replica_content_id, - project_id=ids["project"], - resource_type="submission", - resource_id="submission-isolation", - logical_role="submission_packet", - scope_version=1, - actor_id="actor-1", - attribution_type="submitted_by", - ) - ) - session.add( - ArtifactReplica( - id=secondary_replica_id, - content_id=replica_content_id, - adapter="secondary", - provider_artifact_id="secondary-provider-copy", - verification_state="verified", - retention_state="retained", - availability_state="available", - integrity_state="valid", - ) - ) - binding = await session.get(ArtifactBinding, binding_id) - secondary_replica = await session.get(ArtifactReplica, secondary_replica_id) - assert binding is not None - assert secondary_replica is not None - binding_before = { - column.name: getattr(binding, column.name) - for column in ArtifactBinding.__table__.columns - } - secondary_replica_before = { - column.name: getattr(secondary_replica, column.name) - for column in ArtifactReplica.__table__.columns - } - await session.rollback() - service = ArtifactIngestService( - session, - LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2), - adapter_name="local", - ) - retain_request_identity = retain_identity( - replica_provider_id, - "review-hold", - "review", - key="retain-coordinator", - principal="workstream.artifact.lifecycle", - ) - await service.retain_replica( - replica_adapter, - replica_provider_id, - "review-hold", - "review", - retain_request_identity, - ) - second_retain_identity = retain_identity( - replica_provider_id, - "payment-hold", - "payment", - key="retain-payment", - principal="workstream.artifact.lifecycle", - ) - await service.retain_replica( - replica_adapter, - replica_provider_id, - "payment-hold", - "payment", - second_retain_identity, - ) - release_request_identity = release_identity( - replica_provider_id, - "review-hold", - key="release-coordinator", - principal="workstream.artifact.lifecycle", - ) - await service.release_replica( - replica_adapter, - replica_provider_id, - "review-hold", - release_request_identity, - ) - async with session.begin(): - current_replica = await session.get(ArtifactReplica, replica.id) - assert current_replica is not None - current_replica.verification_state = "failed" - retained_after_one_release = await service.reconcile_replica( - replica_adapter, replica_provider_id - ) - assert retained_after_one_release.retention_state.value == "retained" - await service.reconcile_replica(replica_adapter, replica_provider_id) - async with session.begin(): - current_replica = await session.get(ArtifactReplica, replica.id) - assert current_replica is not None - current_replica.integrity_state = "unknown" - await service.quarantine_existing_replica( - replica_adapter, replica_provider_id, reason="integrity mismatch" - ) - await service.quarantine_existing_replica( - replica_adapter, replica_provider_id, reason="integrity mismatch" - ) - await service.reconcile_replica(replica_adapter, replica_provider_id) + async with _minted_source(tmp_path / "scratch", b"terminal") as source: + async with factory() as session: + item_id = await _seed_reserved_item(session, source.commitment) + with pytest.raises(ArtifactInputMismatchError): + await ArtifactStorageOrchestrator( + session, + store, + artifact_storage_namespace_spec(_settings(tmp_path), store), + ).put_reserved_item(item_id, source) + async with factory() as session: - assert await session.scalar(select(func.count()).select_from(ArtifactContent)) == 1 - assert await session.scalar(select(func.count()).select_from(ArtifactBinding)) == 1 - assert await session.scalar(select(func.count()).select_from(ArtifactReplica)) == 2 - assert ( - await session.scalar(select(func.count()).select_from(ArtifactOperationReceipt)) - == 4 - ) - assert await session.scalar( - select(func.count()) - .select_from(AuditEvent) - .where(AuditEvent.event_type == "artifact_replica_quarantined") - ) == 1 - assert await session.scalar( - select(func.count()) - .select_from(AuditEvent) - .where(AuditEvent.event_type == "artifact_retention_released") - ) == 1 - reconciliation_event = ( - await session.scalars( - select(AuditEvent).where( - AuditEvent.event_type == "artifact_replica_reconciled" - ) - ) - ).one() - assert len(reconciliation_event.from_status or "") <= 30 - assert len(reconciliation_event.to_status or "") <= 30 - assert reconciliation_event.event_payload["before"]["verification_state"] == ( - "failed" - ) - quarantine_event = ( - await session.scalars( - select(AuditEvent).where( - AuditEvent.event_type == "artifact_replica_quarantined" - ) - ) - ).one() - assert quarantine_event.from_status == "unknown" - affected_replica = await session.scalar( - select(ArtifactReplica).where( - ArtifactReplica.provider_artifact_id == replica_provider_id - ) - ) - secondary_replica = await session.get(ArtifactReplica, secondary_replica_id) - binding = await session.get(ArtifactBinding, binding_id) - assert affected_replica is not None - assert affected_replica.integrity_state == "quarantined" - assert secondary_replica is not None - assert binding is not None - assert { - column.name: getattr(binding, column.name) - for column in ArtifactBinding.__table__.columns - } == binding_before - assert { - column.name: getattr(secondary_replica, column.name) - for column in ArtifactReplica.__table__.columns - } == secondary_replica_before + item = await session.get(ArtifactUploadItem, item_id) + assert item is not None + upload_session = await session.get(ArtifactUploadSession, item.session_id) + assert item.state == "failed" + assert item.error_code == "artifact_input_mismatch" + assert upload_session is not None + assert upload_session.reserved_bytes == 0 + assert upload_session.reserved_items == 0 + assert store.put_calls == 1 finally: await engine.dispose() @pytest.mark.asyncio -async def test_ingest_service_input_failure_promotes_zero_facts( - artifact_database_env: str, tmp_path: Path +async def test_changed_commitment_is_rejected_before_provider_io( + artifact_database_env: str, + tmp_path: Path, ) -> None: - """Prove failed input commitment creates no immutable storage facts.""" engine = create_async_engine(artifact_database_env) factory = async_sessionmaker(engine, expire_on_commit=False) + store = _UnavailableStore() try: - request = store_request(expected_sha256="sha256:" + "0" * 64, expected_size=4) - ids = await _seed_reserved_item(factory, request) - async with factory() as session: - service = ArtifactIngestService( - session, - LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2), - adapter_name="local", - ) - with pytest.raises(ArtifactInputMismatchError): - await service.ingest_reserved_item(ids["item"], byte_stream(b"data"), request) - async with factory() as session: - assert await session.scalar(select(func.count()).select_from(ArtifactContent)) == 0 - assert await session.scalar(select(func.count()).select_from(ArtifactReplica)) == 0 - assert ( - await session.scalar(select(func.count()).select_from(ArtifactOperationReceipt)) - == 0 - ) - assert await session.scalar(select(func.count()).select_from(ArtifactBinding)) == 0 - item = await session.get(ArtifactUploadItem, ids["item"]) - assert item is not None and item.state == "failed" + async with _minted_source(tmp_path / "scratch-first", b"first") as first: + async with factory() as session: + item_id = await _seed_reserved_item(session, first.commitment) + async with _minted_source(tmp_path / "scratch-second", b"second") as second: + async with factory() as session: + with pytest.raises(ArtifactIngestStateError, match="commitment changed"): + await ArtifactStorageOrchestrator( + session, + store, + artifact_storage_namespace_spec(_settings(tmp_path), store), + ).put_reserved_item(item_id, second) + assert store.put_calls == 0 finally: await engine.dispose() @pytest.mark.asyncio -async def test_ingest_service_reconciles_provider_committed_gap( - artifact_database_env: str, tmp_path: Path +async def test_namespace_mismatch_fails_before_provider_io( + artifact_database_env: str, + tmp_path: Path, ) -> None: - """Finalize from provider receipt after a committed effect misses transaction B.""" engine = create_async_engine(artifact_database_env) factory = async_sessionmaker(engine, expire_on_commit=False) - payload = b"recovery" - request = store_request( - key="provider-gap", - expected_sha256="sha256:" + hashlib.sha256(payload).hexdigest(), - expected_size=len(payload), - maximum_bytes=len(payload), + store = _UnavailableStore() + settings = _settings(tmp_path) + canonical = artifact_storage_namespace_spec(settings, store) + conflicting_descriptor = dict(canonical.namespace_descriptor) + conflicting_descriptor["private_root_identity"] = "sha256:" + "f" * 64 + conflicting = ArtifactStorageNamespaceSpec( + backend=canonical.backend, + adapter=canonical.adapter, + provider_profile=canonical.provider_profile, + namespace_descriptor=conflicting_descriptor, + namespace_fingerprint=canonical_json_hash(conflicting_descriptor), ) try: - ids = await _seed_reserved_item(factory, request) - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - fence = await service._start_provider_operation(ids["item"], request) - stored = await adapter.store(byte_stream(payload), request) - await service._mark_provider_recovery_required( - ids["item"], request, fence, provider_commit_confirmed=True - ) - async with factory() as session: - item = await session.get(ArtifactUploadItem, ids["item"]) - assert item is not None and item.state == "provider_committed" - async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - recovered = await service.reconcile_committed_item(ids["item"], request) - assert recovered.provider_artifact_id == stored.provider_artifact_id - assert recovered.replayed is True async with factory() as session: - item = await session.get(ArtifactUploadItem, ids["item"]) - upload_session = await session.get(ArtifactUploadSession, ids["session"]) - assert item is not None and item.state == "ready" - assert upload_session is not None - assert upload_session.reserved_bytes == 0 - assert upload_session.current_bytes == len(payload) - assert await session.scalar(select(func.count()).select_from(ArtifactContent)) == 1 - assert await session.scalar(select(func.count()).select_from(ArtifactReplica)) == 1 - assert ( - await session.scalar(select(func.count()).select_from(ArtifactOperationReceipt)) - == 1 - ) + await ArtifactStorageOrchestrator(session, store, canonical).ensure_storage_namespace() + async with _minted_source(tmp_path / "scratch", b"blocked") as source: + async with factory() as session: + item_id = await _seed_reserved_item(session, source.commitment) + with pytest.raises(ArtifactStorageNamespaceError): + await ArtifactStorageOrchestrator( + session, store, conflicting + ).put_reserved_item(item_id, source) + assert store.put_calls == 0 finally: await engine.dispose() @pytest.mark.asyncio -async def test_ingest_service_replays_ambiguous_provider_failure( +async def test_concurrent_different_first_namespace_writers_have_one_winner( artifact_database_env: str, tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, ) -> None: - """Recover a committed provider effect after the call reports retryable failure.""" engine = create_async_engine(artifact_database_env) factory = async_sessionmaker(engine, expire_on_commit=False) - payload = b"ambiguous-commit" - request = store_request( - key="ambiguous", - expected_sha256="sha256:" + hashlib.sha256(payload).hexdigest(), - expected_size=len(payload), - maximum_bytes=len(payload), + first_store = _NamespaceObservingAcknowledgingStore(factory) + second_store = _NamespaceObservingAcknowledgingStore(factory) + first = artifact_storage_namespace_spec(_settings(tmp_path), first_store) + second_descriptor = dict(first.namespace_descriptor) + second_descriptor["private_root_identity"] = "sha256:" + "a" * 64 + second = ArtifactStorageNamespaceSpec( + backend=first.backend, + adapter=first.adapter, + provider_profile=first.provider_profile, + namespace_descriptor=second_descriptor, + namespace_fingerprint=canonical_json_hash(second_descriptor), ) - try: - ids = await _seed_reserved_item(factory, request) - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - original_store = adapter.store - - async def commit_then_fail( - stream: AsyncIterator[bytes], store_operation: StoreArtifactRequest - ): - await original_store(stream, store_operation) - raise ArtifactStoreUnavailableError("provider acknowledgement was lost") - monkeypatch.setattr(adapter, "store", commit_then_fail) + async def publish( + item_id: str, + source: CommittedArtifactSource, + store: _NamespaceObservingAcknowledgingStore, + spec: ArtifactStorageNamespaceSpec, + ) -> object: async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - with pytest.raises(ArtifactStoreUnavailableError): - await service.ingest_reserved_item(ids["item"], byte_stream(payload), request) - async with factory() as session: - item = await session.get(ArtifactUploadItem, ids["item"]) - assert item is not None and item.state == "replay_required" - monkeypatch.setattr(adapter, "store", original_store) - async with factory() as session: - recovered = await ArtifactIngestService( - session, adapter, adapter_name="local" - ).ingest_reserved_item(ids["item"], byte_stream(payload), request) - assert recovered.sha256 == request.expected_sha256 - assert recovered.replayed is True - async with factory() as session: - item = await session.get(ArtifactUploadItem, ids["item"]) - assert item is not None and item.state == "ready" - assert await session.scalar(select(func.count()).select_from(ArtifactContent)) == 1 - assert await session.scalar(select(func.count()).select_from(ArtifactReplica)) == 1 - assert ( - await session.scalar(select(func.count()).select_from(ArtifactOperationReceipt)) - == 1 - ) - assert await session.scalar(select(func.count()).select_from(ArtifactBinding)) == 0 - assert await session.scalar(select(func.count()).select_from(AuditEvent)) == 0 - assert len(list((tmp_path / "artifacts" / "objects").iterdir())) == 1 - - pre_effect_request = store_request( - key="pre-effect-failure", - expected_sha256=request.expected_sha256, - expected_size=request.expected_size, - maximum_bytes=len(payload), - ) - pre_effect_ids = await _seed_reserved_item(factory, pre_effect_request) - - async def fail_before_effect( - stream: AsyncIterator[bytes], store_operation: StoreArtifactRequest - ): - del stream, store_operation - raise ArtifactStoreUnavailableError("provider was unavailable") + try: + return await ArtifactStorageOrchestrator(session, store, spec).put_reserved_item( + item_id, source + ) + except ArtifactStorageNamespaceError as exc: + return exc - monkeypatch.setattr(adapter, "store", fail_before_effect) - async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - with pytest.raises(ArtifactStoreUnavailableError): - await service.ingest_reserved_item( - pre_effect_ids["item"], byte_stream(payload), pre_effect_request + try: + async with _minted_source(tmp_path / "scratch-first", b"first") as first_source: + async with _minted_source(tmp_path / "scratch-second", b"second") as second_source: + async with factory() as session: + first_item_id = await _seed_reserved_item(session, first_source.commitment) + second_item_id = await _seed_reserved_item(session, second_source.commitment) + outcomes = await asyncio.gather( + publish(first_item_id, first_source, first_store, first), + publish(second_item_id, second_source, second_store, second), ) - async with factory() as session: - item = await session.get(ArtifactUploadItem, pre_effect_ids["item"]) - assert item is not None and item.state == "replay_required" - - sha_only_request = store_request( - key="sha-only-confirmed", - expected_sha256=request.expected_sha256, - expected_size=None, - maximum_bytes=len(payload), - ) - sha_only_ids = await _seed_reserved_item(factory, sha_only_request) - async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - fence = await service._start_provider_operation(sha_only_ids["item"], sha_only_request) - await service._mark_provider_recovery_required( - sha_only_ids["item"], - sha_only_request, - fence, - provider_commit_confirmed=True, - ) - async with factory() as session: - item = await session.get(ArtifactUploadItem, sha_only_ids["item"]) - assert item is not None and item.state == "replay_required" - - finalize_cancel_request = store_request( - key="finalize-cancelled", - expected_sha256=request.expected_sha256, - expected_size=request.expected_size, - maximum_bytes=len(payload), + + assert sum(isinstance(value, ArtifactPutResult) for value in outcomes) == 1 + assert sum(isinstance(value, ArtifactStorageNamespaceError) for value in outcomes) == 1 + winning_index = next( + index for index, value in enumerate(outcomes) if isinstance(value, ArtifactPutResult) ) - finalize_cancel_ids = await _seed_reserved_item(factory, finalize_cancel_request) - monkeypatch.setattr(adapter, "store", original_store) - async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - - async def cancel_finalization(*args) -> None: - del args - raise asyncio.CancelledError - - monkeypatch.setattr(service, "_finalize_provider_operation", cancel_finalization) - with pytest.raises(asyncio.CancelledError): - await service.ingest_reserved_item( - finalize_cancel_ids["item"], - byte_stream(payload), - finalize_cancel_request, - ) - async with factory() as session: - item = await session.get(ArtifactUploadItem, finalize_cancel_ids["item"]) - assert item is not None and item.state == "provider_committed" - - reconcile_cancel_request = store_request( - key="reconcile-cancelled", - expected_sha256=request.expected_sha256, - expected_size=request.expected_size, - maximum_bytes=len(payload), + stores = (first_store, second_store) + specs = (first, second) + assert stores[winning_index].put_calls == 1 + assert ( + stores[winning_index].observed_namespace_fingerprint + == specs[winning_index].namespace_fingerprint ) - reconcile_cancel_ids = await _seed_reserved_item(factory, reconcile_cancel_request) - monkeypatch.setattr(adapter, "store", original_store) + assert stores[1 - winning_index].put_calls == 0 + assert stores[1 - winning_index].observed_namespace_fingerprint is None async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - fence = await service._start_provider_operation( - reconcile_cancel_ids["item"], reconcile_cancel_request - ) - await adapter.store(byte_stream(payload), reconcile_cancel_request) - await service._mark_provider_recovery_required( - reconcile_cancel_ids["item"], - reconcile_cancel_request, - fence, - provider_commit_confirmed=True, - ) + namespace = await session.scalar(select(ArtifactStorageNamespace)) + assert namespace is not None + assert namespace.namespace_fingerprint == specs[winning_index].namespace_fingerprint + finally: + await engine.dispose() - recovery_started = asyncio.Event() - recovery_block = asyncio.Event() - async def block_recovery(*args) -> None: - del args - recovery_started.set() - await recovery_block.wait() +def test_namespace_descriptor_is_canonical_and_does_not_store_local_path( + tmp_path: Path, +) -> None: + settings = _settings(tmp_path) + store = _UnavailableStore() + spec = artifact_storage_namespace_spec(settings, store) + serialized = repr(spec.namespace_descriptor) - monkeypatch.setattr(adapter, "recover_committed_store", block_recovery) - async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - original_mark_recovery = service._mark_provider_recovery_required - cleanup_started = asyncio.Event() - cleanup_block = asyncio.Event() - cleanup_finished = asyncio.Event() - - async def block_then_mark_recovery(*args, **kwargs) -> None: - cleanup_started.set() - await cleanup_block.wait() - await original_mark_recovery(*args, **kwargs) - cleanup_finished.set() - - monkeypatch.setattr( - service, "_mark_provider_recovery_required", block_then_mark_recovery - ) - reconcile_task = asyncio.create_task( - service.reconcile_committed_item( - reconcile_cancel_ids["item"], reconcile_cancel_request - ) - ) - await recovery_started.wait() - reconcile_task.cancel() - await cleanup_started.wait() - reconcile_task.cancel() - await asyncio.sleep(0) - assert not reconcile_task.done() - cleanup_block.set() - with pytest.raises(asyncio.CancelledError): - await reconcile_task - assert cleanup_finished.is_set() - async with factory() as session: - item = await session.get(ArtifactUploadItem, reconcile_cancel_ids["item"]) - assert item is not None and item.state == "provider_committed" - finally: - await engine.dispose() + assert spec.backend == "local" + assert spec.provider_profile == "local-v2" + assert spec.namespace_fingerprint == canonical_json_hash(spec.namespace_descriptor) + assert str(settings.artifact_local_root) not in serialized + assert spec.namespace_descriptor["private_root_identity"].startswith("sha256:") + + +def test_namespace_descriptor_changes_when_local_root_is_replaced(tmp_path: Path) -> None: + settings = _settings(tmp_path) + first_store = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) + original = artifact_storage_namespace_spec(settings, first_store) + first_store.close() + + root = settings.artifact_local_root + assert root is not None + root.rename(tmp_path / "replaced-store") + root.mkdir(mode=0o700) + + replacement_store = LocalStorageBootstrap(LocalStorageAdapter(root=root)) + replacement = artifact_storage_namespace_spec(settings, replacement_store) + replacement_store.close() + assert replacement.namespace_descriptor != original.namespace_descriptor + assert replacement.namespace_fingerprint != original.namespace_fingerprint + + +def test_store_bootstrap_requires_preprovisioned_private_root(tmp_path: Path) -> None: + with pytest.raises(ArtifactConfigurationError, match="storage is unavailable"): + LocalStorageAdapter(root=tmp_path / "missing-store") + + +def test_namespace_spec_rejects_adapter_drift(tmp_path: Path) -> None: + local_store = _UnavailableStore() + with pytest.raises(ArtifactStorageNamespaceError, match="identity"): + artifact_storage_namespace_spec(Settings(), local_store) @pytest.mark.asyncio -async def test_reconciliation_rehashes_bytes_and_fails_corrupt_item( - artifact_database_env: str, tmp_path: Path +@pytest.mark.parametrize( + "provider_failure", + [asyncio.CancelledError("cancelled"), RuntimeError("provider crashed")], +) +async def test_unexpected_or_cancelled_provider_failure_requires_replay( + tmp_path: Path, + provider_failure: BaseException, ) -> None: - """Reject altered committed bytes and fail the unpromoted item.""" - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - payload = b"trusted" - request = store_request( - key="altered-recovery", - expected_sha256="sha256:" + hashlib.sha256(payload).hexdigest(), - expected_size=len(payload), - maximum_bytes=len(payload), + store = _UnavailableStore() + store.put = AsyncMock(side_effect=provider_failure) # type: ignore[method-assign] + orchestrator = ArtifactStorageOrchestrator( + None, # type: ignore[arg-type] + store, + artifact_storage_namespace_spec(_settings(tmp_path), store), ) - try: - ids = await _seed_reserved_item(factory, request) - root = tmp_path / "artifacts" - adapter = LocalStorageAdapter(root=root, buffer_bytes=2) - async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - fence = await service._start_provider_operation(ids["item"], request) - await adapter.store(byte_stream(payload), request) - await service._mark_provider_recovery_required( - ids["item"], request, fence, provider_commit_confirmed=True - ) - next((root / "objects").iterdir()).write_bytes(b"altered") - async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - with pytest.raises(ArtifactIntegrityError): - await service.reconcile_committed_item(ids["item"], request) - async with factory() as session: - item = await session.get(ArtifactUploadItem, ids["item"]) - assert item is not None and item.state == "failed" - assert item.error_code == "artifact_integrity_failure" - assert await session.scalar(select(func.count()).select_from(ArtifactContent)) == 0 - - missing_request = store_request( - key="missing-confirmed-recovery", - expected_sha256=request.expected_sha256, - expected_size=request.expected_size, - maximum_bytes=request.maximum_bytes, - ) - missing_ids = await _seed_reserved_item(factory, missing_request) - missing_root = tmp_path / "missing-artifacts" - missing_adapter = LocalStorageAdapter(root=missing_root, buffer_bytes=2) - async with factory() as session: - service = ArtifactIngestService(session, missing_adapter, adapter_name="local") - fence = await service._start_provider_operation( - missing_ids["item"], missing_request - ) - await missing_adapter.store(byte_stream(payload), missing_request) - await service._mark_provider_recovery_required( - missing_ids["item"], - missing_request, - fence, - provider_commit_confirmed=True, - ) - next((missing_root / "objects").iterdir()).unlink() - async with factory() as session: - service = ArtifactIngestService(session, missing_adapter, adapter_name="local") - with pytest.raises(ArtifactIntegrityError, match="missing"): - await service.reconcile_committed_item(missing_ids["item"], missing_request) - async with factory() as session: - missing_item = await session.get(ArtifactUploadItem, missing_ids["item"]) - assert missing_item is not None and missing_item.state == "failed" - assert missing_item.error_code == "artifact_integrity_failure" - finally: - await engine.dispose() + orchestrator._start_put = AsyncMock( # type: ignore[method-assign] + return_value=ProviderAttemptFence(1) + ) + orchestrator._mark_replay_required = AsyncMock() # type: ignore[method-assign] + + async with _minted_source(tmp_path / "scratch", b"provider") as source: + with pytest.raises(type(provider_failure)): + await orchestrator.put_reserved_item("item", source) + orchestrator._mark_replay_required.assert_awaited_once() # type: ignore[attr-defined] @pytest.mark.asyncio -async def test_identical_content_creates_distinct_provider_replicas( - artifact_database_env: str, tmp_path: Path +@pytest.mark.parametrize( + "finalization_failure", + [ + ArtifactInputMismatchError("mismatch"), + asyncio.CancelledError("cancelled"), + RuntimeError("database unavailable"), + ], +) +async def test_finalization_failure_never_leaves_uploading_unresolved( + tmp_path: Path, + finalization_failure: BaseException, ) -> None: - """Deduplicate content facts without collapsing distinct physical replicas.""" - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - payload = b"shared-content" - digest = "sha256:" + hashlib.sha256(payload).hexdigest() - first_request = store_request( - key="shared-first", - expected_sha256=digest, - expected_size=len(payload), - maximum_bytes=len(payload), + store = _UnavailableStore() + store.put = AsyncMock( # type: ignore[method-assign] + return_value=ArtifactPutResult("sha256/00/" + "0" * 62, replayed=False) ) - second_request = store_request( - key="shared-second", - expected_sha256=digest, - expected_size=len(payload), - maximum_bytes=len(payload), + orchestrator = ArtifactStorageOrchestrator( + None, # type: ignore[arg-type] + store, + artifact_storage_namespace_spec(_settings(tmp_path), store), ) - try: - first_ids = await _seed_reserved_item(factory, first_request) - second_ids = await _seed_reserved_item(factory, second_request) - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - await service.ingest_reserved_item( - first_ids["item"], byte_stream(payload), first_request - ) - async with factory() as session: - service = ArtifactIngestService(session, adapter, adapter_name="local") - await service.ingest_reserved_item( - second_ids["item"], byte_stream(payload), second_request - ) - async with factory() as session: - assert await session.scalar(select(func.count()).select_from(ArtifactContent)) == 1 - assert await session.scalar(select(func.count()).select_from(ArtifactReplica)) == 2 - provider_ids = set(await session.scalars(select(ArtifactReplica.provider_artifact_id))) - assert len(provider_ids) == 2 - finally: - await engine.dispose() + orchestrator._start_put = AsyncMock( # type: ignore[method-assign] + return_value=ProviderAttemptFence(1) + ) + orchestrator._finalize_put = AsyncMock( # type: ignore[method-assign] + side_effect=finalization_failure + ) + orchestrator._fail_put = AsyncMock() # type: ignore[method-assign] + orchestrator._mark_replay_required = AsyncMock() # type: ignore[method-assign] + + async with _minted_source(tmp_path / "scratch", b"finalize") as source: + with pytest.raises(type(finalization_failure)): + await orchestrator.put_reserved_item("item", source) + if isinstance(finalization_failure, ArtifactInputMismatchError): + orchestrator._fail_put.assert_awaited_once() # type: ignore[attr-defined] + orchestrator._mark_replay_required.assert_not_awaited() # type: ignore[attr-defined] + else: + orchestrator._mark_replay_required.assert_awaited_once() # type: ignore[attr-defined] + orchestrator._fail_put.assert_not_awaited() # type: ignore[attr-defined] -async def _seed_reserved_item( - factory: async_sessionmaker, request: StoreArtifactRequest -) -> dict[str, str]: - """Create one project, upload session, and reserved upload item.""" - ids = {name: str(uuid4()) for name in ("project", "session", "item")} - async with factory() as session, session.begin(): - session.add(Project(id=ids["project"], name="Artifact test", slug=f"artifact-{uuid4()}")) - await session.flush() - session.add( - ArtifactUploadSession( - id=ids["session"], - actor_id="actor-1", - project_id=ids["project"], - permitted_roles=["submission_packet"], - state="open", - maximum_bytes=max(64, request.maximum_bytes), - current_bytes=0, - reserved_bytes=request.maximum_bytes, - maximum_items=4, - current_items=0, - reserved_items=1, - expires_at=datetime.now(UTC) + timedelta(hours=1), - cas_version=0, - ) - ) - await session.flush() - session.add( - ArtifactUploadItem( - id=ids["item"], - session_id=ids["session"], - logical_role="submission_packet", - display_name="packet.bin", - media_type="application/octet-stream", - reserved_bytes=request.maximum_bytes, - expected_sha256=request.expected_sha256, - expected_size=request.expected_size, - idempotency_key=request.idempotency.key, - request_digest=request.idempotency.request_digest, - state="reserved", - cas_version=0, - ) - ) - return ids +@pytest.mark.asyncio +async def test_startup_namespace_validation_uses_canonical_session_factory( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + settings = _settings(tmp_path) + store = _UnavailableStore() + class _TransactionContext: + async def __aenter__(self) -> None: + return None -async def _truncate_artifact_tables(database_url: str) -> None: - """Clear additive artifact rows before exercising the guarded downgrade.""" - engine = create_async_engine(database_url) - try: - async with engine.begin() as connection: - await connection.exec_driver_sql( - """ - truncate table - artifact_operation_receipts, - artifact_replicas, - artifact_bindings, - artifact_upload_items, - artifact_contents, - artifact_upload_sessions - cascade - """ - ) - finally: - await engine.dispose() + async def __aexit__(self, *_args: object) -> None: + return None + + class _Session: + def begin(self) -> _TransactionContext: + return _TransactionContext() + + session = _Session() + + class _SessionContext: + async def __aenter__(self) -> object: + return session + + async def __aexit__(self, *_args: object) -> None: + return None + + repository = object() + ensured = AsyncMock() + + def create_repository(candidate_session: object) -> object: + assert candidate_session is session + return repository + + monkeypatch.setattr( + artifact_service_module, + "get_session_factory", + lambda: lambda: _SessionContext(), + ) + monkeypatch.setattr(artifact_service_module, "ArtifactRepository", create_repository) + monkeypatch.setattr( + artifact_service_module, + "_claim_and_validate_storage_namespace", + ensured, + ) + + claim = await validate_artifact_storage_namespace_at_startup(store, settings) + assert isinstance(ensured.await_args.args[1], ArtifactStorageNamespaceSpec) + assert ensured.await_args.args[0] is repository + assert claim.namespace_identity == store.namespace_identity + + +def test_models_remove_v1_provider_retention_and_receipt_fields() -> None: + assert not hasattr(ArtifactReplica, "retention_state") + assert not hasattr(ArtifactReplica, "provider_artifact_id") + assert not hasattr(ArtifactReplica, "provider_manifest_id") + assert not hasattr(ArtifactOperationReceipt, "provider_receipt_id") + assert not hasattr(ArtifactOperationReceipt, "retention_reference") + assert not hasattr(ArtifactOperationReceipt, "service_principal") + assert not hasattr(ArtifactUploadItem, "provider_operation_reference") + assert hasattr(ArtifactReplica, "provider_object_ref") + assert hasattr(ArtifactStorageNamespace, "namespace_fingerprint") diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 982777002..114e6740d 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -4,17 +4,21 @@ import json from collections.abc import Mapping from pathlib import Path +from typing import get_args import pytest from pydantic import SecretStr, ValidationError from pydantic_settings import BaseSettings -from app.adapters.artifacts import resolve_artifact_store +from app.adapters.artifacts import create_artifact_store_bootstrap from app.adapters.auth.flow import FlowAuthVerifier from app.api.deps.auth import get_application_auth_verifier from app.core.auth import clear_auth_verifier_cache, get_auth_verifier from app.core.config import Settings, get_settings -from app.interfaces.artifacts import ArtifactConfigurationError +from app.interfaces.external_services import ( + ExternalServiceConfigurationError, + UnknownExternalServiceProviderError, +) from app.main import create_app @@ -459,33 +463,59 @@ def test_settings_reject_local_artifacts_outside_development(environment: str) - environment=environment, artifact_store_backend="local", artifact_local_root="/tmp/workstream-artifacts", - artifact_retention_policy_version="retention-v1", + artifact_scratch_root="/tmp/workstream-artifact-scratch", + ) + + +def test_settings_require_scratch_root_for_enabled_artifacts() -> None: + with pytest.raises(ValidationError, match="artifact scratch root"): + Settings( + environment="test", + artifact_store_backend="s3_compatible", ) -def test_settings_require_retention_policy_for_enabled_artifacts() -> None: - """Fail construction when enabled storage lacks approved retention.""" - with pytest.raises(ValidationError, match="retention policy"): - Settings(environment="test", artifact_store_backend="flow_node") +@pytest.mark.parametrize("interval", [0, -1, 86_401]) +def test_artifact_scratch_cleanup_interval_is_positive_and_bounded(interval: int) -> None: + with pytest.raises(ValidationError): + Settings(artifact_scratch_cleanup_interval_seconds=interval) + + +def test_artifact_backend_enum_is_exact_and_flow_node_is_rejected(tmp_path: Path) -> None: + annotation = Settings.model_fields["artifact_store_backend"].annotation + assert get_args(annotation) == ("disabled", "local", "s3_compatible") + + with pytest.raises(ValidationError): + Settings( + environment="test", + artifact_store_backend="flow_node", # type: ignore[arg-type] + artifact_scratch_root=tmp_path / "scratch", + ) -def test_local_artifact_settings_and_resolver(tmp_path) -> None: - """Resolve local storage only from complete development configuration.""" +def test_local_artifact_settings_and_factory(tmp_path: Path) -> None: + """Construct local storage only from complete development configuration.""" settings = Settings( environment="test", artifact_store_backend="local", artifact_local_root=tmp_path / "artifacts", - artifact_retention_policy_version="retention-v1", + artifact_scratch_root=tmp_path / "scratch", artifact_stream_buffer_bytes=64, artifact_operation_lock_timeout_seconds=17, ) - adapter = resolve_artifact_store(settings) - assert adapter.adapter_name == "local" - assert adapter._lock_timeout_seconds == 17 + (tmp_path / "artifacts").mkdir(mode=0o700) + adapter = create_artifact_store_bootstrap(settings) + assert adapter.identity.capability_key == "artifact_store" + assert adapter.identity.provider_key == "local" + assert adapter._adapter._lock_timeout_seconds == 17 adapter.close() incomplete = settings.model_copy(update={"artifact_local_root": None}) - with pytest.raises(ArtifactConfigurationError, match="root is not configured"): - resolve_artifact_store(incomplete) + with pytest.raises(UnknownExternalServiceProviderError) as disabled: + create_artifact_store_bootstrap(Settings()) + assert disabled.value.identity is not None + assert disabled.value.identity.provider_key == "disabled" + with pytest.raises(ExternalServiceConfigurationError): + create_artifact_store_bootstrap(incomplete) def test_artifact_scratch_settings_are_bounded_and_separate(tmp_path) -> None: @@ -521,12 +551,15 @@ def test_artifact_scratch_settings_are_bounded_and_separate(tmp_path) -> None: ) -def test_flow_node_resolver_fails_until_adapter_chunk() -> None: - """Do not imply that the reserved Flow Node backend is implemented.""" +def test_s3_compatible_factory_fails_typed_until_adapter_chunk(tmp_path: Path) -> None: + """Keep valid future S3 configuration unavailable until its owned chunk.""" settings = Settings( - environment="production", - artifact_store_backend="flow_node", - artifact_retention_policy_version="retention-v1", + environment="test", + artifact_store_backend="s3_compatible", + artifact_scratch_root=tmp_path / "scratch", ) - with pytest.raises(ArtifactConfigurationError, match="not available"): - resolve_artifact_store(settings) + with pytest.raises(UnknownExternalServiceProviderError) as caught: + create_artifact_store_bootstrap(settings) + + assert caught.value.identity is not None + assert caught.value.identity.provider_key == "s3_compatible" diff --git a/backend/tests/test_local_artifact_store.py b/backend/tests/test_local_artifact_store.py index 818e68222..fa61bf421 100644 --- a/backend/tests/test_local_artifact_store.py +++ b/backend/tests/test_local_artifact_store.py @@ -1,429 +1,870 @@ -"""Focused LocalStorage regression proof for the private write refactor.""" +"""Filesystem, cancellation, and security proof for LocalStorage v2.""" from __future__ import annotations import asyncio from collections.abc import AsyncIterator -from dataclasses import replace -import errno +from concurrent.futures import ThreadPoolExecutor import hashlib import os from pathlib import Path import threading +from typing import cast import pytest -from app.adapters.artifacts.local import LocalStorageAdapter -from app.core.file_locks import acquire_exclusive_file_lock +from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap from app.interfaces.artifacts import ( + ArtifactByteRange, + ArtifactConfigurationError, + ArtifactInputMismatchError, ArtifactIntegrityError, - ArtifactOperation, + ArtifactOperationConflictError, ArtifactStoreUnavailableError, - IdempotencyIdentity, - StoreArtifactRequest, - canonical_store_request_digest, + ArtifactStoreNamespaceClaim, +) +from app.modules.artifacts.sources import ArtifactCommitment, CommittedArtifactSource +from tests.artifact_store_helpers import ( + initialize_local_store, + local_namespace_claim, + minted_source, ) -async def byte_stream(*chunks: bytes) -> AsyncIterator[bytes]: - """Yield exact LocalStorage test bytes.""" - for chunk in chunks: - yield chunk +def object_path(root: Path, provider_object_ref: str) -> Path: + """Resolve a test-only local provider reference inside its private root.""" + return root / "objects" / provider_object_ref + + +def test_local_layout_is_private_content_derived_and_v2_only(tmp_path: Path) -> None: + """Persist one private content path without v1 metadata or product identity.""" + + async def exercise() -> str: + root = tmp_path / "artifacts" + adapter = initialize_local_store(root=root, buffer_bytes=2) + try: + async with minted_source(tmp_path / "scratch", b"private bytes") as source: + return (await adapter.put(source)).provider_object_ref + finally: + adapter.close() + + provider_object_ref = asyncio.run(exercise()) + digest = hashlib.sha256(b"private bytes").hexdigest() + root = tmp_path / "artifacts" + assert provider_object_ref == f"sha256/{digest[:2]}/{digest[2:]}" + assert object_path(root, provider_object_ref).read_bytes() == b"private bytes" + assert (root.stat().st_mode & 0o777) == 0o700 + assert (object_path(root, provider_object_ref).stat().st_mode & 0o777) == 0o400 + assert set(path.name for path in root.iterdir()) == { + ".workstream-artifact-store-v2", + "locks", + "objects", + "tmp", + } + assert not {"intents", "metadata", "receipts", "retentions", "quarantine"} & { + path.name for path in root.iterdir() + } + assert "private" not in provider_object_ref + + +def test_local_layout_initialization_serializes_and_publishes_marker_atomically( + tmp_path: Path, +) -> None: + """Allow concurrent bootstrap only after one complete marker is durable.""" + root = tmp_path / "artifacts" + root.mkdir(mode=0o700) + bootstraps = [ + LocalStorageBootstrap(LocalStorageAdapter(root=root)), + LocalStorageBootstrap(LocalStorageAdapter(root=root)), + ] + claims = [local_namespace_claim(bootstrap) for bootstrap in bootstraps] + + def initialize(index: int) -> None: + bootstraps[index].initialize_after_namespace_claim(claims[index]) + try: + with ThreadPoolExecutor(max_workers=2) as pool: + futures = [pool.submit(initialize, index) for index in range(2)] + for future in futures: + future.result(timeout=5) + finally: + for bootstrap in bootstraps: + bootstrap.close() -def store_request() -> StoreArtifactRequest: - """Build one canonical active-v1 store request.""" - identity = IdempotencyIdentity( - service_principal="workstream.artifact.ingest", - operation=ArtifactOperation.STORE, - key="private-helper-regression", - request_digest="sha256:" + "0" * 64, + assert (root / ".workstream-artifact-store-v2").read_bytes() == ( + b"workstream-artifact-store-v2\n" ) - request = StoreArtifactRequest( - expected_sha256="sha256:" + hashlib.sha256(b"hello").hexdigest(), - expected_size=5, - maximum_bytes=5, - media_type="text/plain", - metadata={}, - idempotency=identity, + assert not (root / ".workstream-artifact-store-v2.initializing").exists() + + +def test_local_recovers_only_one_complete_durable_marker_temporary( + tmp_path: Path, +) -> None: + """Finish an interrupted atomic rename only for exact marker bytes.""" + root = tmp_path / "artifacts" + root.mkdir(mode=0o700) + temporary = root / ".workstream-artifact-store-v2.initializing" + temporary.write_bytes(b"workstream-artifact-store-v2\n") + temporary.chmod(0o600) + + adapter = initialize_local_store(root=root) + adapter.close() + + assert (root / ".workstream-artifact-store-v2").read_bytes() == ( + b"workstream-artifact-store-v2\n" ) - return replace( - request, - idempotency=replace( - identity, - request_digest=canonical_store_request_digest(request), - ), + assert not temporary.exists() + + +def test_local_recovers_linked_marker_after_interrupted_temporary_unlink( + tmp_path: Path, +) -> None: + """Reduce the exact two-link crash state back to one canonical marker.""" + root = tmp_path / "artifacts" + root.mkdir(mode=0o700) + temporary = root / ".workstream-artifact-store-v2.initializing" + marker = root / ".workstream-artifact-store-v2" + temporary.write_bytes(b"workstream-artifact-store-v2\n") + temporary.chmod(0o600) + os.link(temporary, marker) + assert temporary.stat().st_ino == marker.stat().st_ino + assert temporary.stat().st_nlink == 2 + + adapter = initialize_local_store(root=root) + adapter.close() + + assert marker.read_bytes() == b"workstream-artifact-store-v2\n" + assert marker.stat().st_nlink == 1 + assert not temporary.exists() + + +def test_local_refuses_v1_or_unknown_disk_layout(tmp_path: Path) -> None: + """Do not install a dual reader over an incompatible pre-cutover root.""" + root = tmp_path / "artifacts" + root.mkdir(mode=0o700) + (root / "intents").mkdir(mode=0o700) + with pytest.raises(ArtifactConfigurationError, match="layout is incompatible"): + initialize_local_store(root=root) + + marker_root = tmp_path / "bad-marker" + marker_root.mkdir(mode=0o700) + marker = marker_root / ".workstream-artifact-store-v2" + marker.write_bytes(b"wrong\n") + marker.chmod(0o600) + with pytest.raises(ArtifactConfigurationError, match="layout is incompatible"): + initialize_local_store(root=marker_root) + + executable_marker_root = tmp_path / "executable-marker" + executable_marker_root.mkdir(mode=0o700) + executable_marker = executable_marker_root / ".workstream-artifact-store-v2" + executable_marker.write_bytes(b"workstream-artifact-store-v2\n") + executable_marker.chmod(0o700) + with pytest.raises(ArtifactConfigurationError, match="layout is incompatible"): + initialize_local_store(root=executable_marker_root) + + for index, relative_path in enumerate( + ( + "legacy", + "objects/legacy", + "objects/sha256/zz", + "objects/sha256/aa/not-an-object", + "tmp/not-a-temporary", + "locks/not-a-lock", + ) + ): + extra_entry_root = tmp_path / f"extra-entry-{index}" + initialized = initialize_local_store(root=extra_entry_root) + initialized.close() + extra_entry = extra_entry_root / relative_path + extra_entry.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + if "not-" in extra_entry.name: + extra_entry.write_bytes(b"foreign") + extra_entry.chmod(0o600) + else: + extra_entry.mkdir(mode=0o700) + with pytest.raises(ArtifactConfigurationError, match="layout is incompatible"): + initialize_local_store(root=extra_entry_root) + + +def test_local_sanitizes_root_initialization_oserror( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + secret = str(tmp_path / "secret") + + def fail(_self: object, _root: Path) -> None: + raise OSError(secret) + + monkeypatch.setattr(LocalStorageAdapter, "_open_root", fail) + with pytest.raises(ArtifactConfigurationError) as raised: + LocalStorageAdapter(root=tmp_path / "artifacts") + assert secret not in str(raised.value) + assert raised.value.__cause__ is None + + +def test_local_rejects_symlink_and_nonprivate_roots(tmp_path: Path) -> None: + """Pin only a dedicated owner-private, no-follow root directory.""" + target = tmp_path / "target" + target.mkdir(mode=0o700) + linked = tmp_path / "linked" + linked.symlink_to(target, target_is_directory=True) + with pytest.raises(ArtifactConfigurationError, match="root is unsafe"): + LocalStorageAdapter(root=linked) + + public = tmp_path / "public" + public.mkdir(mode=0o755) + with pytest.raises(ArtifactConfigurationError, match="directory is unsafe"): + LocalStorageAdapter(root=public) + + +def test_local_rejects_root_and_files_not_owned_by_runtime_user( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Apply the canonical scratch ownership rule to every local descriptor.""" + root = tmp_path / "artifacts" + root.mkdir(mode=0o700) + regular = tmp_path / "object" + regular.write_bytes(b"bytes") + regular.chmod(0o600) + regular_descriptor = os.open(regular, os.O_RDONLY) + actual_euid = os.geteuid() + monkeypatch.setattr(os, "geteuid", lambda: actual_euid + 1) + try: + with pytest.raises(ArtifactConfigurationError, match="directory is unsafe"): + LocalStorageAdapter(root=root) + with pytest.raises(ArtifactIntegrityError, match="object is unsafe"): + LocalStorageAdapter._assert_private_regular(regular_descriptor) + with pytest.raises(ArtifactIntegrityError, match="object is unsafe"): + LocalStorageAdapter._assert_recoverable_object(regular_descriptor) + finally: + os.close(regular_descriptor) + + +@pytest.mark.parametrize("buffer_bytes", [True, 0, -1, 1024 * 1024 + 1, 1.5]) +def test_local_rejects_invalid_buffer_configuration( + tmp_path: Path, + buffer_bytes: object, +) -> None: + with pytest.raises(ValueError, match="artifact buffer"): + LocalStorageAdapter( + root=tmp_path / "artifacts", + buffer_bytes=buffer_bytes, # type: ignore[arg-type] + ) + + +@pytest.mark.parametrize("timeout", [True, 0, -1, float("inf"), "1"]) +def test_local_rejects_invalid_lock_timeout(tmp_path: Path, timeout: object) -> None: + with pytest.raises(ValueError, match="lock timeout"): + LocalStorageAdapter( + root=tmp_path / "artifacts", + lock_timeout_seconds=timeout, # type: ignore[arg-type] + ) + + +def test_local_requires_claim_and_rejects_same_path_replacement_before_mutation( + tmp_path: Path, +) -> None: + """Bind layout mutation to the root descriptor admitted by PostgreSQL.""" + root = tmp_path / "artifacts" + root.mkdir(mode=0o700) + bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=root)) + claim = local_namespace_claim(bootstrap) + + with pytest.raises(ArtifactConfigurationError, match="namespace is not initialized"): + asyncio.run(bootstrap._adapter.head("sha256/00/" + "0" * 62)) + + original = tmp_path / "original-artifacts" + root.rename(original) + root.mkdir(mode=0o700) + with pytest.raises(ArtifactConfigurationError, match="root changed"): + bootstrap.initialize_after_namespace_claim(claim) + + assert list(root.iterdir()) == [] + assert list(original.iterdir()) == [] + + +def test_local_rejects_nonmatching_namespace_claim_before_layout_mutation( + tmp_path: Path, +) -> None: + """Do not accept a proof for any fingerprint other than this root.""" + root = tmp_path / "artifacts" + root.mkdir(mode=0o700) + bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=root)) + claim = local_namespace_claim(bootstrap) + mismatched = ArtifactStoreNamespaceClaim( + adapter_identity=claim.adapter_identity, + namespace_identity=claim.namespace_identity, + namespace_fingerprint="sha256:" + "f" * 64, ) + with pytest.raises(ArtifactConfigurationError, match="claim does not match"): + bootstrap.initialize_after_namespace_claim(mismatched) + + assert list(root.iterdir()) == [] + assert bootstrap._adapter._root_fd == -1 + + +def test_local_closes_after_repeated_initialization_attempt(tmp_path: Path) -> None: + """A repeated bootstrap call fails closed instead of retaining descriptors.""" + root = tmp_path / "artifacts" + root.mkdir(mode=0o700) + bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=root)) + claim = local_namespace_claim(bootstrap) + bootstrap.initialize_after_namespace_claim(claim) + + with pytest.raises(ArtifactConfigurationError, match="already initialized"): + bootstrap.initialize_after_namespace_claim(claim) + + assert bootstrap._adapter._root_fd == -1 + + +def test_local_closes_pinned_descriptors_after_integrity_failure(tmp_path: Path) -> None: + """Release bootstrap ownership after any sanitized store initialization error.""" + root = tmp_path / "artifacts" + root.mkdir(mode=0o700) + marker = root / ".workstream-artifact-store-v2" + marker.write_bytes(b"workstream-artifact-store-v2\n") + marker.chmod(0o644) + bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=root)) + claim = local_namespace_claim(bootstrap) + + with pytest.raises(ArtifactIntegrityError, match="object is unsafe"): + bootstrap.initialize_after_namespace_claim(claim) + + assert bootstrap._adapter._root_fd == -1 + + +def test_local_reopens_one_valid_populated_layout(tmp_path: Path) -> None: + """Accept canonical immutable objects and persistent digest locks on restart.""" -def test_private_file_lock_rejects_invalid_durations() -> None: - """Reject invalid lock timing before touching a descriptor.""" - with pytest.raises(ValueError, match="durations are invalid"): - acquire_exclusive_file_lock(-1, timeout_seconds=0) + async def publish(root: Path) -> str: + adapter = initialize_local_store(root=root, buffer_bytes=2) + try: + async with minted_source(tmp_path / "scratch", b"restart-safe") as source: + return (await adapter.put(source)).provider_object_ref + finally: + adapter.close() + + root = tmp_path / "populated" + provider_object_ref = asyncio.run(publish(root)) + reopened = initialize_local_store(root=root) + try: + assert asyncio.run(reopened.head(provider_object_ref)).exists is True + finally: + reopened.close() + + +@pytest.mark.parametrize("mode", [0o600, 0o400]) +def test_local_startup_fails_closed_for_orphan_provider_temporary( + tmp_path: Path, + mode: int, +) -> None: + """Require explicit cleanup instead of deleting a possibly live provider temp.""" + root = tmp_path / f"orphan-{mode:o}" + initialized = initialize_local_store(root=root) + initialized.close() + orphan = root / "tmp" / (".put." + "a" * 32 + ".tmp") + orphan.write_bytes(b"unpublished") + orphan.chmod(mode) + + with pytest.raises( + ArtifactConfigurationError, + match="startup requires orphan temporary cleanup", + ): + initialize_local_store(root=root) + + assert orphan.read_bytes() == b"unpublished" @pytest.mark.asyncio -async def test_private_write_refactor_preserves_active_v1_behavior(tmp_path: Path) -> None: - """Keep store, replay, bounded chunks, and read behavior unchanged.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - request = store_request() - stored = await adapter.store(byte_stream(b"hel", b"lo"), request) - replay = await adapter.store(byte_stream(b"hello"), request) - assert stored.sha256 == request.expected_sha256 - assert stored.byte_count == 5 - assert replay.provider_artifact_id == stored.provider_artifact_id - assert replay.replayed is True - chunks = [chunk async for chunk in adapter.open(stored.provider_artifact_id)] - assert chunks == [b"he", b"ll", b"o"] - assert list((tmp_path / "artifacts" / "tmp").iterdir()) == [] - adapter.close() +async def test_v2_operations_reject_unsealed_or_malformed_values(tmp_path: Path) -> None: + adapter = initialize_local_store(root=tmp_path / "artifacts") + try: + with pytest.raises(ArtifactInputMismatchError, match="not sealed"): + await adapter.put(cast(CommittedArtifactSource, object())) + with pytest.raises(ArtifactOperationConflictError, match="commitment"): + await adapter.observe_put_result(cast(ArtifactCommitment, object())) + with pytest.raises(ArtifactOperationConflictError, match="byte range"): + adapter.open("sha256/00/" + "0" * 62, cast(ArtifactByteRange, object())) + with pytest.raises(ArtifactOperationConflictError, match="provider reference"): + await adapter.head(cast(str, 7)) + with pytest.raises(ArtifactOperationConflictError, match="commitment"): + adapter._acquire_lock("not-a-digest") + finally: + adapter.close() @pytest.mark.asyncio -async def test_private_io_finishes_after_repeated_cancellation(tmp_path: Path) -> None: - """Do not leave one blocking LocalStorage operation running unowned.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - started = threading.Event() - finish = threading.Event() - completed = threading.Event() +async def test_exact_replay_rejects_corrupt_existing_bytes_without_overwrite( + tmp_path: Path, +) -> None: + """Treat a pre-existing deterministic path as untrusted until fully hashed.""" + root = tmp_path / "artifacts" + adapter = initialize_local_store(root=root, buffer_bytes=2) + try: + async with minted_source(tmp_path / "scratch", b"expected") as source: + commitment = source.commitment + provider_object_ref = ( + f"sha256/{commitment.sha256[7:9]}/{commitment.sha256[9:]}" + ) + target = object_path(root, provider_object_ref) + target.parent.mkdir(mode=0o700) + target.write_bytes(b"corrupt!") + os.chmod(target, 0o400) + with pytest.raises(ArtifactIntegrityError, match="violates commitment"): + await adapter.put(source) + assert target.read_bytes() == b"corrupt!" + finally: + adapter.close() - def blocking_io() -> None: - started.set() - finish.wait(timeout=5) - completed.set() - - task = asyncio.create_task(adapter._run_io(blocking_io)) - assert await asyncio.to_thread(started.wait, 5) - task.cancel() - await asyncio.sleep(0) - task.cancel() - await asyncio.sleep(0) - finish.set() - with pytest.raises(asyncio.CancelledError): - await task - assert completed.is_set() - adapter.close() + +@pytest.mark.asyncio +@pytest.mark.parametrize("replacement", ["symlink", "directory", "public_file"]) +async def test_head_rejects_symlink_nonregular_and_nonprivate_objects( + tmp_path: Path, + replacement: str, +) -> None: + """Fail closed when private immutable object invariants are violated.""" + root = tmp_path / replacement + adapter = initialize_local_store(root=root, buffer_bytes=2) + try: + async with minted_source(tmp_path / f"scratch-{replacement}", b"object") as source: + result = await adapter.put(source) + target = object_path(root, result.provider_object_ref) + target.unlink() + if replacement == "symlink": + outside = tmp_path / "secret" + outside.write_bytes(b"do not read") + target.symlink_to(outside) + elif replacement == "directory": + target.mkdir(mode=0o700) + else: + target.write_bytes(b"object") + os.chmod(target, 0o644) + with pytest.raises(ArtifactIntegrityError): + await adapter.head(result.provider_object_ref) + finally: + adapter.close() @pytest.mark.asyncio -async def test_private_io_preserves_cancellation_after_late_failure(tmp_path: Path) -> None: - """Keep active-v1 cancellation semantics when blocking I/O later fails.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - started = threading.Event() - finish = threading.Event() +async def test_put_rejects_symlink_digest_directory(tmp_path: Path) -> None: + """Never follow an attacker-controlled digest-prefix directory.""" + root = tmp_path / "artifacts" + adapter = initialize_local_store(root=root, buffer_bytes=2) + try: + async with minted_source(tmp_path / "scratch", b"prefix") as source: + prefix = source.commitment.sha256[7:9] + outside = tmp_path / "outside" + outside.mkdir(mode=0o700) + (root / "objects" / "sha256" / prefix).symlink_to( + outside, + target_is_directory=True, + ) + with pytest.raises(ArtifactIntegrityError): + await adapter.put(source) + assert list(outside.iterdir()) == [] + finally: + adapter.close() - def failing_io() -> None: - started.set() - finish.wait(timeout=5) - raise OSError("late filesystem failure") - - task = asyncio.create_task(adapter._run_io(failing_io)) - assert await asyncio.to_thread(started.wait, 5) - task.cancel() - await asyncio.sleep(0) - task.cancel() - finish.set() - with pytest.raises(asyncio.CancelledError): - await task - adapter.close() + +@pytest.mark.asyncio +async def test_provider_failures_are_sanitized_without_paths_or_causes( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Do not retain raw provider path details in stable adapter failures.""" + root = tmp_path / "private-secret-root" + adapter = initialize_local_store(root=root) + provider_object_ref = "sha256/00/" + "0" * 62 + + def fail(_: str) -> int: + raise OSError(f"permission denied: {root}/credential-value") + + monkeypatch.setattr(adapter, "_head_object", fail) + try: + with pytest.raises(ArtifactStoreUnavailableError) as raised: + await adapter.head(provider_object_ref) + assert str(root) not in str(raised.value) + assert "credential-value" not in str(raised.value) + assert raised.value.__cause__ is None + finally: + adapter.close() @pytest.mark.asyncio -async def test_private_lock_releases_after_repeated_cancellation( +async def test_put_and_open_map_raw_oserrors_to_sanitized_failures( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - """Release a thread-acquired LocalStorage lock before propagating cancellation.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - acquired = threading.Event() - finish_acquire = threading.Event() - release_started = threading.Event() - finish_release = threading.Event() - released = threading.Event() - original_acquire = adapter._acquire_lock - original_release = adapter._release_lock - - def delayed_acquire(scope: str) -> tuple[object, int]: - lock = original_acquire(scope) - acquired.set() - finish_acquire.wait(timeout=5) - return lock - - def delayed_release(lock: tuple[object, int]) -> None: - release_started.set() - finish_release.wait(timeout=5) - original_release(lock) - released.set() - - monkeypatch.setattr(adapter, "_acquire_lock", delayed_acquire) - monkeypatch.setattr(adapter, "_release_lock", delayed_release) - task = asyncio.create_task(adapter._acquire_lock_async("repeated-cancellation")) - assert await asyncio.to_thread(acquired.wait, 5) - task.cancel() - finish_acquire.set() - assert await asyncio.to_thread(release_started.wait, 5) - task.cancel() - await asyncio.sleep(0) - finish_release.set() - with pytest.raises(asyncio.CancelledError): - await task - assert released.is_set() - adapter.close() + adapter = initialize_local_store(root=tmp_path / "artifacts") + + async def fail_lock(_digest: str) -> tuple[object, int]: + raise OSError("secret lock path") + + monkeypatch.setattr(adapter, "_acquire_lock_async", fail_lock) + try: + async with minted_source(tmp_path / "scratch", b"put") as source: + with pytest.raises(ArtifactStoreUnavailableError) as put_error: + await adapter.put(source) + assert put_error.value.__cause__ is None + + async def fail_open(_provider_ref: str) -> tuple[int, int]: + raise OSError("secret object path") + + monkeypatch.setattr(adapter, "_open_object_async", fail_open) + with pytest.raises(ArtifactStoreUnavailableError) as open_error: + _ = [ + chunk + async for chunk in adapter.open("sha256/00/" + "0" * 62) + ] + assert open_error.value.__cause__ is None + finally: + adapter.close() @pytest.mark.asyncio -async def test_private_nested_locks_all_release_before_cancellation( +@pytest.mark.parametrize( + "stream_values", + [("not-bytes",), (b"too-long",), (b"changed",)], +) +async def test_put_defends_against_broken_sealed_source_streams( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, + stream_values: tuple[object, ...], ) -> None: - """Release outer and inner operation locks before cancellation escapes.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - first = await adapter._acquire_lock_async("nested-first") - second = await adapter._acquire_lock_async("nested-second") - release_started = threading.Event() - finish_release = threading.Event() - released: list[int] = [] - original_release = adapter._release_lock - - def delayed_first_release(lock: tuple[object, int]) -> None: - if lock is first: - release_started.set() - finish_release.wait(timeout=5) - original_release(lock) - released.append(lock[1]) - - monkeypatch.setattr(adapter, "_release_lock", delayed_first_release) - release_task = asyncio.create_task(adapter._release_locks(first, second)) - assert await asyncio.to_thread(release_started.wait, 5) - release_task.cancel() - await asyncio.sleep(0) - release_task.cancel() - finish_release.set() - with pytest.raises(asyncio.CancelledError): - await release_task - assert released == [first[1], second[1]] - - monkeypatch.setattr(adapter, "_release_lock", original_release) - reacquired_first = await adapter._acquire_lock_async("nested-first") - reacquired_second = await adapter._acquire_lock_async("nested-second") - await adapter._release_locks(reacquired_second, reacquired_first) - adapter.close() + adapter = initialize_local_store(root=tmp_path / "artifacts", buffer_bytes=2) + try: + async with minted_source(tmp_path / "scratch", b"content") as source: + async def broken_stream(_self: CommittedArtifactSource) -> AsyncIterator[object]: + for value in stream_values: + yield value + + monkeypatch.setattr(CommittedArtifactSource, "stream", broken_stream) + with pytest.raises(ArtifactInputMismatchError): + await adapter.put(source) + finally: + adapter.close() @pytest.mark.asyncio -async def test_private_lock_acquisition_has_a_bounded_deadline(tmp_path: Path) -> None: - """Fail locally instead of hanging when another operation owns the lock.""" - adapter = LocalStorageAdapter( - root=tmp_path / "artifacts", - buffer_bytes=2, - lock_timeout_seconds=0.05, - ) - held_lock = adapter._acquire_lock("bounded-lock") - started = asyncio.get_running_loop().time() +async def test_put_handles_exclusive_publication_race_as_exact_replay( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + adapter = initialize_local_store(root=tmp_path / "artifacts", buffer_bytes=2) + original_publish = adapter._publish_exclusive + + def publish_then_report_race(temporary_name: str, provider_ref: str) -> bool: + assert original_publish(temporary_name, provider_ref) is True + return False + + monkeypatch.setattr(adapter, "_publish_exclusive", publish_then_report_race) try: - with pytest.raises(ArtifactStoreUnavailableError, match="lock deadline"): - await adapter._acquire_lock_async("bounded-lock") + async with minted_source(tmp_path / "scratch", b"race") as source: + result = await adapter.put(source) + assert result.replayed is True finally: - adapter._release_lock(held_lock) - assert asyncio.get_running_loop().time() - started < 1 - adapter.close() + adapter.close() @pytest.mark.asyncio -async def test_cancelled_private_lock_wait_preserves_cancellation_after_timeout( +async def test_head_recovers_crash_after_link_before_temporary_unlink( tmp_path: Path, ) -> None: - """Keep caller cancellation when its background lock wait times out.""" - adapter = LocalStorageAdapter( - root=tmp_path / "artifacts", - buffer_bytes=2, - lock_timeout_seconds=0.05, - ) - held_lock = adapter._acquire_lock("cancelled-timeout") - waiting = asyncio.create_task(adapter._acquire_lock_async("cancelled-timeout")) - await asyncio.sleep(0.01) - waiting.cancel() + """Repair the only valid two-link state while owning the digest lock.""" + root = tmp_path / "artifacts" + adapter = initialize_local_store(root=root, buffer_bytes=2) try: - with pytest.raises(asyncio.CancelledError): - await waiting + async with minted_source(tmp_path / "scratch", b"crash recovery") as source: + commitment = source.commitment + provider_ref = adapter._provider_object_ref(commitment) + descriptor, temporary_name = adapter._create_temporary() + try: + adapter._write_all(descriptor, memoryview(b"crash recovery")) + adapter._seal_temporary(descriptor) + prefix, filename = adapter._parse_provider_object_ref(provider_ref) + prefix_fd = adapter._open_prefix(prefix, create=True) + try: + os.link( + temporary_name, + filename, + src_dir_fd=adapter._tmp_fd, + dst_dir_fd=prefix_fd, + follow_symlinks=False, + ) + os.fsync(prefix_fd) + finally: + os.close(prefix_fd) + finally: + os.close(descriptor) + + target = object_path(root, provider_ref) + assert target.stat().st_nlink == 2 + assert [path.name for path in (root / "tmp").iterdir()] == [temporary_name] + + observed = await adapter.head(provider_ref) + + assert observed.exists is True + assert observed.byte_count == len(b"crash recovery") + assert target.stat().st_nlink == 1 + assert list((root / "tmp").iterdir()) == [] + assert b"".join([chunk async for chunk in adapter.open(provider_ref)]) == b"crash recovery" finally: - adapter._release_lock(held_lock) + adapter.close() - reacquired = await adapter._acquire_lock_async("cancelled-timeout") - await adapter._release_locks(reacquired) - adapter.close() + +@pytest.mark.asyncio +async def test_head_rejects_unowned_second_hard_link(tmp_path: Path) -> None: + """Never treat an arbitrary external hard link as interrupted publication.""" + root = tmp_path / "artifacts" + adapter = initialize_local_store(root=root) + try: + async with minted_source(tmp_path / "scratch", b"linked") as source: + result = await adapter.put(source) + target = object_path(root, result.provider_object_ref) + outside_link = tmp_path / "unowned-link" + os.link(target, outside_link) + + with pytest.raises(ArtifactIntegrityError, match="ambiguous"): + await adapter.head(result.provider_object_ref) + finally: + adapter.close() @pytest.mark.asyncio -async def test_store_cleanup_survives_repeated_cancellation( +async def test_new_digest_prefix_is_fsynced_before_publication( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - """Preserve cancellation and retry ownership when temporary cleanup fails.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - request = store_request() - write_started = asyncio.Event() - never = asyncio.Event() - cleanup_started = asyncio.Event() - finish_cleanup = asyncio.Event() - temporary: Path | None = None - original_write_stream = adapter._write_stream - original_cleanup = adapter._cleanup_unpublished - - async def blocked_write( - _: object, - provider_id: str, - __: object, - ) -> tuple[str, int]: - nonlocal temporary - temporary = tmp_path / "artifacts" / "tmp" / f"{provider_id}_cancelled.part" - temporary.write_bytes(b"partial") - write_started.set() - await never.wait() - raise AssertionError("blocked write unexpectedly resumed") - - async def delayed_cleanup(scope: str) -> None: - cleanup_started.set() - await finish_cleanup.wait() - raise OSError(f"injected cleanup failure for {scope}") - - monkeypatch.setattr(adapter, "_write_stream", blocked_write) - monkeypatch.setattr(adapter, "_cleanup_unpublished", delayed_cleanup) - task = asyncio.create_task(adapter.store(byte_stream(b"hello"), request)) - await asyncio.wait_for(write_started.wait(), timeout=1) - task.cancel("caller cancelled artifact storage") - await asyncio.wait_for(cleanup_started.wait(), timeout=1) - task.cancel() - await asyncio.sleep(0) - finish_cleanup.set() - with pytest.raises( - asyncio.CancelledError, - match="caller cancelled artifact storage", - ) as cancellation: - await task - assert cancellation.value.args == ("caller cancelled artifact storage",) - - assert temporary is not None - assert temporary.is_file() - scope = adapter._scope_key(request.idempotency) - await original_cleanup(scope) - assert not temporary.exists() - monkeypatch.setattr(adapter, "_write_stream", original_write_stream) - monkeypatch.setattr(adapter, "_cleanup_unpublished", original_cleanup) - stored = await adapter.store(byte_stream(b"hello"), request) - assert stored.sha256 == request.expected_sha256 - adapter.close() + """Persist the new digest-directory entry before acknowledging publication.""" + adapter = initialize_local_store(root=tmp_path / "artifacts") + fsynced: list[int] = [] + original_fsync = os.fsync + + def record_fsync(descriptor: int) -> None: + fsynced.append(descriptor) + original_fsync(descriptor) + + monkeypatch.setattr(os, "fsync", record_fsync) + try: + async with minted_source(tmp_path / "scratch", b"durable prefix") as source: + await adapter.put(source) + assert adapter._sha256_fd in fsynced + finally: + adapter.close() @pytest.mark.asyncio -async def test_store_preserves_cancellation_during_final_lock_release( +async def test_lost_prefix_creation_race_fsyncs_parent_before_link( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - """Finish final lock release without converting cancellation into success.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - request = store_request() - release_started = threading.Event() - finish_release = threading.Event() - released = threading.Event() - original_release = adapter._release_lock - - def delayed_release(lock: tuple[object, int]) -> None: - release_started.set() - finish_release.wait(timeout=5) - original_release(lock) - released.set() - - monkeypatch.setattr(adapter, "_release_lock", delayed_release) - task = asyncio.create_task(adapter.store(byte_stream(b"hello"), request)) - assert await asyncio.to_thread(release_started.wait, 5) - task.cancel() - await asyncio.sleep(0) - task.cancel() - finish_release.set() - with pytest.raises(asyncio.CancelledError): - await task - assert released.is_set() - - monkeypatch.setattr(adapter, "_release_lock", original_release) - replay = await adapter.store(byte_stream(b"hello"), request) - assert replay.replayed - adapter.close() + """Durably observe a concurrently created prefix before publishing into it.""" + adapter = initialize_local_store(root=tmp_path / "artifacts") + original_mkdir = os.mkdir + original_fsync = os.fsync + original_link = os.link + events: list[str] = [] + injected_race = False + + def race_mkdir( + path: str, + mode: int = 0o777, + *, + dir_fd: int | None = None, + ) -> None: + nonlocal injected_race + if dir_fd == adapter._sha256_fd and not injected_race: + original_mkdir(path, mode, dir_fd=dir_fd) + injected_race = True + raise FileExistsError + original_mkdir(path, mode, dir_fd=dir_fd) + + def record_fsync(descriptor: int) -> None: + if descriptor == adapter._sha256_fd: + events.append("parent_fsync") + original_fsync(descriptor) + + def record_link(*args: object, **kwargs: object) -> None: + events.append("link") + original_link(*args, **kwargs) + + monkeypatch.setattr(os, "mkdir", race_mkdir) + monkeypatch.setattr(os, "fsync", record_fsync) + monkeypatch.setattr(os, "link", record_link) + try: + async with minted_source(tmp_path / "scratch", b"prefix race") as source: + await adapter.put(source) + assert injected_race is True + assert events.index("parent_fsync") < events.index("link") + finally: + adapter.close() + + +@pytest.mark.asyncio +async def test_invalid_provider_refs_fail_before_filesystem_io( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Reject traversal and v1 opaque IDs without touching provider storage.""" + adapter = initialize_local_store(root=tmp_path / "artifacts") + called = False + + def unexpected(_: str) -> int: + nonlocal called + called = True + raise AssertionError + + monkeypatch.setattr(adapter, "_head_object", unexpected) + try: + for provider_object_ref in ("../secret", "art_" + "0" * 32, "sha256/AA/x"): + with pytest.raises(ArtifactOperationConflictError): + await adapter.head(provider_object_ref) + assert called is False + finally: + adapter.close() @pytest.mark.asyncio -async def test_object_only_recovery_still_requires_exact_replay_without_commitment( +async def test_put_cancellation_finishes_write_cleanup_and_lock_release( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - """Preserve active-v1 exact replay when no client commitment exists.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - committed_request = store_request() - request = replace(committed_request, expected_sha256=None, expected_size=None) - request = replace( - request, - idempotency=replace( - request.idempotency, - request_digest=canonical_store_request_digest(request), - ), + """Preserve caller cancellation after off-loop I/O and owned cleanup finish.""" + root = tmp_path / "artifacts" + adapter = initialize_local_store(root=root, buffer_bytes=2) + started = threading.Event() + finish = threading.Event() + original_write = adapter._write_all + + def blocked_write(descriptor: int, chunk: memoryview) -> None: + started.set() + finish.wait(timeout=5) + original_write(descriptor, chunk) + + monkeypatch.setattr(adapter, "_write_all", blocked_write) + try: + async with minted_source(tmp_path / "scratch", b"cancel") as source: + digest_hex = source.commitment.sha256[7:] + task = asyncio.create_task(adapter.put(source)) + assert await asyncio.to_thread(started.wait, 5) + task.cancel("caller cancelled local put") + await asyncio.sleep(0) + task.cancel() + finish.set() + with pytest.raises(asyncio.CancelledError, match="caller cancelled local put"): + await task + assert list((root / "tmp").iterdir()) == [] + lock = await adapter._acquire_lock_async(digest_hex) + await adapter._run_io(adapter._release_lock, lock) + finally: + adapter.close() + + +@pytest.mark.asyncio +async def test_lock_wait_has_a_bounded_deadline(tmp_path: Path) -> None: + """Fail retryably instead of blocking forever behind another writer.""" + adapter = initialize_local_store( + root=tmp_path / "artifacts", + lock_timeout_seconds=0.05, ) - original_write_json = adapter._write_json_exclusive - interrupted = False - - async def cancel_before_metadata(path: Path, value: dict[str, object]) -> None: - nonlocal interrupted - if path.parent.name == "metadata" and not interrupted: - interrupted = True - raise asyncio.CancelledError - await original_write_json(path, value) - - monkeypatch.setattr(adapter, "_write_json_exclusive", cancel_before_metadata) - with pytest.raises(asyncio.CancelledError): - await adapter.store(byte_stream(b"hello"), request) - assert interrupted - - monkeypatch.setattr(adapter, "_write_json_exclusive", original_write_json) - with pytest.raises(ArtifactIntegrityError, match="recovery commitment failed"): - await adapter.store(byte_stream(b"wrong"), request) - assert list((tmp_path / "artifacts" / "objects").iterdir()) == [] - assert len(list((tmp_path / "artifacts" / "quarantine").iterdir())) == 1 - adapter.close() + digest_hex = "0" * 64 + held = adapter._acquire_lock(digest_hex) + try: + with pytest.raises(ArtifactStoreUnavailableError, match="lock deadline"): + await adapter._acquire_lock_async(digest_hex) + finally: + adapter._release_lock(held) + adapter.close() @pytest.mark.asyncio -async def test_failed_write_unlinks_temporary_after_close_failure( +async def test_all_operation_file_io_runs_off_the_event_loop( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: - """Do not leave active-v1 temporary bytes after a late close error.""" - adapter = LocalStorageAdapter(root=tmp_path / "artifacts", buffer_bytes=2) - request = store_request() - original_open = adapter._open_exclusive - original_close = os.close - opened_descriptor: int | None = None - close_failed = False - - def capture_open(path: Path) -> int: - nonlocal opened_descriptor - opened_descriptor = original_open(path) - return opened_descriptor - - async def fail_write(*_: object) -> tuple[str, int]: - raise OSError(errno.EIO, "injected artifact write failure") - - def close_then_fail(descriptor: int) -> None: - nonlocal close_failed - if descriptor == opened_descriptor and not close_failed: - original_close(descriptor) - close_failed = True - raise OSError(errno.EIO, "injected temporary close failure") - original_close(descriptor) - - monkeypatch.setattr(adapter, "_open_exclusive", capture_open) - monkeypatch.setattr(adapter, "_write_bounded_stream_to_private_file", fail_write) - monkeypatch.setattr(os, "close", close_then_fail) - with pytest.raises(ArtifactStoreUnavailableError, match="local artifact operation failed"): - await adapter.store(byte_stream(b"hello"), request) - assert close_failed - assert list((tmp_path / "artifacts" / "tmp").iterdir()) == [] - adapter.close() + """Keep object writes, publication, head, and reads off the event-loop thread.""" + adapter = initialize_local_store(root=tmp_path / "artifacts", buffer_bytes=2) + event_loop_thread = threading.get_ident() + observed_threads: list[int] = [] + + def track(name: str) -> None: + original = getattr(adapter, name) + + def wrapped(*args: object) -> object: + observed_threads.append(threading.get_ident()) + return original(*args) + + monkeypatch.setattr(adapter, name, wrapped) + + for method_name in ("_write_all", "_publish_exclusive", "_head_object", "_open_object"): + track(method_name) + try: + async with minted_source(tmp_path / "scratch", b"bounded io") as source: + result = await adapter.put(source) + await adapter.head(result.provider_object_ref) + assert b"".join( + [chunk async for chunk in adapter.open(result.provider_object_ref)] + ) == b"bounded io" + assert observed_threads + assert event_loop_thread not in observed_threads + finally: + adapter.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("read_result", [b"", b"oversized"]) +async def test_open_rejects_truncated_or_oversized_provider_reads( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + read_result: bytes, +) -> None: + adapter = initialize_local_store(root=tmp_path / "artifacts", buffer_bytes=2) + try: + async with minted_source(tmp_path / "scratch", b"content") as source: + result = await adapter.put(source) + monkeypatch.setattr(os, "pread", lambda *_args: read_result) + with pytest.raises(ArtifactIntegrityError): + _ = b"".join( + [chunk async for chunk in adapter.open(result.provider_object_ref)] + ) + finally: + adapter.close() + + +def test_short_write_and_idempotent_cleanup_fail_safely( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + adapter = initialize_local_store(root=tmp_path / "artifacts") + try: + monkeypatch.setattr(os, "write", lambda *_args: 0) + with pytest.raises(OSError, match="short artifact write"): + adapter._write_all(adapter._tmp_fd, memoryview(b"x")) + adapter._cleanup_resources(-1, ".put." + "0" * 32 + ".tmp", None) + finally: + adapter.close() + + +def test_cleanup_attempts_lock_release_after_temporary_unlink_failure( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Contain cleanup errors without leaking the owned digest lock.""" + adapter = initialize_local_store(root=tmp_path / "artifacts") + released: list[tuple[object, int]] = [] + lock = (object(), -1) + + def fail_unlink(*_args: object, **_kwargs: object) -> None: + raise OSError("private temporary path") + + monkeypatch.setattr(os, "unlink", fail_unlink) + monkeypatch.setattr(adapter, "_release_lock", released.append) + try: + adapter._cleanup_resources(None, ".put." + "0" * 32 + ".tmp", lock) + finally: + adapter.close() + + assert released == [lock] diff --git a/backend/tests/test_projects.py b/backend/tests/test_projects.py index 46db6b587..148d6a294 100644 --- a/backend/tests/test_projects.py +++ b/backend/tests/test_projects.py @@ -71,6 +71,7 @@ PolicySetupBlocked, ProjectSetupQueueError, ProjectService, + ProjectServiceError, StaleProjectSetupContinuation, ) @@ -2955,6 +2956,100 @@ def capture_error(message: str, *, extra: dict[str, object]) -> None: assert "/srv/private" not in logged_payload +async def test_project_setup_worker_persists_sanitized_domain_failure( + project_client: AsyncClient, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Known setup failures become a sanitized terminal setup-run result.""" + from app.workers import project_setup as project_setup_worker_module + + project = await create_project(project_client) + guide = await create_guide( + project_client, + project["id"], + { + **complete_guide_payload(), + "source_snapshot": source_snapshot_payload(), + }, + ) + async with db_session.get_session_factory()() as session: + snapshot = await session.scalar( + select(GuideSourceSnapshot).where(GuideSourceSnapshot.guide_id == guide["id"]) + ) + assert snapshot is not None + setup_run = ProjectSetupRun( + id=str(uuid4()), + project_id=project["id"], + guide_id=guide["id"], + guide_version=guide["version"], + source_snapshot_id=snapshot.id, + source_snapshot_hash=snapshot.bundle_hash, + status="queued", + current_step="queued", + created_by="test-project-manager", + ) + session.add(setup_run) + await session.commit() + setup_run_id = setup_run.id + snapshot_id = snapshot.id + + async def raise_domain_error(*_: object, **__: object) -> object: + raise ProjectServiceError( + "guide source unavailable at https://storage.flow.test/signed?token=secret" + ) + + monkeypatch.setattr( + project_setup_worker_module.ProjectService, + "run_guide_sufficiency_agent", + raise_domain_error, + ) + warning_logs: list[dict[str, object]] = [] + + def capture_warning(message: str, *, extra: dict[str, object]) -> None: + warning_logs.append({"message": message, "extra": extra}) + + monkeypatch.setattr(project_setup_worker_module.logger, "warning", capture_warning) + + result = await project_setup_worker_module._run_pre_submit_setup_pipeline( + project["id"], + guide["id"], + snapshot_id, + setup_run_id, + ) + + async with db_session.get_session_factory()() as session: + persisted = await session.get(ProjectSetupRun, setup_run_id) + + public_error = "project setup failed; inspect server logs with the setup run id" + assert result == { + "status": "setup_blocked", + "error": public_error, + "guide_sufficiency_report_id": None, + "submission_artifact_policy_id": None, + } + assert persisted is not None + assert persisted.status == "setup_blocked" + assert persisted.current_step == "project_setup" + assert persisted.error_code == "ProjectServiceError" + assert persisted.error_summary == public_error + assert warning_logs == [ + { + "message": "project setup pipeline stopped", + "extra": { + "project_id": project["id"], + "guide_id": guide["id"], + "source_snapshot_id": snapshot_id, + "setup_run_id": setup_run_id, + "error_code": "ProjectServiceError", + "error_summary": public_error, + }, + } + ] + serialized = json.dumps({"result": result, "logs": warning_logs}, sort_keys=True) + assert "token=secret" not in serialized + assert "https://" not in serialized + + async def test_project_setup_run_rejects_cross_context_worker_updates( project_client: AsyncClient, monkeypatch: pytest.MonkeyPatch, diff --git a/docs/decision_0013_immutable_artifact_storage_boundary.md b/docs/decision_0013_immutable_artifact_storage_boundary.md index 007f0b98c..354e70fe4 100644 --- a/docs/decision_0013_immutable_artifact_storage_boundary.md +++ b/docs/decision_0013_immutable_artifact_storage_boundary.md @@ -206,11 +206,13 @@ contribution, compensation-award, or reputation effect. ## Shared Adapter Convention Artifact storage follows ADR 0014. `LocalStorageAdapter` and -`S3CompatibleArtifactStore` are explicitly registered through -`ExternalServiceAdapterFactory[ArtifactStore]`. Only artifact-storage -orchestration receives the writable port through composition-root dependency -injection. Product modules and Celery jobs receive typed artifact -ingest/read/materialization operations instead. +`S3CompatibleArtifactStore` are reached through explicitly registered, +non-mutating `ArtifactStoreBootstrap` implementations in +`ExternalServiceAdapterFactory[ArtifactStoreBootstrap]`. The composition root +claims the bootstrap's exact namespace in PostgreSQL before initialization +yields the byte-only `ArtifactStore`. Only artifact-storage orchestration +receives that writable port. Product modules and Celery jobs receive typed +artifact ingest/read/materialization operations instead. There is no service locator, runtime plugin discovery, concrete-adapter import in product services, fallback constructor, compatibility alias, or dual factory @@ -220,12 +222,14 @@ path. The existing ArtifactStore v1 provider `verify`, `retain`, `release`, and receipt methods are removed in the same chunk that migrates LocalStorage and -all active callers. The dormant `flow_node` backend setting is removed. The new +all active callers. The dormant Flow Node backend setting is removed. The new backend values are exactly `disabled`, `local`, and `s3_compatible`. -Pre-production rows may be rebuilt when obsolete URI/hash or provider-retention -contracts cannot be derived from authoritative stored bytes. No alias, fake -verified backfill, nullable shadow field, dual write, or fallback remains. +Migration `0025` refuses populated v1 artifact tables before DDL and preserves +their prior schema and rows. An empty pre-production environment may be +reprovisioned out of band and authoritative bytes reingested through v2; the +migration performs no automated rebuild or fabricated backfill. No alias, +nullable shadow field, dual write, or fallback remains. ## Review Boundary diff --git a/docs/glossary.md b/docs/glossary.md index 28d0625ef..89110d247 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -84,7 +84,7 @@ protocol observations are replica details, not this record's identity. The per-item mutable upload-operation ledger inside an `ArtifactUploadSession`. It owns byte reservation, logical role, scoped -idempotency, request digest, CAS state, provider operation reference, and the +idempotency, request digest, CAS state, opaque `provider_object_ref`, and the resulting `ArtifactContent`. It is not an `ArtifactBinding`. ## ArtifactBinding @@ -96,15 +96,19 @@ records Workstream meaning and provenance, not storage-provider state. ## ArtifactReplica -One provider copy of `ArtifactContent`, identified by opaque provider artifact -and optional bounded protocol observations. Verification, availability, and -integrity states belong here and do not create task or review states. Logical -Workstream references are represented only by `ArtifactBinding`. +One provider copy of `ArtifactContent`, identified by an opaque +`provider_object_ref` and optional bounded protocol observations. Verification, +availability, and integrity states belong here and do not create task or review +states. Logical Workstream references are represented only by +`ArtifactBinding`. ## ArtifactOperationReceipt -An append-only record of an idempotent storage-provider operation, canonical -request/response digests, provider reference, bounded outcome, and timestamps. +Append-only Workstream evidence for one immutable put acknowledgement. It links +the exact upload item and replica and records operation, idempotency key, +`request_digest`, opaque `provider_object_ref`, replay observation, bounded +outcome/details, attempt number, correlation ID, and creation time. It contains +no response digest or provider receipt. ## ArtifactSetManifest @@ -271,12 +275,24 @@ Proof supporting task completion or review decision. Examples: logs, hashes, tes ## Artifact Store The provider-neutral typed capability through which Workstream stores and reads -private immutable bytes. `LocalStorageAdapter` implements it for development -and focused tests. `S3CompatibleArtifactStore` implements it for MinIO -integration and AWS S3 v0.1 production deployments. Providers do +private immutable bytes. Its v0.1 byte-only operations are `put`, read-only +`observe_put_result`, `open`, and `head`. `LocalStorageAdapter` implements it +for development and focused tests. `S3CompatibleArtifactStore` implements it +for MinIO integration and AWS S3 v0.1 production deployments. Providers do not own Workstream authorization, binding, lifecycle, audit, or integrity decisions. +## Artifact Storage Namespace + +The immutable deployment-level PostgreSQL fence that binds Workstream to one +configured artifact backend, adapter, provider profile, and non-secret storage +namespace fingerprint. Startup and every provider operation must validate the +same singleton before provider I/O. Changing a populated deployment requires a +separately reviewed storage migration. +For LocalStorage, the pre-provisioned private root's normalized path and +filesystem identity are hashed into that fingerprint, so replacing the root at +the same path fails closed before adapter construction mutates it. + ## Artifact Verification Job A durable Celery request to read and hash a complete stored object before its diff --git a/docs/operations_authorization_service.md b/docs/operations_authorization_service.md index 8b01f4bb9..9c17a49ea 100644 --- a/docs/operations_authorization_service.md +++ b/docs/operations_authorization_service.md @@ -642,7 +642,7 @@ composition. AUTH-10 is a clean cut to independent `submitter`, `reviewer`, and `adjudicator` grants. Before rollout, scan current typed schemas, audit facts, idempotency records, and PostgreSQL validators for `both`, replacement fields, -replacement events, and replacement reasons. Migration `0025` must stop on any +replacement events, and replacement reasons. Migration `0026` must stop on any incompatible evidence; operators must remediate through a separately approved data decision, never an automatic conversion. A safe downgrade also refuses rather than deleting adjudicator or new exact-role evidence. diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 6c59a016d..7d9bd4228 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -15,8 +15,9 @@ or Flow Node. ```text FastAPI and Celery composition roots --> ExternalServiceAdapterFactory[ArtifactStore] --> ArtifactStore v2 +-> ExternalServiceAdapterFactory[ArtifactStoreBootstrap] +-> PostgreSQL exact namespace claim +-> initialized ArtifactStore v2 -> LocalStorageAdapter -> S3CompatibleArtifactStore -> MinIO integration @@ -203,8 +204,20 @@ MinIO, and AWS S3, including region, bucket or private-root identity, normalized endpoint identity where applicable, private prefix, and addressing style; it contains no credential or resolved secret. -Startup and every provider operation transactionally insert-or-validate this -singleton before I/O. A concurrent first writer claims the namespace; a loser +For LocalStorage, the durable root must already exist with owner-private +permissions before application startup. Its descriptor stores only a hash over +the normalized path plus filesystem device/inode identity. Replacing or +remounting that root at the same configured path therefore changes the +fingerprint. The typed factory constructs a non-mutating bootstrap that pins +the root descriptor. PostgreSQL admits that exact namespace identity, then the +bootstrap rechecks that the configured path still names the pinned descriptor +before it creates layout files through that descriptor. Replacement before +initialization fails without mutating either root; replacement after the final +check cannot redirect descriptor-relative writes into the replacement root. + +Startup inspects and pins only the configured namespace identity before the +transaction. It and every provider operation transactionally insert-or-validate +the singleton before layout mutation or artifact-byte I/O. A concurrent first writer claims the namespace; a loser with a different fingerprint fails before provider access. Replicas and put attempts reference the namespace row and fingerprint. A populated deployment cannot change it without a separately reviewed maintenance migration. @@ -243,11 +256,13 @@ preserves history. ### ArtifactOperationReceipt -Append-only Workstream evidence for one operation identity: adapter, -operation, idempotency key, request digest, bounded provider observation, -outcome, correlation/attempt references, and provider/database timestamps. -Credentials, signed headers, endpoints, raw provider responses, and product -secrets are forbidden. +Append-only Workstream evidence for one immutable put acknowledgement. It links +the exact upload item and replica and records operation, idempotency key, +`request_digest`, opaque `provider_object_ref`, replay observation, bounded +outcome/details, attempt number, correlation ID, and `created_at`. Adapter and +namespace identity resolve through the linked replica. A response digest, +provider receipt, credentials, signed headers, endpoints, raw provider +responses, and product secrets are forbidden. ### ArtifactVerificationJob @@ -333,7 +348,8 @@ ArtifactBindingPort.bind_verified(ArtifactBindingCreateRequest) ArtifactMaterializationPort.materialize_ready_upload_set(ReadyUploadSetRequest) ArtifactMaterializationPort.materialize_bindings(BindingMaterializationRequest) CheckerArtifactOutputPort.store(CheckerOutputArtifactRequest) -ArtifactOperatorReadPort.list/get/admission_usage(...) +ArtifactOperatorReadPort.list_bindings/list_replicas/list_receipts/ +get_verification_job/get_recovery_attempt/list_audit_events/admission_usage(...) ArtifactOperatorRecoveryPort.retry_verification(ArtifactRecoveryRequest) ``` @@ -353,16 +369,28 @@ admission-usage request contains only bounded scope filters and pagination. It returns current reserved/completed byte usage and configured limits; it cannot release charges or mutate capacity. +The binding lookup resource vocabulary is exactly `project`, `project_guide`, +`guide_source_snapshot`, `guide_source_snapshot_item`, `task`, `submission`, +`checker_run`, or `review`. The audit resource vocabulary is exactly +`artifact_binding`, `artifact_content`, `artifact_replica`, `artifact_receipt`, +`artifact_verification_job`, or `artifact_recovery_attempt`. Adding a product or +artifact resource requires an explicit contract change; callers cannot submit +an unregistered string. + None accepts an adapter, provider object reference, storage namespace, caller- -assembled quota scopes, arbitrary resource type, or caller-selected server -content ID. The implementation resolves canonical relationships, authorization +assembled quota scopes, a resource type outside the closed binding/audit +vocabularies, or caller-selected server content ID. The implementation resolves +canonical relationships, authorization resources, producer identity, and admission scopes inside artifact orchestration. The internal orchestrator is not itself a product capability. Operator routes receive the read or recovery capability they require, never the raw store or broad orchestrator. -FastAPI and Celery composition roots construct the selected `ArtifactStore`, -then construct one internal orchestrator, then expose only the required narrow +FastAPI and Celery composition roots construct the selected non-mutating +`ArtifactStoreBootstrap` through +`ExternalServiceAdapterFactory[ArtifactStoreBootstrap]`, claim its exact +namespace in PostgreSQL, and only then initialize the byte-only `ArtifactStore`. +They construct one internal orchestrator and expose only the required narrow port to each route/service/task. Architecture tests parse imports, constructor annotations, dependency providers, and Celery task parameters; checking only textual `put` call sites is insufficient. @@ -617,9 +645,26 @@ and the AWS live readiness profile. ## LocalStorage Adapter LocalStorage implements the same v2 port and conformance vectors. It uses a -private configured root, opaque content-derived paths, exclusive no-overwrite -publication, bounded off-event-loop file I/O, no-follow path handling, private -permissions, full read verification, and sanitized failures. +pre-provisioned private configured root, opaque content-derived paths, +exclusive no-overwrite publication, bounded off-event-loop file I/O, no-follow +path handling, private permissions, full read verification, and sanitized +failures. + +Startup accepts only an empty provider-temporary directory or the exact sealed +two-link state left after immutable publication and before temporary unlink. A +single-link writable or sealed provider temporary blocks startup with an +operator-cleanup error; initialization does not delete it because another live +process may still own that file. + +Local development recovery requires a maintenance window in which every API +and Celery process using that local root is stopped. The Operator may remove or +move a file from `tmp/` to an owner-private quarantine directory outside the +artifact root only after verifying that its name matches the canonical +`.put.<32 lowercase hex>.tmp` shape, it is a regular file owned by the service +account, it has exactly one link, and its mode is `0600` or `0400`. Workstream +must then restart and revalidate the complete layout. Any different name, +owner, type, link count, or mode is an integrity incident and must not be +repaired by this procedure. The v1 local provider metadata for retain/release/provider receipts is removed in the v2 clean cut. No compatibility adapter or dual format remains. @@ -712,6 +757,7 @@ The preparation settings use the standard `WORKSTREAM_` environment prefix: | `WORKSTREAM_ARTIFACT_SCRATCH_RESERVATION_TTL_SECONDS` | `2400` | Database-independent wall-clock expiry for abandoned reservations. | | `WORKSTREAM_ARTIFACT_PREPARATION_TOTAL_DEADLINE_SECONDS` | `1800` | Total first-pass and provider-consumption deadline. | | `WORKSTREAM_ARTIFACT_SCRATCH_CLEANUP_MARGIN_SECONDS` | `300` | Required margin between the total deadline and reservation TTL. | +| `WORKSTREAM_ARTIFACT_SCRATCH_CLEANUP_INTERVAL_SECONDS` | `300` | Celery Beat cadence for the named stale-scratch cleanup task; accepted range is 1 through 86400 seconds. | | `WORKSTREAM_ARTIFACT_STREAM_BUFFER_BYTES` | `1048576` | Bounded streaming buffer, limited to at most 1 MiB. | | `WORKSTREAM_ARTIFACT_OPERATION_LOCK_TIMEOUT_SECONDS` | `1800` | Maximum wait for a private cross-process artifact-store operation lock before failing closed. | @@ -986,7 +1032,7 @@ Implementation is a clean cut: without changing the active v1 port; - ArtifactStore v1 methods, active callers, LocalStorage's public adapter surface, and schema then migrate atomically in the v2 clean-cut chunk; -- `flow_node` is removed from configuration without an alias; +- the Flow Node backend is removed from configuration without an alias; - obsolete caller-owned URI/hash and storage-scheme fields are removed in their owning guide/submission cutovers; - the misleading submission `/finalize` repair route is replaced by @@ -994,7 +1040,12 @@ Implementation is a clean cut: normal submission creation still enters evaluation automatically; - no dual write, nullable shadow field, fake verified backfill, fallback adapter, compatibility constructor, or second factory remains; -- pre-production data is rebuilt when authoritative bytes are unavailable. +- migration `0025` refuses every populated v1 artifact table before DDL and + leaves the prior schema and rows unchanged. It performs no automated rebuild + or fabricated backfill. Because this is a pre-production clean cut, the + Operator must reprovision an empty database/storage namespace out of band and + reingest authoritative bytes through v2; records whose authoritative bytes + are unavailable are not migrated. Every migration proves fresh upgrade, prior-head upgrade, populated-state preservation or explicit refusal, empty downgrade/re-upgrade, and no artifact diff --git a/docs/spec_authorization_service.md b/docs/spec_authorization_service.md index 2bf5afbd6..27a513296 100644 --- a/docs/spec_authorization_service.md +++ b/docs/spec_authorization_service.md @@ -119,7 +119,7 @@ The active model has no `both`, replacement field, replacement event, or replacement reason. Qualification evidence is bound to the same actor, project, and exact requested role. One active row is permitted per actor/project/role. Issue idempotency includes the requested role; revoke derives -the role from the locked grant. Migration `0025` refuses upgrade when obsolete +the role from the locked grant. Migration `0026` refuses upgrade when obsolete combined or replacement evidence exists and never converts or deletes those rows. It replaces current typed and PostgreSQL validators without changing historical migrations. diff --git a/scripts/check_stale_artifact_contracts.py b/scripts/check_stale_artifact_contracts.py index 0127af735..eaf54d06c 100644 --- a/scripts/check_stale_artifact_contracts.py +++ b/scripts/check_stale_artifact_contracts.py @@ -12,8 +12,8 @@ ROOT = Path(__file__).resolve().parents[1] -# ARTIFACT_CONTRACT_PHASE: foundation -ARTIFACT_CONTRACT_PHASE = "foundation" +# ARTIFACT_CONTRACT_PHASE: artifact_store_cutover +ARTIFACT_CONTRACT_PHASE = "artifact_store_cutover" PHASES = ( "foundation", diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index cf1a9c371..494ba7d14 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -46,7 +46,7 @@ FOUNDATION_ARTIFACT_COVERAGE_COMMAND = ( "coverage report " "--include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py," - "app/interfaces/artifacts.py," + "app/interfaces/artifact_operations.py,app/interfaces/artifacts.py," "app/modules/artifacts/*' --precision=2 --fail-under=90" ) ARTIFACT_COVERAGE_COMMAND_OWNERS = { @@ -4114,7 +4114,7 @@ def test_auth_spec_orders_service_admission_before_project_roles() -> None: def test_parallel_initiative_status_matches_trusted_main() -> None: - """Auth and artifact maps cannot regress already merged prerequisites.""" + """Auth prerequisites and the current ART gate remain internally consistent.""" auth_map = Path( ".agent-loop/initiatives/WS-AUTH-001-workstream-authorization-service/" "CHUNK_MAP.md" @@ -4128,6 +4128,12 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: artifact_status = Path( ".agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md" ).read_text(encoding="utf-8") + artifact_contract = Path( + ".agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/" + "WS-ART-001-02A3-artifact-store-v2-local-clean-cut.md" + ).read_text(encoding="utf-8") + work_queue = Path(".agent-loop/WORK_QUEUE.md").read_text(encoding="utf-8") + loop_state = Path(".agent-loop/LOOP_STATE.md").read_text(encoding="utf-8") assert "Merged through PR #131 as `aa0fdcd`" in auth_map assert "`WS-AUTH-001-09B` - Controlled Service Actor Provisioning" in auth_status @@ -4136,12 +4142,34 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: assert "| `WS-AUTH-001-09A` | Merged |" in auth_status assert "| `WS-AUTH-001-09B` | In progress |" in auth_status assert "Merged through PR #129 as `9a04434`" in artifact_map - assert "Reviewed in isolated worktree; PR publication pending" in artifact_map - assert "No artifact implementation chunk is active." in artifact_status + artifact_phases = ( + "Deterministic proof passed; exact-SHA internal review in progress", + "Internal review and deterministic evidence passed; external checks pending", + ) + selected_phases = [phase for phase in artifact_phases if phase in artifact_map] + assert len(selected_phases) == 1 + selected_phase = selected_phases[0] + assert selected_phase in work_queue + assert f"Status: {selected_phase}" in artifact_contract + assert ( + "AUTH's owner reconciliation merged through PR #140 as\n" + "`d541521`" in artifact_status + ) assert ( - "`WS-ART-001-02A3` implementation\nand review are complete" in artifact_status + "`WS-ART-001-02A3` implementation and merged-main deterministic repair are\n" + "complete" in artifact_status ) - assert "PR #129 merged `WS-ART-001-02A2` as `9a04434`" in artifact_status + if selected_phase == artifact_phases[0]: + assert "The current gate is all nine\nexact-SHA internal tracks" in artifact_status + assert "Current gate: complete all nine exact-SHA internal reviewer tracks" in loop_state + else: + assert "The current gate is GitHub Actions, CodeRabbit, and explicit human review" in ( + artifact_status.replace("\n", " ") + ) + assert "Current gate: publish PR #141 for GitHub Actions, CodeRabbit, and explicit" in ( + loop_state.replace("\n", " ") + ) + assert "No\nlater artifact chunk starts automatically" in artifact_status def test_stale_authorization_discovery_includes_new_untracked_docs() -> None: @@ -4550,25 +4578,25 @@ def test_artifact_coverage_phase_is_derived_from_work_queue() -> None: ROOT = original_root -def test_stale_artifact_contracts_foundation_keeps_later_terms_inactive() -> None: - """Foundation checks generic neutrality without preempting later cutovers.""" +def test_stale_artifact_contracts_cutover_rejects_reached_terms_only() -> None: + """The active cutover rejects reached terms without preempting later phases.""" gate = load_module( "stale_artifact_contracts_foundation", "scripts/check_stale_artifact_contracts.py", ) - assert gate.ARTIFACT_CONTRACT_PHASE == "foundation" + assert gate.ARTIFACT_CONTRACT_PHASE == "artifact_store_cutover" assert ( gate.scan_text( "backend/app/modules/tasks/schemas.py", "package_uri content_cid artifact_manifest_hash", - "foundation", + gate.ARTIFACT_CONTRACT_PHASE, ) == [] ) failures = gate.scan_text( "contracts/artifact-store/version_1/schema/example.json", '{"cid": "provider-specific"}', - "foundation", + gate.ARTIFACT_CONTRACT_PHASE, ) assert failures == [ "contracts/artifact-store/version_1/schema/example.json:1: " @@ -5456,7 +5484,7 @@ def main() -> int: test_agent_gate_dependencies_and_workflow_are_pinned, test_backend_coverage_thresholds_are_regression_protected, test_artifact_coverage_phase_is_derived_from_work_queue, - test_stale_artifact_contracts_foundation_keeps_later_terms_inactive, + test_stale_artifact_contracts_cutover_rejects_reached_terms_only, test_stale_artifact_contracts_active_later_phase_owns_only_reached_terms, test_stale_artifact_contracts_malformed_phase_fails_closed, test_stale_artifact_contracts_enforce_aws_first_v01,