diff --git a/.agent-loop/LOOP_STATE.md b/.agent-loop/LOOP_STATE.md index 1cc333487..18cc796b3 100644 --- a/.agent-loop/LOOP_STATE.md +++ b/.agent-loop/LOOP_STATE.md @@ -20,12 +20,18 @@ `eeb3dc2` recorded its two administrative reads and stopped. - PR #141 merged `WS-ART-001-02A3` into `main` as `a10d901` on 2026-07-18. PR #151 then merged `WS-ART-001-02B1` as `1b5422f` on 2026-07-19; - ART-02C1 remains inactive pending signed memory and a separate explicit start. + the user then explicitly started ART-02C1. +- Active ART implementation chunk: `WS-ART-001-02C1` on + `codex/ws-art-001-02c1-admission-put-attempt`. +- The ART worktree consumes merged AUTH, REV, and CON contracts without + editing or activating their independently owned runtime behavior. - AUTH-09D-A merged through PR #148 as `99ae4c9`; signed schema-v2 memory at `cf8a3e8` recorded the stopped gate and exact 09D-B successor. - AUTH-09D-B merged through PR #152 as `93dd392`; signed schema-v2 memory at `912a6254` stopped and named the contributor foundation as its exact successor. +- PR #153 merged `WS-AUTH-001-CONTRIBUTOR-FOUNDATION` as `8d5eb15`; ART-02C1 + now integrates that trusted `main` state without activating AUTH-09E. - PR #119 merged `WS-AUTH-001-05B` as `ad71c7e`. - PR #120 merged `WS-ART-001-OBJECT-STORAGE-AMENDMENT` as `4408256`. - PR #122 merged the first automated post-merge memory implementation as @@ -38,6 +44,11 @@ required internal tracks. PR publication and external checks are the current gate; Backend must still prove 78/90 percent aggregate coverage. It changes no action availability, and no service caller becomes executable before AUTH-09E. +- Current ART gate: integrate trusted `main`, complete deterministic 02C1 + proof, and pass all nine exact-SHA internal reviewer tracks. Those steps are + complete; publish the final candidate to existing PR #154, then pass fresh + external checks and explicit human review. No later ART chunk starts + automatically. - Scope checkpoint: AWS S3 is the only v0.1 production provider; MinIO is local/CI S3 protocol proof; LocalStorage is focused development/test; R2 and Flow Node are deferred. Product modules receive narrow artifact capabilities, @@ -55,13 +66,11 @@ admission, prepared mutation authority, and exact AUTH-only activation chunks; neither reconciliation PR activates feature behavior. - Parallel artifact checkpoint: ART-02A1, ART-02A2, ART-02A3, and ART-02B1 - merged through PRs #127, #129, #141, and #151. ART-02B1 adds real MinIO - protocol proof plus a fail-closed, runtime-ineligible native AWS profile; - ART-02C1 remains inactive. -- Authorization checkpoint: AUTH-07B through AUTH-09D-B merged through PRs - #130, #131, #132, #143, #146, #148, and #152. The contributor foundation is - internally approved at code SHA `4d1fc507`; AUTH-09E remains - inactive. + merged through PRs #127, #129, #141, and #151. ART-02C1 is active and adds + only durable admission plus prepared put-attempt state before provider I/O. +- Authorization checkpoint: AUTH-07B through AUTH-09D-B and the contributor + foundation merged through PRs #130, #131, #132, #143, #146, #148, #152, + and #153. AUTH-09E remains inactive. - Parallel coverage work: `WS-QUAL-001-01B2` remains paused. Its last official whole-app result is `6466/8159` statements (`79.249908%`); no replacement evidence exists. diff --git a/.agent-loop/WORK_QUEUE.md b/.agent-loop/WORK_QUEUE.md index 429b977c2..2ece0c24b 100644 --- a/.agent-loop/WORK_QUEUE.md +++ b/.agent-loop/WORK_QUEUE.md @@ -5,6 +5,7 @@ | Chunk | Title | Risk | Status | |---|---|---:|---| | `WS-AUTH-001-CONTRIBUTOR-FOUNDATION` | Contributor Fields And Canonical-Human Lineage | L1 | Internal review passed at `4d1fc507`; PR/external checks pending; aggregate coverage mandatory in Backend | +| `WS-ART-001-02C1` | Admission And Put-Attempt Foundation | L1 | Active after PR #151 and explicit user start; implementation and deterministic proof in progress | Live post-merge state remains read from signed `automation/loop-memory` output. This authored queue records the separately approved parallel chunks. @@ -17,7 +18,6 @@ output. This authored queue records the separately approved parallel chunks. | `WS-AUTH-001-09E` | Fixed Service Runtime Admission | L1 | Inactive until contributor-foundation merge/memory and explicit user start | | `WS-QUAL-001-02` | Project Service Coverage | L1 | Inactive until 01B2 merge/memory plus explicit user start | | `WS-POL-002-04` | Locked Runtime Execution And Routing Hardening | L1 | Inactive pending relevant authorization proof and a separate explicit user start | -| `WS-ART-001-02C1` | Admission And Put-Attempt Foundation | L1 | Inactive until signed 02B1 merge memory and explicit user start | | `WS-ART-001-02C2` | Verification Publication And Fencing | L1 | Inactive until 02C1 merge and explicit user start | | `WS-ART-001-02C3` | Recovery Attempt And Idempotency Chain | L1 | Inactive until 02C2 merge and explicit user start | | `WS-ART-001-02D` | Operator Artifact Operations And AWS Readiness | L1 | Inactive until 02C3 and exact AUTH prerequisites | @@ -26,7 +26,7 @@ output. This authored queue records the separately approved parallel chunks. | Chunk | Title | Risk | Status | |---|---|---:|---| -| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Merged through PR #151 as `1b5422f` on 2026-07-19 | +| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Merged through PR #151 as `1b5422fc` on 2026-07-19 | | `WS-AUTH-001-09D-A` | Profile Lifecycle And Evidence Repair | L1 | Merged through PR #148 as `99ae4c9` on 2026-07-18 | | `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 | Merged through PR #146 as `0ffdabf` on 2026-07-18 | | `WS-ENG-001-01` | Codex-native zero-trust loop bootstrap | L1 | Merged through PR #23 on 2026-06-20 | @@ -105,9 +105,9 @@ Coverage R10 merged through PR #108. Do not start 01B2, chunk 02, or another coverage implementation chunk from this worktree. `WS-ART-001-01`, the AWS-first planning amendment, `02A1`, `02A2`, `02A3`, and -`02B1` are merged; PR #151 merged `02B1` as `1b5422f`. R2 and Flow Node are -deferred. `02C1` remains inactive until signed merge memory and a separate -explicit start. +`02B1` are merged; PR #151 merged `02B1` as `1b5422fc`. R2 and Flow Node are +deferred. The user explicitly started `02C1` on 2026-07-19. `02C2` remains +inactive until `02C1` merges and receives a separate explicit start. Coverage work proceeds independently in its own worktree and is not owned by this AUTH queue update. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md index 087bd357b..dd01eba89 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md @@ -10,8 +10,8 @@ Each chunk is one PR. No later chunk starts automatically. | `WS-ART-001-02A1` | Install only ADR 0014's small typed external-service adapter/factory foundation without migrating a capability. | L1 | Merged through PR #127 as `f64a8e5` | | `WS-ART-001-02A2` | Add bounded committed-source preparation and inactive scratch-cleanup mechanics without changing the active v1 port. | L1 | Merged through PR #129 as `9a04434` on 2026-07-16 | | `WS-ART-001-02A3` | Replace ArtifactStore v1 with byte-only v2, activate API-startup and Celery Beat scratch cleanup, migrate schema/callers/factory, and remove `flow_node` in one atomic clean cut. | L1 | Merged through PR #141 as `a10d901` on 2026-07-18 | -| `WS-ART-001-02B1` | Implement the S3-compatible adapter, MinIO integration, and AWS S3 production profile. | L1 | Active after 02A3 merged through PR #141 and explicit user start | -| `WS-ART-001-02C1` | Add the generic durable-byte admission ledger and durable put-attempt state foundation without provider execution. | L1 | Proposed after 02B1 | +| `WS-ART-001-02B1` | Implement the S3-compatible adapter, MinIO integration, and AWS S3 production profile. | L1 | Merged through PR #151 as `1b5422fc` on 2026-07-19 | +| `WS-ART-001-02C1` | Add the generic durable-byte admission ledger and durable put-attempt state foundation without provider execution. | L1 | Active after PR #151 and explicit user start on 2026-07-19 | | `WS-ART-001-02C2` | Add put resolution, verification publication, complete-object observation, immutable receipts, and PostgreSQL execution fencing without recovery attempts or routes. | L1 | Proposed after 02C1 | | `WS-ART-001-02C3` | Add the recovery-attempt model and exact idempotent source-job to retry-job chain without public or Operator routes. | L1 | Proposed after 02C2 | | `WS-ART-001-02D` | Add hidden Operator content/job/retry/recovery/audit APIs, canonical resource composition, and production-readiness checks while actions and provider profiles remain inactive. | L1 | Proposed after 02C3, AUTH-09E, and `WS-AUTH-001-ART-CUSTODY` | diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index af1a1509b..07efb6332 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -1,4 +1,4 @@ -# Status: WS-ART-001 S3-Compatible Object Storage Amendment +# Status: WS-ART-001 Immutable Artifact Storage ## Current State @@ -6,16 +6,19 @@ Original planning merged through PR #97, artifact/LocalStorage foundation merged through PR #101, the AWS-first object-storage amendment merged through PR #120 as `4408256`, the external-service adapter foundation merged through PR #127 as `f64a8e5`, committed-source preparation merged through PR #129 as -`9a04434`, and the ArtifactStore v2 Local clean cut merged through PR #141 as -`a10d901` on 2026-07-18. The user explicitly started `WS-ART-001-02B1` on -2026-07-18. +`9a04434`, the ArtifactStore v2 Local clean cut merged through PR #141 as +`a10d901`, and S3-compatible MinIO/AWS preparation merged through PR #151 as +`1b5422fc` on 2026-07-19. The user explicitly started `WS-ART-001-02C1` on +2026-07-19. The planning-only cross-initiative boundary reconciliation merged through PR #139 as `5d353b6`, and AUTH's owner reconciliation merged through PR #140 as `d541521`. ART now consumes AUTH's canonical activation-custody and prepared mutation contracts without editing or activating AUTH runtime behavior. -AUTH-09D-A merged through PR #148 as `99ae4c9` and is integrated into the ART -candidate; AUTH-09D-B remains inactive. +AUTH-09D-A merged through PR #148 as `99ae4c9`, AUTH-09D-B merged through PR +#152 as `93dd392`, and the contributor foundation merged through PR #153 as +`8d5eb15b`; all are integrated into the ART candidate. AUTH-09E remains +inactive. The Flow Node-focused amendment candidate `6cc422d` passed deterministic checks but failed internal review on recovery/API completeness. Before repair, the user @@ -34,29 +37,32 @@ approval or reusable evidence. Its source remains on branch ## Current Work -`WS-ART-001-02B1` is active. It adds one `S3CompatibleArtifactStore`, runs the -shared ArtifactStore v2 vectors against real digest-pinned MinIO, and validates -an isolated native-AWS workload-identity profile. MinIO is runtime-eligible -only in local/development/test after the PostgreSQL namespace claim. Native AWS -remains runtime-ineligible and fails with -`artifact_provider_live_proof_required` before factory construction, -credential resolution, namespace claim, or provider I/O. No product ingest, -durable admission, put-attempt resolution, verification job, recovery route, -or optional-provider runtime is activated. R2 and Flow Node remain deferred. +`WS-ART-001-02C1` is active. It adds the PostgreSQL durable-byte admission +ledger, closed internal guide/contributor/checker-output requests, and one +`prepared` `ArtifactPutAttempt` created atomically before provider I/O. Scope +limits are explicit configuration; callers cannot supply scope collections; +exact content is charged once per task, producer, project, and deployment +scope. Provider execution, verification, publication, recovery, routes, and +product cutover remain inactive. Native AWS remains runtime-ineligible. R2 and +Flow Node remain deferred. + +Final implementation SHA `535069cfb1a7312d731bb14a6023ceb0894402e9` +passed 371 focused tests with 94.02 percent scoped coverage and all nine +required internal reviewer tracks. The current gate is publication of that +reviewed candidate to existing PR #154 followed by fresh GitHub and CodeRabbit +evidence. ## Next Proposed Chunk -`02C1` owns generic durable-byte admission and put-attempt state only after -`02B1` merges and receives a separate explicit start. Neither R2 nor Flow Node +`02C2` may add fenced put resolution and verification publication only after +`02C1` merges and receives a separate explicit start. Neither R2 nor Flow Node has a v0.1 chunk. ## Gate -The reviewed implementation recorded: "The current gate is deterministic 02B1 -proof followed by all nine exact-SHA internal reviewer tracks." After integrating -latest `main`, including merged REV planning, that gate passed again at -`9cd5620e` after addressing all four valid CodeRabbit findings, with no ART -runtime or ownership drift. The remaining gate is the post-fix GitHub Actions -and CodeRabbit rerun plus explicit human review. Durable admission, put attempts, -verification publication, and recovery remain in later owning chunks. No later -artifact chunk starts automatically, and only the user may approve merge. +The current gate is deterministic 02C1 proof followed by all nine exact-SHA +internal reviewer tracks; that gate is complete. GitHub Actions, CodeRabbit, +and explicit human review remain pending on the published final candidate. +Provider execution, verification publication, and recovery remain in later +owning chunks. No later artifact chunk starts automatically, and only the user +may approve merge. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md index 75b25d2ec..9ce5fc1c8 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C1-admission-put-attempt-foundation.md @@ -1,6 +1,6 @@ # Chunk Contract: WS-ART-001-02C1 - Admission And Put-Attempt Foundation -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02B1 +Initiative: `WS-ART-001` | Risk: L1 | Status: Active after explicit user start Artifact contract phase: `artifact_store_cutover` @@ -15,9 +15,13 @@ publication, recovery, Operator routes, and product cutovers inactive. - one artifact-foundation migration; - artifact admission and put-attempt models, schemas, repository, service, and contracts; +- the actors-owned frozen admission-proof contract, repository lock, and + service method required to revalidate an exact profile/link pair in the + caller-owned admission transaction; - `backend/app/core/config.py` for durable byte limits; - generic audit repository only when existing audit support is insufficient; - focused PostgreSQL admission, concurrency, migration, and state tests; +- focused actor/artifact ownership-boundary and transactional proof tests; - `.github/workflows/backend.yml` only to expand the exact 90 percent scoped gate; - `scripts/test_agent_gates.py` only to assert that backend CI retains this chunk's exact scoped coverage sources and fail-closed 90 percent threshold; @@ -31,6 +35,8 @@ publication, recovery, Operator routes, and product cutovers inactive. - provider mutation replay, overwrite, delete, retain, or release; - task-claim or reviewer-lease changes; - production dispatch or activation. +- AUTH permission decisions or action activation, actor provisioning or + lifecycle mutation, and actor-facing routes or product cutover. ## Acceptance Criteria @@ -80,7 +86,7 @@ coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 ```bash docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_artifact_admission.py tests/test_config.py -q --cov=app.interfaces.artifact_operations --cov=app.modules.artifacts --cov=app.modules.audit --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_artifact_admission.py tests/test_artifact_architecture.py tests/test_artifact_cleanup_wiring.py tests/test_artifact_preparation.py tests/test_artifact_store_conformance.py tests/test_artifacts.py tests/test_local_artifact_store.py tests/test_s3_artifact_store.py tests/test_audit.py tests/test_config.py -q --cov=app.interfaces.artifact_operations --cov=app.modules.artifacts --cov=app.modules.audit --cov=app.core.config --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) (cd backend && .venv/bin/ruff check app tests) python3 scripts/check_stale_artifact_contracts.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md index 9d434824c..f929fdcf2 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-02C2-verification-publication-fencing.md @@ -1,4 +1,4 @@ -# Chunk Contract: WS-ART-001-02C2 Verification Publication And Fencing +# Chunk Contract: WS-ART-001-02C2 - Verification Publication And Fencing Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 02C1 diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C1-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C1-external-review-response.md new file mode 100644 index 000000000..f77dca201 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C1-external-review-response.md @@ -0,0 +1,67 @@ +# External Review Response: WS-ART-001-02C1 + +## Boundary + +This file records GitHub Actions and CodeRabbit separately from internal +review. It does not replace internal exact-SHA evidence. + +Reviewed implementation SHA: `535069cfb1a7312d731bb14a6023ceb0894402e9` + +Trusted base: `8d5eb15b384fd75787ce98a099400a1d335d2560` + +PR: #154, `https://github.com/Flow-Research/workstream/pull/154` + +## Historical Evidence + +Earlier GitHub and CodeRabbit results ran on pre-rebase heads and are not +evidence for the reviewed implementation SHA. The old remote Backend failure +was caused by stale migration-head expectations before the contributor +foundation rebase; local current-SHA migration and focused proof pass. No old +external PASS is carried forward. + +## Current Status + +| Source | Status | Notes | +|---|---:|---| +| GitHub Agent Gates | Pending | Await publication of the rebased final candidate. | +| GitHub Backend | Pending | Must run the full isolated suite, all scoped gates, and 78-percent floor on the final head. | +| CodeRabbit | Pending | Request a fresh review on the published final head. | +| Human review | Pending | Only the user may approve merge. | + +## Current-Head Triage + +Comments addressed: + +- The first Agent Gates and Backend runs on evidence head `a93be2ec` failed at + the shared internal-review evidence parser before tests. The regenerated + evidence used non-canonical provenance labels and verdict text. +- Evidence now uses the required `Reviewed code SHA`, UTC `Reviewed at`, and + `Reviewer run IDs` labels. QA and docs use canonical `PASS after fixes` with + no remaining blocking findings. + +Comments deferred: none. + +Human decisions needed: explicit merge approval only after external checks pass. + +Commands rerun: + +```text +PR_HEAD_SHA=a93be2ec25689e1f8e036321d8a45b1fe35455ed python3 scripts/check_internal_review_evidence.py: PASS +python3 scripts/test_agent_gates.py: PASS, 88 tests +python3 scripts/check_markdown_links.py: PASS +git diff --check: PASS +``` + +Remaining risks: GitHub and CodeRabbit must complete against the corrected +published evidence head. + +## Response Rule + +Assess only external findings verified against the published final head. Any +implementation, test, workflow, policy, specification, or chunk-contract repair +requires affected internal reviewers to rerun and evidence to be rebound. + +## Stop Condition + +Wait for fresh external checks and explicit user approval of PR #154. Do not +merge and do not start `WS-ART-001-02C2` automatically. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C1-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C1-internal-review-evidence.md new file mode 100644 index 000000000..b988d5ba1 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C1-internal-review-evidence.md @@ -0,0 +1,113 @@ +# Internal Review Evidence: WS-ART-001-02C1 + +## Chunk + +`WS-ART-001-02C1`: Admission And Put-Attempt Foundation + +Open sub-agent sessions: none. + +Valid findings addressed: yes. + +## Reviewed Revision + +Reviewed code SHA: `535069cfb1a7312d731bb14a6023ceb0894402e9` + +Trusted base: `8d5eb15b384fd75787ce98a099400a1d335d2560` + +Reviewed at: 2026-07-19T15:20:00Z + +Reviewer run IDs: senior-engineering/security-auth/ci-integrity=/root/review_senior_6392825f; architecture/product-ops/test-delta=/root/review_arch_6392825f; qa-test/reuse-dedup/docs=/root/plan_review_actor_boundary + +Each track explicitly rebound its review to the final SHA above. All sessions +completed. + +Only review evidence, trust-bundle, external-response, and initiative-status +files may change after the reviewed implementation SHA. Any implementation, +test, workflow, policy, specification, or chunk-contract change invalidates +this evidence and requires a new exact-SHA review cycle. + +## Reviewed Change + +- Adds server-owned deployment, project, producer, and task admission scopes, + unique content charges, and one atomic prepared put attempt before provider + I/O. +- Accepts only closed guide, contributor, and checker-output requests and + derives canonical relationships and scope limits inside Workstream. +- Uses an actors-owned same-transaction proof boundary that locks the exact + profile then identity link and returns frozen primitive state; ART no longer + imports or queries actor persistence. +- Leaves every prepared execution field inactive: `next_run_at`, executor, + lease, terminal result, replica, and receipt are null and execution generation + is zero. +- Makes release timestamps biconditional with released charge state. +- Adds deterministic database contention proof for same-content deduplication + and distinct-content oversubscription. +- Adds migration `0028_artifact_admission`, following + `0027_contributor_foundation`, with populated-state downgrade refusal. + +## Reviewer Results + +| Reviewer | Result | Blocking findings | Disposition | +|---|---:|---|---| +| senior engineering | PASS | None | Transaction and ownership repairs are bounded and maintainable. | +| architecture | PASS | None | Actors own persistence proof; ART remains provider-neutral and dormant. | +| QA/test | PASS after fixes | None | Distinct actors and a scope-reservation barrier now prove real ledger contention. | +| security/auth | PASS | None | Exact identity, relationship, configuration, and quota checks fail closed without duplicating AUTH. | +| product/ops | PASS | None | No task, submission, checker, review, contribution, compensation, or reputation lifecycle mutation. | +| reuse/dedup | PASS | None | Canonical actor and artifact interfaces are reused; optional test-only barrier duplication is documented. | +| CI integrity | PASS | None | All prior 90-percent gates and the repository 78-percent floor remain fail closed. | +| test delta | PASS | None | Removed tests match intentionally removed provider execution; retained behavior gains stronger proof. | +| docs | PASS after fixes | None | Migration, inactive scheduling, status, evidence, and trust bundle are synchronized. | + +## Findings Addressed + +- Removed ART's direct `ActorProfile` and `ActorIdentityLink` queries and added + an actors-owned frozen admission proof in the caller's transaction. +- Made `next_run_at` nullable and null for `prepared`, enforced by the prepared + execution-inactive database constraint. +- Enforced `(state = 'released') = (released_at is not null)`. +- Preclaimed the namespace in concurrency fixtures, used distinct actors, and + synchronized immediately before the real scope reservation method. The + same-content case proves two attempts, seven unique charges, eight links, and + four counted deployment bytes; oversubscription proves one success, one typed + capacity failure, one attempt, and four charges. +- Corrected active migration wording from `0027` to + `0028_artifact_admission`. + +## Deterministic Proof + +The isolated real-PostgreSQL/MinIO focused matrix ran against the reviewed SHA: + +```text +371 passed in 757.38s +scoped coverage: 94.02% +required scoped floor: 90% +Alembic head: 0028_artifact_admission +``` + +Additional results: + +- Ruff: PASS. +- configured docstring coverage: PASS at 90.5 percent. +- stale artifact contract scan: PASS at `artifact_store_cutover`. +- agent gates: PASS, 88 tests. +- Markdown links: PASS. +- schema-v2 merge-intent validation: PASS. +- `git diff --check`: PASS. + +GitHub Backend remains authoritative for the isolated full repository suite and +78-percent repository-wide floor on the published final head. + +## Remaining Risks + +- Provider execution, acknowledgement observation, verification, publication, + recovery, routes, and product cutover remain intentionally unavailable. +- Native AWS remains runtime-ineligible pending separately owned live proof. +- External GitHub and CodeRabbit checks remain pending until the rebased final + candidate is published. + +## Stop Condition + +Publish the evidence-bound candidate to existing PR #154, wait for fresh +external checks, and stop for explicit user merge approval. Do not start +`WS-ART-001-02C2` automatically. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C1-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C1-pr-trust-bundle.md new file mode 100644 index 000000000..df39978c7 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-02C1-pr-trust-bundle.md @@ -0,0 +1,104 @@ +# PR Trust Bundle: WS-ART-001-02C1 + +## Chunk And Intent + +`WS-ART-001-02C1` - Admission And Put-Attempt Foundation + +Reviewed implementation SHA: `535069cfb1a7312d731bb14a6023ceb0894402e9` + +Trusted base: `8d5eb15b384fd75787ce98a099400a1d335d2560` + +Merge intent: `.agent-loop/merge-intents/WS-ART-001-02C1.json` + +Create the durable PostgreSQL admission and prepared-attempt transaction that +must commit before provider I/O while keeping execution and product cutover +inactive. + +## Design And Scope + +- Workstream derives deployment, project, producer, and applicable task scopes. +- Unique provisional/completed charges count once per scope and exact content; + released charges may be reacquired only under locked capacity checks. +- Actors owns exact profile/link locking and returns frozen primitive proof in + the caller's admission transaction; ART makes no AUTH permission decision. +- One namespace claim, complete charge set, and `prepared` attempt commit + atomically. Prepared scheduling/execution fields remain inactive and null. +- No provider write/observation, verification, publication, recovery, Celery + execution, route, product cutover, task claim, reviewer lease, R2, or Flow + Node path is included. + +## Acceptance Proof + +- [x] Callers cannot supply, omit, or weaken admission scopes or limits. +- [x] Exact actor/link and canonical product relationships are locked and + revalidated without crossing actor persistence ownership. +- [x] Same content is charged once per applicable scope under real concurrent + ledger contention. +- [x] Concurrent distinct content cannot oversubscribe a shared scope. +- [x] Reservation failure leaves no partial charge, attempt, receipt, audit + success, or provider call. +- [x] A committed prepared attempt is required before any later provider work. +- [x] Prepared `next_run_at`, executor, lease, terminal, replica, and receipt + fields are null and execution generation is zero. +- [x] Exactly one schema-v2 merge intent names only inactive successor `02C2` + and requires a separate explicit start. + +## Tests And Checks + +```text +371 focused tests PASS in 757.38s +Scoped changed-subsystem coverage 94.02% (required 90%) +Alembic head 0028_artifact_admission +Ruff PASS +Configured docstring coverage PASS at 90.5% +Stale artifact contract scan PASS +88 agent-gate tests PASS +Markdown links PASS +Schema-v2 merge-intent validation PASS +git diff --check PASS +``` + +GitHub Backend remains authoritative for the full isolated repository suite and +78-percent repository coverage floor. + +## Internal Review + +All nine required tracks reviewed exact SHA `535069cf...`: senior engineering, +architecture, QA/test, security/auth, product/ops, reuse/dedup, CI integrity, +test delta, and docs. QA's initial High concurrency-proof finding was repaired +and passed on rerun. Docs' stale-evidence finding was repaired by regenerating +the exact-SHA evidence, this bundle, external status, and initiative state. No +blocking finding remains and every reviewer session is closed. + +## External Review + +| Source | Status | Notes | +|---|---:|---| +| GitHub Agent Gates | Pending | Must run on the published rebased final head. | +| GitHub Backend | Pending | Must prove the full suite, scoped gates, and 78-percent floor. | +| CodeRabbit | Pending | A fresh current-head review must be requested after publication. | +| Human review | Pending | Only the user may approve PR #154 for merge. | + +## Remaining Risks + +- Admission is intentionally not connected to provider execution or product + submission routes. +- Native AWS remains unavailable until its separately owned live proof. +- External checks have not yet run on the final rebased head. + +## Human Review Focus + +- Can a producer omit a scope, weaken capacity, or substitute identity/context? +- Does every success commit the complete ledger and exactly one prepared + attempt before provider side effects? +- Do rollback, replay, release/reacquisition, and concurrent contention preserve + exact accounting? +- Are provider execution, verification, recovery, and product routes absent? + +## Human Merge Ownership + +- [ ] I can explain what changed and why. +- [ ] I know what could break. +- [ ] I accept the remaining risks. +- [ ] GitHub CI and external review pass on the final head. +- [ ] The user explicitly approved PR #154 for merge. diff --git a/.agent-loop/merge-intents/WS-ART-001-02C1.json b/.agent-loop/merge-intents/WS-ART-001-02C1.json new file mode 100644 index 000000000..2b1ebeb24 --- /dev/null +++ b/.agent-loop/merge-intents/WS-ART-001-02C1.json @@ -0,0 +1,9 @@ +{ + "chunk_id": "WS-ART-001-02C1", + "chunk_title": "Admission And Put-Attempt Foundation", + "initiative_id": "WS-ART-001", + "next_chunk_id": "WS-ART-001-02C2", + "next_chunk_title": "Verification Publication And Fencing", + "next_requires_explicit_start": true, + "schema_version": 2 +} diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml index 56e887446..b3101fb87 100644 --- a/.github/workflows/backend.yml +++ b/.github/workflows/backend.yml @@ -142,6 +142,14 @@ jobs: --precision=2 --fail-under=90 + - name: Audit subsystem coverage + working-directory: backend + run: >- + coverage report + --include='app/modules/audit/*' + --precision=2 + --fail-under=90 + - name: Actor subsystem coverage working-directory: backend run: >- diff --git a/backend/alembic/versions/0028_artifact_admission.py b/backend/alembic/versions/0028_artifact_admission.py new file mode 100644 index 000000000..c51a49a0b --- /dev/null +++ b/backend/alembic/versions/0028_artifact_admission.py @@ -0,0 +1,384 @@ +"""add durable-byte admission and prepared put attempts + +Revision ID: 0028_artifact_admission +Revises: 0027_contributor_foundation +Create Date: 2026-07-19 +""" + +from __future__ import annotations + +from alembic import op +import sqlalchemy as sa + + +revision = "0028_artifact_admission" +down_revision = "0027_contributor_foundation" +branch_labels = depends_on = None + +_ADMISSION_TABLES = ( + "artifact_put_attempt_charges", + "artifact_put_attempts", + "artifact_admission_charges", + "artifact_admission_scopes", +) + + +def _refuse_populated_admission_downgrade() -> None: + """Refuse to destroy durable admission, charge, or attempt evidence.""" + connection = op.get_bind() + connection.execute( + sa.text( + "lock table " + + ", ".join(_ADMISSION_TABLES) + + " in access exclusive mode" + ) + ) + if any( + connection.execute( + sa.text(f"select exists(select 1 from {table_name})") + ).scalar() + for table_name in _ADMISSION_TABLES + ): + raise RuntimeError("cannot downgrade populated artifact admission ledger") + + +def upgrade() -> None: + """Install generic admission and pre-I/O attempt state.""" + op.create_unique_constraint( + "uq_artifact_storage_namespace_id_fingerprint", + "artifact_storage_namespaces", + ["id", "namespace_fingerprint"], + ) + + op.create_table( + "artifact_admission_scopes", + sa.Column("scope_type", sa.String(20), nullable=False), + sa.Column("scope_id", sa.String(120), nullable=False), + sa.Column("limit_bytes", sa.BigInteger(), nullable=False), + sa.Column("counted_bytes", sa.BigInteger(), nullable=False, server_default="0"), + sa.Column("cas_version", sa.BigInteger(), nullable=False, server_default="0"), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.Column( + "updated_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.CheckConstraint( + "scope_type in ('deployment', 'project', 'producer', 'task')", + name="scope_type", + ), + sa.CheckConstraint( + "octet_length(scope_id) between 1 and 120", + name="scope_id_bounds", + ), + sa.CheckConstraint("limit_bytes > 0", name="limit_positive"), + sa.CheckConstraint( + "counted_bytes >= 0 and counted_bytes <= limit_bytes", + name="counted_bytes_within_limit", + ), + sa.CheckConstraint("cas_version >= 0", name="cas_nonnegative"), + sa.PrimaryKeyConstraint("scope_type", "scope_id"), + ) + + op.create_table( + "artifact_admission_charges", + sa.Column("id", sa.String(36), nullable=False), + sa.Column("scope_type", sa.String(20), nullable=False), + sa.Column("scope_id", sa.String(120), nullable=False), + sa.Column("sha256", sa.String(71), nullable=False), + sa.Column("byte_count", sa.BigInteger(), nullable=False), + sa.Column("producer_type", sa.String(30), nullable=False), + sa.Column("producer_ref", sa.String(120), nullable=False), + sa.Column("creating_operation_identity", sa.String(71), nullable=False), + sa.Column("state", sa.String(20), nullable=False, server_default="provisional"), + sa.Column("cas_version", sa.BigInteger(), nullable=False, server_default="0"), + sa.Column( + "reserved_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.Column("completed_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("released_at", sa.DateTime(timezone=True), nullable=True), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.Column( + "updated_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.CheckConstraint( + "sha256 ~ '^sha256:[0-9a-f]{64}$'", + name="sha256_shape", + ), + sa.CheckConstraint("byte_count >= 0", name="byte_count_nonnegative"), + sa.CheckConstraint( + "producer_type in ('actor_profile', 'service_identity')", + name="producer_type", + ), + sa.CheckConstraint( + "creating_operation_identity ~ '^sha256:[0-9a-f]{64}$'", + name="operation_identity_shape", + ), + sa.CheckConstraint( + "state in ('provisional', 'completed', 'released')", + name="state", + ), + sa.CheckConstraint("cas_version >= 0", name="cas_nonnegative"), + sa.CheckConstraint( + "(state = 'completed') = (completed_at is not null)", + name="completed_timestamp", + ), + sa.CheckConstraint( + "(state = 'released') = (released_at is not null)", + name="released_timestamp", + ), + sa.ForeignKeyConstraint( + ["scope_type", "scope_id"], + [ + "artifact_admission_scopes.scope_type", + "artifact_admission_scopes.scope_id", + ], + name="fk_artifact_admission_charges_scope", + ondelete="RESTRICT", + ), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint( + "scope_type", + "scope_id", + "sha256", + "byte_count", + name="uq_artifact_admission_charge_scope_content", + ), + ) + + op.create_table( + "artifact_put_attempts", + sa.Column("id", sa.String(36), nullable=False), + sa.Column("producer_request_type", sa.String(30), nullable=False), + sa.Column("producer_type", sa.String(30), nullable=False), + sa.Column("producer_ref", sa.String(120), nullable=False), + sa.Column("project_id", sa.String(36), nullable=False), + sa.Column("task_id", sa.String(36), nullable=True), + sa.Column("guide_source_item_id", sa.String(36), nullable=True), + sa.Column("upload_item_id", sa.String(36), nullable=True), + sa.Column("checker_run_id", sa.String(36), nullable=True), + sa.Column("logical_role", sa.String(100), nullable=True), + sa.Column("sha256", sa.String(71), nullable=False), + sa.Column("byte_count", sa.BigInteger(), nullable=False), + sa.Column("media_type", sa.String(255), nullable=False), + sa.Column("storage_namespace_id", sa.String(20), nullable=False), + sa.Column("namespace_fingerprint", sa.String(71), nullable=False), + sa.Column("canonical_target", sa.String(1024), nullable=False), + sa.Column("operation_identity", sa.String(71), nullable=False), + sa.Column("request_digest", sa.String(71), nullable=False), + sa.Column("status", sa.String(40), nullable=False, server_default="prepared"), + sa.Column( + "next_run_at", + sa.DateTime(timezone=True), + nullable=True, + ), + sa.Column("executor_id", sa.String(36), nullable=True), + sa.Column("lease_expires_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("execution_generation", sa.BigInteger(), nullable=False, server_default="0"), + sa.Column("terminal_result_code", sa.String(100), nullable=True), + sa.Column("replica_id", sa.String(36), nullable=True), + sa.Column("receipt_id", sa.String(36), nullable=True), + sa.Column("cas_version", sa.BigInteger(), nullable=False, server_default="0"), + sa.Column( + "prepared_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.Column("terminal_at", sa.DateTime(timezone=True), nullable=True), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.Column( + "updated_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.CheckConstraint( + "producer_request_type in ('guide', 'contributor', 'checker_output')", + name="producer_request_type", + ), + sa.CheckConstraint( + "producer_type in ('actor_profile', 'service_identity')", + name="producer_type", + ), + sa.CheckConstraint( + "((producer_request_type in ('guide', 'contributor') " + "and producer_type = 'actor_profile' and " + "producer_ref ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-" + "[89ab][0-9a-f]{3}-[0-9a-f]{12}$') or " + "(producer_request_type = 'checker_output' " + "and producer_type = 'service_identity' " + "and producer_ref = 'workstream.artifact.checker_output'))", + name="producer_identity", + ), + sa.CheckConstraint("sha256 ~ '^sha256:[0-9a-f]{64}$'", name="sha256_shape"), + sa.CheckConstraint("byte_count >= 0", name="byte_count_nonnegative"), + sa.CheckConstraint( + "canonical_target ~ '^sha256/[0-9a-f]{2}/[0-9a-f]{62}$'", + name="canonical_target_shape", + ), + sa.CheckConstraint( + "operation_identity ~ '^sha256:[0-9a-f]{64}$'", + name="operation_identity_shape", + ), + sa.CheckConstraint( + "request_digest ~ '^sha256:[0-9a-f]{64}$'", + name="request_digest_shape", + ), + sa.CheckConstraint( + "status in ('prepared', 'put_in_flight', 'acknowledgement_unknown', " + "'object_confirmed', 'absent_replay_required', 'integrity_mismatch', " + "'provider_unavailable', 'conflict')", + name="status", + ), + sa.CheckConstraint( + "(executor_id is null) = (lease_expires_at is null)", + name="executor_lease_pair", + ), + sa.CheckConstraint( + "execution_generation >= 0 and cas_version >= 0", + name="versions_nonnegative", + ), + sa.CheckConstraint( + "status != 'prepared' or (next_run_at is null and executor_id is null " + "and lease_expires_at is null " + "and execution_generation = 0 and terminal_result_code is null " + "and terminal_at is null and replica_id is null and receipt_id is null)", + name="prepared_execution_inactive", + ), + sa.CheckConstraint( + "(producer_request_type = 'guide' and guide_source_item_id is not null " + "and upload_item_id is null and checker_run_id is null and task_id is null " + "and logical_role is null) or " + "(producer_request_type = 'contributor' and guide_source_item_id is null " + "and upload_item_id is not null and checker_run_id is null " + "and task_id is not null and logical_role is null) or " + "(producer_request_type = 'checker_output' and guide_source_item_id is null " + "and upload_item_id is null and checker_run_id is not null " + "and task_id is not null and octet_length(logical_role) between 1 and 100)", + name="producer_reference", + ), + sa.ForeignKeyConstraint( + ["storage_namespace_id", "namespace_fingerprint"], + [ + "artifact_storage_namespaces.id", + "artifact_storage_namespaces.namespace_fingerprint", + ], + name="fk_artifact_put_attempts_namespace_fingerprint", + ondelete="RESTRICT", + ), + sa.ForeignKeyConstraint(["project_id"], ["projects.id"], ondelete="RESTRICT"), + sa.ForeignKeyConstraint( + ["task_id"], ["workstream_tasks.id"], ondelete="RESTRICT" + ), + sa.ForeignKeyConstraint( + ["guide_source_item_id"], + ["guide_source_snapshot_items.id"], + ondelete="RESTRICT", + ), + sa.ForeignKeyConstraint( + ["upload_item_id"], ["artifact_upload_items.id"], ondelete="RESTRICT" + ), + sa.ForeignKeyConstraint( + ["checker_run_id"], ["checker_runs.id"], ondelete="RESTRICT" + ), + sa.ForeignKeyConstraint( + ["replica_id"], ["artifact_replicas.id"], ondelete="RESTRICT" + ), + sa.ForeignKeyConstraint( + ["receipt_id"], ["artifact_operation_receipts.id"], ondelete="RESTRICT" + ), + sa.PrimaryKeyConstraint("id"), + sa.UniqueConstraint( + "operation_identity", + name="uq_artifact_put_attempt_operation", + ), + ) + for column in ( + "project_id", + "task_id", + "guide_source_item_id", + "upload_item_id", + "checker_run_id", + "status", + "next_run_at", + "replica_id", + "receipt_id", + ): + op.create_index( + f"ix_artifact_put_attempts_{column}", + "artifact_put_attempts", + [column], + ) + + op.create_table( + "artifact_put_attempt_charges", + sa.Column("attempt_id", sa.String(36), nullable=False), + sa.Column("charge_id", sa.String(36), nullable=False), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.func.now(), + ), + sa.ForeignKeyConstraint( + ["attempt_id"], ["artifact_put_attempts.id"], ondelete="RESTRICT" + ), + sa.ForeignKeyConstraint( + ["charge_id"], ["artifact_admission_charges.id"], ondelete="RESTRICT" + ), + sa.PrimaryKeyConstraint("attempt_id", "charge_id"), + ) + + +def downgrade() -> None: + """Remove only an empty admission foundation.""" + _refuse_populated_admission_downgrade() + op.drop_table("artifact_put_attempt_charges") + for column in reversed( + ( + "project_id", + "task_id", + "guide_source_item_id", + "upload_item_id", + "checker_run_id", + "status", + "next_run_at", + "replica_id", + "receipt_id", + ) + ): + op.drop_index( + f"ix_artifact_put_attempts_{column}", + table_name="artifact_put_attempts", + ) + op.drop_table("artifact_put_attempts") + op.drop_table("artifact_admission_charges") + op.drop_table("artifact_admission_scopes") + op.drop_constraint( + "uq_artifact_storage_namespace_id_fingerprint", + "artifact_storage_namespaces", + type_="unique", + ) diff --git a/backend/app/adapters/artifacts/local.py b/backend/app/adapters/artifacts/local.py index e48891477..0d705344e 100644 --- a/backend/app/adapters/artifacts/local.py +++ b/backend/app/adapters/artifacts/local.py @@ -20,10 +20,6 @@ await_cancellation_resistant, run_blocking_cancellation_resistant, ) -from app.adapters.artifacts.references import ( - artifact_provider_object_ref, - parse_artifact_provider_object_ref, -) from app.core.file_locks import acquire_exclusive_file_lock from app.interfaces.artifacts import ( ARTIFACT_STORE_CAPABILITY_KEY, @@ -42,7 +38,9 @@ ArtifactStoreNamespaceClaim, ArtifactStoreNamespaceIdentity, ArtifactStoreUnavailableError, + artifact_provider_object_ref, artifact_store_namespace_material, + parse_artifact_provider_object_ref, ) from app.interfaces.external_services import ExternalServiceAdapterIdentity from app.core.hashing import canonical_json_hash diff --git a/backend/app/adapters/artifacts/references.py b/backend/app/adapters/artifacts/references.py deleted file mode 100644 index 3ac7cd5b1..000000000 --- a/backend/app/adapters/artifacts/references.py +++ /dev/null @@ -1,29 +0,0 @@ -"""Canonical provider-object references shared by artifact adapters.""" - -from __future__ import annotations - -import re - -from app.interfaces.artifacts import ArtifactOperationConflictError -from app.modules.artifacts.sources import ArtifactCommitment - - -_PROVIDER_OBJECT_REF = re.compile(r"^sha256/([0-9a-f]{2})/([0-9a-f]{62})$") - - -def artifact_provider_object_ref(commitment: ArtifactCommitment) -> str: - """Derive one identity-free reference solely from a server commitment.""" - if type(commitment) is not ArtifactCommitment: - raise ArtifactOperationConflictError("artifact commitment is invalid") - digest_hex = commitment.sha256[7:] - return f"sha256/{digest_hex[:2]}/{digest_hex[2:]}" - - -def parse_artifact_provider_object_ref(provider_object_ref: str) -> tuple[str, str]: - """Return digest path parts after enforcing the canonical grammar.""" - if not isinstance(provider_object_ref, str): - raise ArtifactOperationConflictError("artifact provider reference is invalid") - matched = _PROVIDER_OBJECT_REF.fullmatch(provider_object_ref) - if matched is None: - raise ArtifactOperationConflictError("artifact provider reference is invalid") - return matched.group(1), matched.group(2) diff --git a/backend/app/adapters/artifacts/s3_compatible.py b/backend/app/adapters/artifacts/s3_compatible.py index ff5262d32..76269d39f 100644 --- a/backend/app/adapters/artifacts/s3_compatible.py +++ b/backend/app/adapters/artifacts/s3_compatible.py @@ -46,10 +46,6 @@ get_current_datetime, ) -from app.adapters.artifacts.references import ( - artifact_provider_object_ref, - parse_artifact_provider_object_ref, -) from app.core.config import Settings from app.core.hashing import canonical_json_hash from app.core.s3_validation import ( @@ -75,7 +71,9 @@ ArtifactStoreNamespaceClaim, ArtifactStoreNamespaceIdentity, ArtifactStoreUnavailableError, + artifact_provider_object_ref, artifact_store_namespace_material, + parse_artifact_provider_object_ref, ) from app.interfaces.external_services import ExternalServiceAdapterIdentity from app.modules.artifacts.preparation import HARD_MAXIMUM_ARTIFACT_BYTES diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 3fa52fc46..b99828474 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -149,6 +149,10 @@ class Settings(BaseSettings): ) artifact_s3_max_pool_connections: int = Field(default=16, ge=1, le=256) artifact_maximum_bytes: int = Field(default=512 * 1024 * 1024, gt=0) + artifact_admission_task_maximum_bytes: int | None = Field(default=None, gt=0) + artifact_admission_producer_maximum_bytes: int | None = Field(default=None, gt=0) + artifact_admission_project_maximum_bytes: int | None = Field(default=None, gt=0) + artifact_admission_deployment_maximum_bytes: int | None = Field(default=None, gt=0) artifact_stream_buffer_bytes: int = Field(default=1024 * 1024, gt=0, le=1024 * 1024) artifact_operation_lock_timeout_seconds: float = Field( default=1800.0, @@ -433,6 +437,16 @@ def validate_artifact_storage(self) -> Settings: ) if self.artifact_store_backend == "disabled": return self + if any( + value is None + for value in ( + self.artifact_admission_task_maximum_bytes, + self.artifact_admission_producer_maximum_bytes, + self.artifact_admission_project_maximum_bytes, + self.artifact_admission_deployment_maximum_bytes, + ) + ): + raise ValueError("enabled artifact storage requires all durable-byte admission limits") if self.artifact_scratch_root is None: raise ValueError("enabled artifact storage requires an artifact scratch root") if self.artifact_store_backend == "local": diff --git a/backend/app/interfaces/artifacts.py b/backend/app/interfaces/artifacts.py index 2dd8e11fa..ea182332e 100644 --- a/backend/app/interfaces/artifacts.py +++ b/backend/app/interfaces/artifacts.py @@ -4,6 +4,7 @@ from collections.abc import AsyncIterator from dataclasses import dataclass +import re from typing import Protocol from app.core.hashing import canonical_json_hash @@ -17,6 +18,9 @@ ARTIFACT_STORE_CAPABILITY_KEY = "artifact_store" _MAXIMUM_PROVIDER_OBJECT_REF_LENGTH = 1024 +_CANONICAL_PROVIDER_OBJECT_REF = re.compile( + r"^sha256/([0-9a-f]{2})/([0-9a-f]{62})$" +) _RESERVED_NAMESPACE_DESCRIPTOR_KEYS = frozenset( {"adapter", "backend", "provider_profile"} ) @@ -262,6 +266,24 @@ class ArtifactOperationConflictError(ArtifactStoreError): category = "conflict" +def artifact_provider_object_ref(commitment: ArtifactCommitment) -> str: + """Derive one identity-free provider reference from a commitment.""" + if type(commitment) is not ArtifactCommitment: + raise ArtifactOperationConflictError("artifact commitment is invalid") + digest_hex = commitment.sha256[7:] + return f"sha256/{digest_hex[:2]}/{digest_hex[2:]}" + + +def parse_artifact_provider_object_ref(provider_object_ref: str) -> tuple[str, str]: + """Return digest path parts after enforcing the canonical grammar.""" + if not isinstance(provider_object_ref, str): + raise ArtifactOperationConflictError("artifact provider reference is invalid") + matched = _CANONICAL_PROVIDER_OBJECT_REF.fullmatch(provider_object_ref) + if matched is None: + raise ArtifactOperationConflictError("artifact provider reference is invalid") + return matched.group(1), matched.group(2) + + class ArtifactLimitExceededError(ArtifactStoreError): """Raised when bounded artifact input exceeds its hard limit.""" diff --git a/backend/app/modules/actors/repository.py b/backend/app/modules/actors/repository.py index 6bad7db1c..487c707d6 100644 --- a/backend/app/modules/actors/repository.py +++ b/backend/app/modules/actors/repository.py @@ -92,6 +92,20 @@ async def get_actor_profile( query = query.with_for_update() return await self._session.scalar(query.execution_options(populate_existing=True)) + async def lock_exact_actor_identity( + self, + actor_profile_id: str, + identity_link_id: str, + ) -> tuple[ActorProfile, ActorIdentityLink] | None: + """Lock an exact canonical profile then its requested identity link.""" + profile = await self.get_actor_profile(actor_profile_id, for_update=True) + if profile is None: + return None + link = await self.get_identity_link_by_id(identity_link_id, for_update=True) + if link is None or link.actor_profile_id != actor_profile_id: + return None + return profile, link + async def get_service_actor( self, service_identity: str, diff --git a/backend/app/modules/actors/service.py b/backend/app/modules/actors/service.py index f22daf460..17fcfc078 100644 --- a/backend/app/modules/actors/service.py +++ b/backend/app/modules/actors/service.py @@ -96,6 +96,19 @@ class ResolvedActor: identity_link: ActorIdentityLink +@dataclass(frozen=True, slots=True) +class ActorAdmissionProof: + """Primitive canonical actor/link state locked for a caller transaction.""" + + actor_profile_id: str + actor_kind: str + actor_status: str + service_identity: str | None + identity_link_id: str + identity_link_subject_kind: str + identity_link_status: str + + class ActorService: """Resolve canonical actors and own first-human provisioning transactions.""" @@ -105,6 +118,29 @@ def __init__(self, session: AsyncSession) -> None: self._audit = AuditService(session) self._legacy_audit = AuditRepository(session) + async def lock_admission_proof( + self, + actor_profile_id: UUID, + identity_link_id: UUID, + ) -> ActorAdmissionProof | None: + """Lock and project one exact actor/link pair without deciding authority.""" + locked = await self._repo.lock_exact_actor_identity( + str(actor_profile_id), + str(identity_link_id), + ) + if locked is None: + return None + profile, link = locked + return ActorAdmissionProof( + actor_profile_id=profile.id, + actor_kind=profile.actor_kind, + actor_status=profile.status, + service_identity=profile.service_identity, + identity_link_id=link.id, + identity_link_subject_kind=link.subject_kind, + identity_link_status=link.status, + ) + async def find_verified_actor(self, token: VerifiedIssuerToken) -> ResolvedActor | None: """Return a canonical actor for an existing exact identity link.""" resolved = await self.find_actor_for_authorization(token) diff --git a/backend/app/modules/artifacts/models.py b/backend/app/modules/artifacts/models.py index 2f03ca045..0f005a698 100644 --- a/backend/app/modules/artifacts/models.py +++ b/backend/app/modules/artifacts/models.py @@ -5,9 +5,11 @@ from datetime import datetime from sqlalchemy import ( + BigInteger, CheckConstraint, DateTime, ForeignKey, + ForeignKeyConstraint, Index, Integer, JSON, @@ -21,6 +23,10 @@ SHA256_CHECK = "{column} ~ '^sha256:[0-9a-f]{{64}}$'" +UUID_CHECK = ( + "{column} ~ '^[0-9a-f]{{8}}-[0-9a-f]{{4}}-[1-5][0-9a-f]{{3}}-" + "[89ab][0-9a-f]{{3}}-[0-9a-f]{{12}}$'" +) class ArtifactUploadSession(Base): @@ -213,6 +219,11 @@ class ArtifactStorageNamespace(Base): "namespace_fingerprint", name="uq_artifact_storage_namespace_fingerprint", ), + UniqueConstraint( + "id", + "namespace_fingerprint", + name="uq_artifact_storage_namespace_id_fingerprint", + ), ) id: Mapped[str] = mapped_column(String(20), primary_key=True) @@ -224,6 +235,242 @@ class ArtifactStorageNamespace(Base): created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) +class ArtifactAdmissionScope(Base): + """Serialized durable-byte usage for one canonical admission scope.""" + + __tablename__ = "artifact_admission_scopes" + __table_args__ = ( + CheckConstraint( + "scope_type in ('deployment', 'project', 'producer', 'task')", + name="scope_type", + ), + CheckConstraint("octet_length(scope_id) between 1 and 120", name="scope_id_bounds"), + CheckConstraint("limit_bytes > 0", name="limit_positive"), + CheckConstraint( + "counted_bytes >= 0 and counted_bytes <= limit_bytes", + name="counted_bytes_within_limit", + ), + CheckConstraint("cas_version >= 0", name="cas_nonnegative"), + ) + + scope_type: Mapped[str] = mapped_column(String(20), primary_key=True) + scope_id: Mapped[str] = mapped_column(String(120), primary_key=True) + limit_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False) + counted_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0) + cas_version: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), server_default=func.now(), onupdate=func.now() + ) + + +class ArtifactAdmissionCharge(Base): + """CAS-protected unique-byte charge for one scope and content identity.""" + + __tablename__ = "artifact_admission_charges" + __table_args__ = ( + ForeignKeyConstraint( + ["scope_type", "scope_id"], + ["artifact_admission_scopes.scope_type", "artifact_admission_scopes.scope_id"], + ondelete="RESTRICT", + name="fk_artifact_admission_charges_scope", + ), + UniqueConstraint( + "scope_type", + "scope_id", + "sha256", + "byte_count", + name="uq_artifact_admission_charge_scope_content", + ), + CheckConstraint(SHA256_CHECK.format(column="sha256"), name="sha256_shape"), + CheckConstraint("byte_count >= 0", name="byte_count_nonnegative"), + CheckConstraint( + "producer_type in ('actor_profile', 'service_identity')", + name="producer_type", + ), + CheckConstraint( + SHA256_CHECK.format(column="creating_operation_identity"), + name="operation_identity_shape", + ), + CheckConstraint( + "state in ('provisional', 'completed', 'released')", + name="state", + ), + CheckConstraint("cas_version >= 0", name="cas_nonnegative"), + CheckConstraint( + "(state = 'completed') = (completed_at is not null)", + name="completed_timestamp", + ), + CheckConstraint( + "(state = 'released') = (released_at is not null)", + name="released_timestamp", + ), + ) + + id: Mapped[str] = mapped_column(String(36), primary_key=True) + scope_type: Mapped[str] = mapped_column(String(20), nullable=False) + scope_id: Mapped[str] = mapped_column(String(120), nullable=False) + sha256: Mapped[str] = mapped_column(String(71), nullable=False) + byte_count: Mapped[int] = mapped_column(BigInteger, nullable=False) + producer_type: Mapped[str] = mapped_column(String(30), nullable=False) + producer_ref: Mapped[str] = mapped_column(String(120), nullable=False) + creating_operation_identity: Mapped[str] = mapped_column(String(71), nullable=False) + state: Mapped[str] = mapped_column(String(20), nullable=False, default="provisional") + cas_version: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0) + reserved_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), nullable=False, server_default=func.now() + ) + completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) + released_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), server_default=func.now(), onupdate=func.now() + ) + + +class ArtifactPutAttempt(Base): + """Durable pre-I/O commitment created only after complete admission.""" + + __tablename__ = "artifact_put_attempts" + __table_args__ = ( + ForeignKeyConstraint( + ["storage_namespace_id", "namespace_fingerprint"], + ["artifact_storage_namespaces.id", "artifact_storage_namespaces.namespace_fingerprint"], + ondelete="RESTRICT", + name="fk_artifact_put_attempts_namespace_fingerprint", + ), + UniqueConstraint("operation_identity", name="uq_artifact_put_attempt_operation"), + CheckConstraint( + "producer_request_type in ('guide', 'contributor', 'checker_output')", + name="producer_request_type", + ), + CheckConstraint( + "producer_type in ('actor_profile', 'service_identity')", + name="producer_type", + ), + CheckConstraint( + "((producer_request_type in ('guide', 'contributor') " + "and producer_type = 'actor_profile' and " + + UUID_CHECK.format(column="producer_ref") + + ") or (producer_request_type = 'checker_output' " + "and producer_type = 'service_identity' " + "and producer_ref = 'workstream.artifact.checker_output'))", + name="producer_identity", + ), + CheckConstraint(SHA256_CHECK.format(column="sha256"), name="sha256_shape"), + CheckConstraint("byte_count >= 0", name="byte_count_nonnegative"), + CheckConstraint( + "canonical_target ~ '^sha256/[0-9a-f]{2}/[0-9a-f]{62}$'", + name="canonical_target_shape", + ), + CheckConstraint( + SHA256_CHECK.format(column="operation_identity"), + name="operation_identity_shape", + ), + CheckConstraint( + SHA256_CHECK.format(column="request_digest"), + name="request_digest_shape", + ), + CheckConstraint( + "status in ('prepared', 'put_in_flight', 'acknowledgement_unknown', " + "'object_confirmed', 'absent_replay_required', 'integrity_mismatch', " + "'provider_unavailable', 'conflict')", + name="status", + ), + CheckConstraint( + "(executor_id is null) = (lease_expires_at is null)", + name="executor_lease_pair", + ), + CheckConstraint( + "execution_generation >= 0 and cas_version >= 0", + name="versions_nonnegative", + ), + CheckConstraint( + "status != 'prepared' or (next_run_at is null and executor_id is null " + "and lease_expires_at is null " + "and execution_generation = 0 and terminal_result_code is null " + "and terminal_at is null and replica_id is null and receipt_id is null)", + name="prepared_execution_inactive", + ), + CheckConstraint( + "(producer_request_type = 'guide' and guide_source_item_id is not null " + "and upload_item_id is null and checker_run_id is null and task_id is null " + "and logical_role is null) or " + "(producer_request_type = 'contributor' and guide_source_item_id is null " + "and upload_item_id is not null and checker_run_id is null and task_id is not null " + "and logical_role is null) or " + "(producer_request_type = 'checker_output' and guide_source_item_id is null " + "and upload_item_id is null and checker_run_id is not null and task_id is not null " + "and octet_length(logical_role) between 1 and 100)", + name="producer_reference", + ), + ) + + id: Mapped[str] = mapped_column(String(36), primary_key=True) + producer_request_type: Mapped[str] = mapped_column(String(30), nullable=False) + producer_type: Mapped[str] = mapped_column(String(30), nullable=False) + producer_ref: Mapped[str] = mapped_column(String(120), nullable=False) + project_id: Mapped[str] = mapped_column( + ForeignKey("projects.id", ondelete="RESTRICT"), nullable=False, index=True + ) + task_id: Mapped[str | None] = mapped_column( + ForeignKey("workstream_tasks.id", ondelete="RESTRICT"), index=True + ) + guide_source_item_id: Mapped[str | None] = mapped_column( + ForeignKey("guide_source_snapshot_items.id", ondelete="RESTRICT"), index=True + ) + upload_item_id: Mapped[str | None] = mapped_column( + ForeignKey("artifact_upload_items.id", ondelete="RESTRICT"), index=True + ) + checker_run_id: Mapped[str | None] = mapped_column( + ForeignKey("checker_runs.id", ondelete="RESTRICT"), index=True + ) + logical_role: Mapped[str | None] = mapped_column(String(100)) + sha256: Mapped[str] = mapped_column(String(71), nullable=False) + byte_count: Mapped[int] = mapped_column(BigInteger, nullable=False) + media_type: Mapped[str] = mapped_column(String(255), nullable=False) + storage_namespace_id: Mapped[str] = mapped_column(String(20), nullable=False) + namespace_fingerprint: Mapped[str] = mapped_column(String(71), nullable=False) + canonical_target: Mapped[str] = mapped_column(String(1024), nullable=False) + operation_identity: Mapped[str] = mapped_column(String(71), nullable=False) + request_digest: Mapped[str] = mapped_column(String(71), nullable=False) + status: Mapped[str] = mapped_column(String(40), nullable=False, default="prepared", index=True) + next_run_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), index=True) + executor_id: Mapped[str | None] = mapped_column(String(36)) + lease_expires_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) + execution_generation: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0) + terminal_result_code: Mapped[str | None] = mapped_column(String(100)) + replica_id: Mapped[str | None] = mapped_column( + ForeignKey("artifact_replicas.id", ondelete="RESTRICT"), index=True + ) + receipt_id: Mapped[str | None] = mapped_column( + ForeignKey("artifact_operation_receipts.id", ondelete="RESTRICT"), index=True + ) + cas_version: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0) + prepared_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), nullable=False, server_default=func.now() + ) + terminal_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True)) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), server_default=func.now(), onupdate=func.now() + ) + + +class ArtifactPutAttemptCharge(Base): + """Immutable link from one put attempt to every required scope charge.""" + + __tablename__ = "artifact_put_attempt_charges" + + attempt_id: Mapped[str] = mapped_column( + ForeignKey("artifact_put_attempts.id", ondelete="RESTRICT"), primary_key=True + ) + charge_id: Mapped[str] = mapped_column( + ForeignKey("artifact_admission_charges.id", ondelete="RESTRICT"), primary_key=True + ) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) + + class ArtifactReplica(Base): """Provider observation record for one immutable content object.""" diff --git a/backend/app/modules/artifacts/repository.py b/backend/app/modules/artifacts/repository.py index a27103365..3a4028ac7 100644 --- a/backend/app/modules/artifacts/repository.py +++ b/backend/app/modules/artifacts/repository.py @@ -2,18 +2,64 @@ from __future__ import annotations -from sqlalchemy import select +from collections.abc import Sequence +from dataclasses import dataclass +from datetime import datetime + +from sqlalchemy import func, select, tuple_ from sqlalchemy.dialects.postgresql import insert from sqlalchemy.ext.asyncio import AsyncSession from app.modules.artifacts.models import ( + ArtifactAdmissionCharge, + ArtifactAdmissionScope, ArtifactContent, ArtifactOperationReceipt, + ArtifactPutAttempt, + ArtifactPutAttemptCharge, ArtifactReplica, ArtifactStorageNamespace, ArtifactUploadItem, ArtifactUploadSession, ) +from app.modules.checkers.models import CheckerRun +from app.modules.projects.models import GuideSourceSnapshot, GuideSourceSnapshotItem +from app.modules.tasks.models import Submission, WorkstreamTask + + +@dataclass(frozen=True, slots=True) +class GuideAdmissionFacts: + """Authoritative project ownership for one guide source item.""" + + guide_source_item_id: str + project_id: str + captured_by: str + content_hash: str + media_type: str + + +@dataclass(frozen=True, slots=True) +class ContributorAdmissionFacts: + """Authoritative upload-item ownership and state.""" + + upload_item_id: str + project_id: str + task_id: str | None + actor_profile_id: str + session_state: str + item_state: str + expected_sha256: str | None + expected_size: int | None + media_type: str | None + + +@dataclass(frozen=True, slots=True) +class CheckerOutputAdmissionFacts: + """Authoritative project/task ownership for one checker run.""" + + checker_run_id: str + project_id: str + task_id: str class ArtifactRepository: @@ -23,6 +69,13 @@ def __init__(self, session: AsyncSession) -> None: """Bind the repository to one async database session.""" self._session = session + async def database_now(self) -> datetime: + """Return the PostgreSQL clock for admission timestamps.""" + value = await self._session.scalar(select(func.clock_timestamp())) + if value is None: + raise RuntimeError("PostgreSQL clock did not return a timestamp") + return value + async def lock_upload_item(self, item_id: str) -> ArtifactUploadItem | None: """Load one upload item with a row lock.""" result = await self._session.execute( @@ -41,6 +94,210 @@ async def lock_upload_session(self, session_id: str) -> ArtifactUploadSession | ) return result.scalar_one_or_none() + async def get_guide_admission_facts( + self, guide_source_item_id: str + ) -> GuideAdmissionFacts | None: + """Load canonical project ownership for one guide source item.""" + row = ( + await self._session.execute( + select( + GuideSourceSnapshotItem.id, + GuideSourceSnapshot.project_id, + GuideSourceSnapshot.captured_by, + GuideSourceSnapshotItem.content_hash, + GuideSourceSnapshotItem.media_type, + ) + .join( + GuideSourceSnapshot, + GuideSourceSnapshot.id == GuideSourceSnapshotItem.source_snapshot_id, + ) + .where(GuideSourceSnapshotItem.id == guide_source_item_id) + .with_for_update( + of=(GuideSourceSnapshotItem, GuideSourceSnapshot) + ) + ) + ).one_or_none() + if row is None: + return None + return GuideAdmissionFacts( + guide_source_item_id=row.id, + project_id=row.project_id, + captured_by=row.captured_by, + content_hash=row.content_hash, + media_type=row.media_type, + ) + + async def get_contributor_admission_facts( + self, upload_item_id: str + ) -> ContributorAdmissionFacts | None: + """Load canonical contributor upload ownership and state.""" + row = ( + await self._session.execute( + select( + ArtifactUploadItem.id, + WorkstreamTask.project_id, + WorkstreamTask.id.label("task_id"), + ArtifactUploadSession.actor_id, + ArtifactUploadSession.state.label("session_state"), + ArtifactUploadItem.state.label("item_state"), + ArtifactUploadItem.expected_sha256, + ArtifactUploadItem.expected_size, + ArtifactUploadItem.media_type, + ) + .join( + ArtifactUploadSession, + ArtifactUploadSession.id == ArtifactUploadItem.session_id, + ) + .join( + WorkstreamTask, + (WorkstreamTask.id == ArtifactUploadSession.task_id) + & (WorkstreamTask.project_id == ArtifactUploadSession.project_id), + ) + .where(ArtifactUploadItem.id == upload_item_id) + .with_for_update( + of=(ArtifactUploadSession, ArtifactUploadItem, WorkstreamTask) + ) + ) + ).one_or_none() + if row is None: + return None + return ContributorAdmissionFacts( + upload_item_id=row.id, + project_id=row.project_id, + task_id=row.task_id, + actor_profile_id=row.actor_id, + session_state=row.session_state, + item_state=row.item_state, + expected_sha256=row.expected_sha256, + expected_size=row.expected_size, + media_type=row.media_type, + ) + + async def get_checker_output_admission_facts( + self, checker_run_id: str + ) -> CheckerOutputAdmissionFacts | None: + """Load canonical project/task ownership for one checker run.""" + row = ( + await self._session.execute( + select(CheckerRun.id, Submission.task_id, WorkstreamTask.project_id) + .join( + Submission, + (Submission.id == CheckerRun.submission_id) + & (Submission.version == CheckerRun.submission_version) + & (Submission.task_id == CheckerRun.task_id), + ) + .join(WorkstreamTask, WorkstreamTask.id == Submission.task_id) + .where(CheckerRun.id == checker_run_id) + .with_for_update(of=(CheckerRun, Submission, WorkstreamTask)) + ) + ).one_or_none() + if row is None: + return None + return CheckerOutputAdmissionFacts( + checker_run_id=row.id, + project_id=row.project_id, + task_id=row.task_id, + ) + + async def ensure_and_lock_admission_scopes( + self, + scopes: Sequence[tuple[str, str, int]], + ) -> tuple[ArtifactAdmissionScope, ...]: + """Create missing counters, then lock every scope in canonical order.""" + values = [ + { + "scope_type": scope_type, + "scope_id": scope_id, + "limit_bytes": limit_bytes, + "counted_bytes": 0, + "cas_version": 0, + } + for scope_type, scope_id, limit_bytes in scopes + ] + await self._session.execute( + insert(ArtifactAdmissionScope) + .values(values) + .on_conflict_do_nothing( + index_elements=[ + ArtifactAdmissionScope.scope_type, + ArtifactAdmissionScope.scope_id, + ] + ) + ) + keys = [(scope_type, scope_id) for scope_type, scope_id, _ in scopes] + result = await self._session.execute( + select(ArtifactAdmissionScope) + .where( + tuple_( + ArtifactAdmissionScope.scope_type, + ArtifactAdmissionScope.scope_id, + ).in_(keys) + ) + .order_by(ArtifactAdmissionScope.scope_type, ArtifactAdmissionScope.scope_id) + .with_for_update() + ) + return tuple(result.scalars().all()) + + async def get_admission_charge( + self, + *, + scope_type: str, + scope_id: str, + sha256: str, + byte_count: int, + ) -> ArtifactAdmissionCharge | None: + """Load one exact scope/content charge while its scope is locked.""" + return await self._session.scalar( + select(ArtifactAdmissionCharge).where( + ArtifactAdmissionCharge.scope_type == scope_type, + ArtifactAdmissionCharge.scope_id == scope_id, + ArtifactAdmissionCharge.sha256 == sha256, + ArtifactAdmissionCharge.byte_count == byte_count, + ) + ) + + async def add_admission_charge( + self, charge: ArtifactAdmissionCharge + ) -> ArtifactAdmissionCharge: + """Flush one new charge under its locked scope counter.""" + self._session.add(charge) + await self._session.flush() + return charge + + async def get_put_attempt_by_operation( + self, operation_identity: str + ) -> ArtifactPutAttempt | None: + """Load the durable attempt for one canonical operation identity.""" + return await self._session.scalar( + select(ArtifactPutAttempt).where( + ArtifactPutAttempt.operation_identity == operation_identity + ) + ) + + async def add_put_attempt( + self, + attempt: ArtifactPutAttempt, + charges: Sequence[ArtifactAdmissionCharge], + ) -> ArtifactPutAttempt: + """Flush one attempt and its complete charge links in this transaction.""" + self._session.add(attempt) + await self._session.flush() + self._session.add_all( + ArtifactPutAttemptCharge(attempt_id=attempt.id, charge_id=charge.id) + for charge in charges + ) + await self._session.flush() + return attempt + + async def list_put_attempt_charge_ids(self, attempt_id: str) -> tuple[str, ...]: + """Return one attempt's charge IDs in stable order.""" + result = await self._session.execute( + select(ArtifactPutAttemptCharge.charge_id) + .where(ArtifactPutAttemptCharge.attempt_id == attempt_id) + .order_by(ArtifactPutAttemptCharge.charge_id) + ) + return tuple(result.scalars().all()) + async def get_or_create_content(self, content: ArtifactContent) -> ArtifactContent: """Return the immutable content fact for one digest and size.""" await self._session.execute( diff --git a/backend/app/modules/artifacts/schemas.py b/backend/app/modules/artifacts/schemas.py new file mode 100644 index 000000000..5370e1cef --- /dev/null +++ b/backend/app/modules/artifacts/schemas.py @@ -0,0 +1,61 @@ +"""Closed internal contracts for durable artifact admission.""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import TypeAlias, final +from uuid import UUID + +from app.modules.artifacts.sources import CommittedArtifactSource +from app.modules.authorization.runtime import AuthorizationContext + + +@final +@dataclass(frozen=True, slots=True) +class GuideArtifactAdmissionRequest: + """One prepared guide source item admitted under its canonical project.""" + + authorization_context: AuthorizationContext + guide_source_item_id: UUID + source: CommittedArtifactSource + + +@final +@dataclass(frozen=True, slots=True) +class ContributorArtifactAdmissionRequest: + """One prepared contributor item admitted under its upload session.""" + + authorization_context: AuthorizationContext + upload_item_id: UUID + source: CommittedArtifactSource + + +@final +@dataclass(frozen=True, slots=True) +class CheckerOutputArtifactAdmissionRequest: + """One prepared checker output admitted under its exact checker run.""" + + authorization_context: AuthorizationContext + checker_run_id: UUID + logical_role: str + source: CommittedArtifactSource + + +ArtifactAdmissionRequest: TypeAlias = ( + GuideArtifactAdmissionRequest + | ContributorArtifactAdmissionRequest + | CheckerOutputArtifactAdmissionRequest +) + + +@final +@dataclass(frozen=True, slots=True) +class ArtifactAdmissionResult: + """Committed pre-I/O attempt and its complete admission-charge set.""" + + attempt_id: UUID + status: str + operation_identity: str + request_digest: str + charge_ids: tuple[UUID, ...] + replayed: bool diff --git a/backend/app/modules/artifacts/service.py b/backend/app/modules/artifacts/service.py index 72e1903da..dd5297b94 100644 --- a/backend/app/modules/artifacts/service.py +++ b/backend/app/modules/artifacts/service.py @@ -2,36 +2,45 @@ from __future__ import annotations -import asyncio from dataclasses import dataclass -from uuid import uuid4 +from uuid import UUID, uuid4 from sqlalchemy.ext.asyncio import AsyncSession from app.core.config import Settings -from app.core.cancellation import await_cancellation_resistant +from app.core.hashing import canonical_json_hash from app.db.session import get_session_factory from app.interfaces.artifacts import ( - ArtifactInputMismatchError, - ArtifactIntegrityError, - ArtifactPutResult, ArtifactStore, ArtifactStoreBootstrap, - ArtifactStoreError, ArtifactStoreNamespaceClaim, + artifact_provider_object_ref, artifact_store_namespace_material, ) from app.interfaces.external_services import ExternalServiceAdapterIdentity +from app.modules.actors.service import ActorService +from app.modules.actors.service_identities import ServiceIdentity from app.modules.artifacts.models import ( - ArtifactContent, - ArtifactOperationReceipt, - ArtifactReplica, + ArtifactAdmissionCharge, + ArtifactAdmissionScope, + ArtifactPutAttempt, ArtifactStorageNamespace, - ArtifactUploadItem, - ArtifactUploadSession, ) from app.modules.artifacts.repository import ArtifactRepository -from app.modules.artifacts.sources import ArtifactCommitment, CommittedArtifactSource +from app.modules.artifacts.schemas import ( + ArtifactAdmissionRequest, + ArtifactAdmissionResult, + CheckerOutputArtifactAdmissionRequest, + ContributorArtifactAdmissionRequest, + GuideArtifactAdmissionRequest, +) +from app.modules.artifacts.sources import CommittedArtifactSource +from app.modules.authorization.runtime import ( + ActorKind, + ActorStatus, + AuthorizationContext, + IdentityLinkStatus, +) ARTIFACT_STORAGE_NAMESPACE_ID = "primary" @@ -56,11 +65,49 @@ class ArtifactStorageNamespaceSpec: namespace_fingerprint: str +class ArtifactAdmissionError(ArtifactIngestStateError): + """Base failure for durable-byte admission before provider I/O.""" + + +class ArtifactAdmissionConfigurationError(ArtifactAdmissionError): + """Raised when admission configuration is absent or has drifted.""" + + +class ArtifactAdmissionCapacityError(ArtifactAdmissionError): + """Raised when one required durable-byte scope lacks capacity.""" + + +class ArtifactAdmissionConflictError(ArtifactAdmissionError): + """Raised when an operation identity is replayed with changed input.""" + + +class ArtifactAdmissionRelationshipError(ArtifactAdmissionError): + """Raised when canonical producer ownership cannot be derived.""" + + @dataclass(frozen=True, slots=True) -class ProviderAttemptFence: - """Committed item CAS value fencing one provider call.""" +class _AdmissionScopeSpec: + """One server-derived scope and its exact configured byte limit.""" - item_cas: int + scope_type: str + scope_id: str + limit_bytes: int + + +@dataclass(frozen=True, slots=True) +class _AdmissionFacts: + """Canonical producer and product facts loaded for one closed request.""" + + request_type: str + producer_type: str + producer_ref: str + project_id: str + task_id: str | None + guide_source_item_id: str | None + upload_item_id: str | None + checker_run_id: str | None + logical_role: str | None + operation_identity: str def artifact_storage_namespace_spec( @@ -89,7 +136,7 @@ def artifact_storage_namespace_spec( class ArtifactStorageOrchestrator: - """Sole internal owner of the writable ArtifactStore capability.""" + """Dormant owner of writable storage until 02C2 adds attempt execution.""" def __init__( self, @@ -108,250 +155,472 @@ async def ensure_storage_namespace(self) -> ArtifactStorageNamespace: async with self._session.begin(): return await self._claim_and_validate_namespace() - async def put_reserved_item( - self, - item_id: str, - source: CommittedArtifactSource, - *, - correlation_id: str | None = None, - ) -> ArtifactPutResult: - """Publish one already-reserved internal item without activating product ingest.""" - commitment = source.commitment - fence = await self._start_put(item_id, commitment) - try: - result = await self._store.put(source) - except asyncio.CancelledError as cancellation: - try: - await await_cancellation_resistant( - self._mark_replay_required(item_id, fence) - ) - except BaseException: - raise cancellation from None - raise - except ArtifactStoreError as exc: - if exc.retryable: - await self._mark_replay_required(item_id, fence) - else: - await self._fail_put(item_id, exc.code, fence) - raise - except BaseException: - await self._mark_replay_required(item_id, fence) - raise - - try: - await self._finalize_put( - item_id, - commitment, - result, - fence, - correlation_id=correlation_id or str(uuid4()), - ) - except (ArtifactIntegrityError, ArtifactInputMismatchError) as exc: - await self._fail_put(item_id, exc.code, fence) - raise - except asyncio.CancelledError as cancellation: - try: - await await_cancellation_resistant( - self._mark_replay_required(item_id, fence) - ) - except BaseException: - raise cancellation from None - raise - except BaseException: - await self._mark_replay_required(item_id, fence) - raise - return result - - async def _start_put( - self, - item_id: str, - commitment: ArtifactCommitment, - ) -> ProviderAttemptFence: - """Transaction A validates namespace and exact reserved commitment.""" - async with self._session.begin(): - await self._claim_and_validate_namespace() - item = await self._repo.lock_upload_item(item_id) - if item is None: - raise ArtifactIngestStateError("artifact upload item does not exist") - upload_session = await self._repo.lock_upload_session(item.session_id) - if upload_session is None or upload_session.state != "open": - raise ArtifactIngestStateError("artifact upload session is not open") - if item.state not in {"reserved", "replay_required"}: - raise ArtifactIngestStateError("artifact upload item cannot be stored") - if ( - item.expected_sha256 != commitment.sha256 - or item.expected_size != commitment.byte_count - or item.media_type != commitment.media_type - or commitment.byte_count > item.reserved_bytes - ): - raise ArtifactIngestStateError("artifact upload commitment changed") - item.state = "uploading" - item.error_code = None - item.cas_version += 1 - upload_session.cas_version += 1 - return ProviderAttemptFence(item.cas_version) - - async def _finalize_put( + async def _claim_and_validate_namespace(self) -> ArtifactStorageNamespace: + """Atomically claim the singleton or reject deployment identity drift.""" + return await _claim_and_validate_storage_namespace(self._repo, self._namespace) + + +class ArtifactAdmissionService: + """Create one fully admitted put attempt without provider execution.""" + + def __init__( self, - item_id: str, - commitment: ArtifactCommitment, - result: ArtifactPutResult, - fence: ProviderAttemptFence, - *, - correlation_id: str, + session: AsyncSession, + settings: Settings, + namespace: ArtifactStorageNamespaceSpec, ) -> None: - """Transaction B records acknowledgement without making bytes bindable.""" + """Bind admission to one transaction owner and configured namespace.""" + self._session = session + self._settings = settings + self._namespace = namespace + self._repo = ArtifactRepository(session) + self._actors = ActorService(session) + + async def admit( + self, + request: ArtifactAdmissionRequest, + ) -> ArtifactAdmissionResult: + """Reserve every derived scope and persist one prepared attempt atomically.""" + self._validate_request_boundary(request) + commitment = request.source.commitment async with self._session.begin(): - namespace = await self._claim_and_validate_namespace() - item, upload_session = await self._locked_attempt(item_id, fence) - existing_receipt = await self._repo.get_receipt_for_item(item.id) - if existing_receipt is not None: - raise ArtifactIntegrityError("artifact upload item already has put evidence") - - content = await self._repo.get_or_create_content( - ArtifactContent( - id=str(uuid4()), - sha256=commitment.sha256, - byte_count=commitment.byte_count, - media_type=commitment.media_type, - normalized_display_name=item.display_name, - ) - ) - replica = await self._repo.get_or_create_replica( - ArtifactReplica( - id=str(uuid4()), - content_id=content.id, - storage_namespace_id=namespace.id, - namespace_fingerprint=namespace.namespace_fingerprint, - adapter=namespace.adapter, - provider_profile=namespace.provider_profile, - provider_object_ref=result.provider_object_ref, - verification_state="pending", - availability_state="unknown", - integrity_state="unknown", - ) + namespace = await _claim_and_validate_storage_namespace( + self._repo, + self._namespace, ) - if ( - replica.content_id != content.id - or replica.namespace_fingerprint != namespace.namespace_fingerprint - or replica.adapter != namespace.adapter - or replica.provider_profile != namespace.provider_profile - ): - raise ArtifactIntegrityError("provider object reference changed content identity") - - await self._repo.add_receipt( - ArtifactOperationReceipt( - id=str(uuid4()), - upload_item_id=item.id, - replica_id=replica.id, - operation="put", - idempotency_key=item.idempotency_key, - request_digest=item.request_digest, - provider_object_ref=result.provider_object_ref, - replayed=result.replayed, - outcome="stored_pending_verification", - attempt_number=1, - correlation_id=correlation_id, - details=[ + facts = await self._derive_admission_facts(request) + scopes = self._derive_scopes(facts) + request_digest = canonical_json_hash( + { + "operation_identity": facts.operation_identity, + "request_type": facts.request_type, + "producer_type": facts.producer_type, + "producer_ref": facts.producer_ref, + "project_id": facts.project_id, + "task_id": facts.task_id, + "guide_source_item_id": facts.guide_source_item_id, + "upload_item_id": facts.upload_item_id, + "checker_run_id": facts.checker_run_id, + "logical_role": facts.logical_role, + "sha256": commitment.sha256, + "byte_count": commitment.byte_count, + "media_type": commitment.media_type, + "namespace_fingerprint": namespace.namespace_fingerprint, + "scopes": [ { - "name": "namespace_fingerprint", - "value": namespace.namespace_fingerprint, + "scope_type": scope.scope_type, + "scope_id": scope.scope_id, + "limit_bytes": scope.limit_bytes, } + for scope in scopes ], + } + ) + counters = await self._repo.ensure_and_lock_admission_scopes( + [ + (scope.scope_type, scope.scope_id, scope.limit_bytes) + for scope in scopes + ] + ) + # A concurrent first caller may have committed while these shared + # scope locks were pending. Recheck under serialization before any + # counter or charge mutation. + replay = await self._existing_attempt( + facts.operation_identity, + request_digest, + ) + charges = await self._reserve_charges( + scopes=scopes, + counters=counters, + facts=facts, + sha256=commitment.sha256, + byte_count=commitment.byte_count, + ) + if replay is not None: + linked_charge_ids = await self._repo.list_put_attempt_charge_ids( + replay.id ) + reserved_charge_ids = tuple(sorted(charge.id for charge in charges)) + if linked_charge_ids != reserved_charge_ids: + raise ArtifactAdmissionConfigurationError( + "artifact admission replay charge set is incomplete" + ) + return await self._result(replay, replayed=True) + database_now = await self._repo.database_now() + attempt = ArtifactPutAttempt( + id=str(uuid4()), + producer_request_type=facts.request_type, + producer_type=facts.producer_type, + producer_ref=facts.producer_ref, + project_id=facts.project_id, + task_id=facts.task_id, + guide_source_item_id=facts.guide_source_item_id, + upload_item_id=facts.upload_item_id, + checker_run_id=facts.checker_run_id, + logical_role=facts.logical_role, + sha256=commitment.sha256, + byte_count=commitment.byte_count, + media_type=commitment.media_type, + storage_namespace_id=namespace.id, + namespace_fingerprint=namespace.namespace_fingerprint, + canonical_target=artifact_provider_object_ref(commitment), + operation_identity=facts.operation_identity, + request_digest=request_digest, + status="prepared", + next_run_at=None, + executor_id=None, + lease_expires_at=None, + execution_generation=0, + terminal_result_code=None, + replica_id=None, + receipt_id=None, + cas_version=0, + prepared_at=database_now, + terminal_at=None, ) - item.state = "stored_pending_verification" - item.content_id = content.id - item.provider_object_ref = result.provider_object_ref - item.cas_version += 1 - self._apply_committed_accounting(upload_session, item, commitment.byte_count) - - async def _claim_and_validate_namespace(self) -> ArtifactStorageNamespace: - """Atomically claim the singleton or reject deployment identity drift.""" - return await _claim_and_validate_storage_namespace(self._repo, self._namespace) + await self._repo.add_put_attempt(attempt, charges) + return await self._result(attempt, replayed=False) - async def _locked_attempt( - self, - item_id: str, - fence: ProviderAttemptFence, - ) -> tuple[ArtifactUploadItem, ArtifactUploadSession]: - """Reload and fence the exact item/session pair after provider I/O.""" - item = await self._repo.lock_upload_item(item_id) - if item is None or item.state != "uploading" or item.cas_version != fence.item_cas: - raise ArtifactIngestStateError("artifact upload item is not awaiting acknowledgement") - upload_session = await self._repo.lock_upload_session(item.session_id) + @staticmethod + def _validate_request_boundary(request: ArtifactAdmissionRequest) -> None: + """Reject open-ended or forged internal request shapes.""" + if type(request) not in { + GuideArtifactAdmissionRequest, + ContributorArtifactAdmissionRequest, + CheckerOutputArtifactAdmissionRequest, + }: + raise TypeError("invalid artifact admission request") + if type(request.authorization_context) is not AuthorizationContext: + raise TypeError("invalid artifact admission authorization context") + if type(request.source) is not CommittedArtifactSource: + raise TypeError("invalid artifact admission source") + if type(request) is CheckerOutputArtifactAdmissionRequest: + ArtifactAdmissionService._validate_logical_role(request.logical_role) + context = request.authorization_context if ( - upload_session is None - or upload_session.state != "open" + context.actor_status is not ActorStatus.ACTIVE + or context.identity_link_status is not IdentityLinkStatus.ACTIVE ): - raise ArtifactIngestStateError("artifact upload session is not open") - return item, upload_session + raise ArtifactAdmissionRelationshipError( + "artifact admission actor is not active" + ) - @staticmethod - def _apply_committed_accounting( - upload_session: ArtifactUploadSession, - item: ArtifactUploadItem, - byte_count: int, - ) -> None: - """Move one item from reserved capacity to acknowledged byte usage.""" - ArtifactStorageOrchestrator._validate_reserved_accounting(upload_session, item) - upload_session.reserved_bytes -= item.reserved_bytes - upload_session.reserved_items -= 1 - upload_session.current_bytes += byte_count - upload_session.current_items += 1 - upload_session.cas_version += 1 - - async def _mark_replay_required( - self, - item_id: str, - fence: ProviderAttemptFence, - ) -> None: - """Record an ambiguous acknowledgement only when the attempt fence matches.""" - await self._session.rollback() - async with self._session.begin(): - item = await self._repo.lock_upload_item(item_id) - if item is None or item.state != "uploading" or item.cas_version != fence.item_cas: - return - item.state = "replay_required" - item.error_code = "artifact_put_acknowledgement_unknown" - item.cas_version += 1 - - async def _fail_put( - self, - item_id: str, - error_code: str, - fence: ProviderAttemptFence, + async def _derive_admission_facts( + self, request: ArtifactAdmissionRequest + ) -> _AdmissionFacts: + """Load every product and producer relationship from authoritative rows.""" + if type(request) is GuideArtifactAdmissionRequest: + return await self._guide_facts(request) + if type(request) is ContributorArtifactAdmissionRequest: + return await self._contributor_facts(request) + if type(request) is CheckerOutputArtifactAdmissionRequest: + return await self._checker_output_facts(request) + raise TypeError("invalid artifact admission request") + + async def _guide_facts( + self, request: GuideArtifactAdmissionRequest + ) -> _AdmissionFacts: + """Bind committed bytes to one authoritative guide source item.""" + context = request.authorization_context + if context.actor_kind is not ActorKind.HUMAN: + raise ArtifactAdmissionRelationshipError( + "guide artifact producer must be a human actor" + ) + await self._require_active_human_actor(context) + item_id = str(request.guide_source_item_id) + row = await self._repo.get_guide_admission_facts(item_id) + commitment = request.source.commitment + if ( + row is None + or row.captured_by != str(context.actor_profile_id) + or row.content_hash != commitment.sha256 + or row.media_type != commitment.media_type + ): + raise ArtifactAdmissionRelationshipError( + "guide source item relationship is unavailable" + ) + operation_identity = canonical_json_hash( + {"request_type": "guide", "guide_source_item_id": item_id} + ) + return _AdmissionFacts( + request_type="guide", + producer_type="actor_profile", + producer_ref=row.captured_by, + project_id=row.project_id, + task_id=None, + guide_source_item_id=item_id, + upload_item_id=None, + checker_run_id=None, + logical_role=None, + operation_identity=operation_identity, + ) + + async def _contributor_facts( + self, request: ContributorArtifactAdmissionRequest + ) -> _AdmissionFacts: + """Bind committed bytes to one contributor-owned upload item.""" + context = request.authorization_context + if context.actor_kind is not ActorKind.HUMAN: + raise ArtifactAdmissionRelationshipError( + "contributor artifact producer must be a human actor" + ) + await self._require_active_human_actor(context) + item_id = str(request.upload_item_id) + row = await self._repo.get_contributor_admission_facts(item_id) + commitment = request.source.commitment + if ( + row is None + or row.actor_profile_id != str(context.actor_profile_id) + or row.task_id is None + or row.session_state != "open" + or row.item_state not in {"reserved", "replay_required"} + or row.expected_sha256 != commitment.sha256 + or row.expected_size != commitment.byte_count + or row.media_type != commitment.media_type + ): + raise ArtifactAdmissionRelationshipError( + "contributor upload item relationship is unavailable" + ) + operation_identity = canonical_json_hash( + {"request_type": "contributor", "upload_item_id": item_id} + ) + return _AdmissionFacts( + request_type="contributor", + producer_type="actor_profile", + producer_ref=str(context.actor_profile_id), + project_id=row.project_id, + task_id=row.task_id, + guide_source_item_id=None, + upload_item_id=item_id, + checker_run_id=None, + logical_role=None, + operation_identity=operation_identity, + ) + + async def _checker_output_facts( + self, request: CheckerOutputArtifactAdmissionRequest + ) -> _AdmissionFacts: + """Bind committed bytes to one run and fixed checker service actor.""" + context = request.authorization_context + if context.actor_kind is not ActorKind.SERVICE: + raise ArtifactAdmissionRelationshipError( + "checker output producer must be a service actor" + ) + logical_role = request.logical_role + service_actor = await self._actors.lock_admission_proof( + context.actor_profile_id, + context.identity_link_id, + ) + if ( + service_actor is None + or service_actor.actor_kind != "service" + or service_actor.actor_status != "active" + or service_actor.identity_link_id != str(context.identity_link_id) + or service_actor.identity_link_subject_kind != "service" + or service_actor.identity_link_status != "active" + or service_actor.service_identity + != ServiceIdentity.ARTIFACT_CHECKER_OUTPUT.value + ): + raise ArtifactAdmissionRelationshipError( + "checker output service identity is unavailable" + ) + checker_run_id = str(request.checker_run_id) + row = await self._repo.get_checker_output_admission_facts(checker_run_id) + if row is None: + raise ArtifactAdmissionRelationshipError( + "checker run relationship is unavailable" + ) + operation_identity = canonical_json_hash( + { + "request_type": "checker_output", + "checker_run_id": checker_run_id, + "logical_role": logical_role, + } + ) + return _AdmissionFacts( + request_type="checker_output", + producer_type="service_identity", + producer_ref=ServiceIdentity.ARTIFACT_CHECKER_OUTPUT.value, + project_id=row.project_id, + task_id=row.task_id, + guide_source_item_id=None, + upload_item_id=None, + checker_run_id=checker_run_id, + logical_role=logical_role, + operation_identity=operation_identity, + ) + + async def _require_active_human_actor( + self, context: AuthorizationContext ) -> None: - """Fail one terminal provider attempt and release its reservation once.""" - await self._session.rollback() - async with self._session.begin(): - item = await self._repo.lock_upload_item(item_id) - if item is None or item.state != "uploading" or item.cas_version != fence.item_cas: - return - upload_session = await self._repo.lock_upload_session(item.session_id) - if upload_session is None: - raise ArtifactIntegrityError("artifact upload session is missing") - self._validate_reserved_accounting(upload_session, item) - item.state = "failed" - item.error_code = error_code - item.cas_version += 1 - upload_session.reserved_bytes -= item.reserved_bytes - upload_session.reserved_items -= 1 - upload_session.cas_version += 1 + """Revalidate and lock exact human identity state inside admission.""" + actor = await self._actors.lock_admission_proof( + context.actor_profile_id, + context.identity_link_id, + ) + if ( + actor is None + or actor.actor_kind != "human" + or actor.actor_status != "active" + or actor.service_identity is not None + or actor.identity_link_id != str(context.identity_link_id) + or actor.identity_link_subject_kind != "human" + or actor.identity_link_status != "active" + ): + raise ArtifactAdmissionRelationshipError( + "artifact admission human identity is unavailable" + ) @staticmethod - def _validate_reserved_accounting( - upload_session: ArtifactUploadSession, - item: ArtifactUploadItem, - ) -> None: - """Reject aggregate drift before consuming one item's reservation.""" + def _validate_logical_role(value: str) -> str: + """Require one bounded printable checker-output role.""" if ( - upload_session.reserved_bytes < item.reserved_bytes - or upload_session.reserved_items < 1 + not isinstance(value, str) + or value != value.strip() + or not value + or not value.isascii() + or len(value) > 100 + or any(ord(character) < 32 or ord(character) == 127 for character in value) + ): + raise ArtifactAdmissionRelationshipError( + "checker output logical role is invalid" + ) + return value + + def _derive_scopes(self, facts: _AdmissionFacts) -> tuple[_AdmissionScopeSpec, ...]: + """Derive the complete closed scope set without caller participation.""" + limits = self._configured_limits() + scopes = [ + _AdmissionScopeSpec( + "deployment", + ARTIFACT_STORAGE_NAMESPACE_ID, + limits["deployment"], + ), + _AdmissionScopeSpec("project", facts.project_id, limits["project"]), + _AdmissionScopeSpec( + "producer", + f"{facts.producer_type}:{facts.producer_ref}", + limits["producer"], + ), + ] + if facts.task_id is not None: + scopes.append(_AdmissionScopeSpec("task", facts.task_id, limits["task"])) + return tuple(sorted(scopes, key=lambda value: (value.scope_type, value.scope_id))) + + def _configured_limits(self) -> dict[str, int]: + """Return exact positive limits only for an enabled artifact backend.""" + values = { + "task": self._settings.artifact_admission_task_maximum_bytes, + "producer": self._settings.artifact_admission_producer_maximum_bytes, + "project": self._settings.artifact_admission_project_maximum_bytes, + "deployment": self._settings.artifact_admission_deployment_maximum_bytes, + } + if self._settings.artifact_store_backend == "disabled" or any( + type(value) is not int or value <= 0 for value in values.values() ): - raise ArtifactIntegrityError( - "artifact upload session reservation accounting is invalid" + raise ArtifactAdmissionConfigurationError( + "artifact durable-byte admission is not configured" + ) + return {key: int(value) for key, value in values.items()} + + async def _existing_attempt( + self, + operation_identity: str, + request_digest: str, + ) -> ArtifactPutAttempt | None: + """Load an exact replay or reject changed input for one operation.""" + existing = await self._repo.get_put_attempt_by_operation(operation_identity) + if existing is None: + return None + if existing.request_digest != request_digest: + raise ArtifactAdmissionConflictError( + "artifact admission operation input changed" ) + return existing + + async def _reserve_charges( + self, + *, + scopes: tuple[_AdmissionScopeSpec, ...], + counters: tuple[ArtifactAdmissionScope, ...], + facts: _AdmissionFacts, + sha256: str, + byte_count: int, + ) -> tuple[ArtifactAdmissionCharge, ...]: + """Reserve unique content under every locked scope or fail atomically.""" + counter_by_key = { + (counter.scope_type, counter.scope_id): counter for counter in counters + } + if len(counter_by_key) != len(scopes): + raise ArtifactAdmissionConfigurationError( + "artifact admission scope set is incomplete" + ) + database_now = await self._repo.database_now() + charges: list[ArtifactAdmissionCharge] = [] + for scope in scopes: + counter = counter_by_key[(scope.scope_type, scope.scope_id)] + if counter.limit_bytes != scope.limit_bytes: + raise ArtifactAdmissionConfigurationError( + "artifact admission scope limit does not match configuration" + ) + charge = await self._repo.get_admission_charge( + scope_type=scope.scope_type, + scope_id=scope.scope_id, + sha256=sha256, + byte_count=byte_count, + ) + if charge is not None and charge.state in {"provisional", "completed"}: + charges.append(charge) + continue + if counter.counted_bytes + byte_count > counter.limit_bytes: + raise ArtifactAdmissionCapacityError( + f"artifact durable-byte limit exceeded for {scope.scope_type} scope" + ) + counter.counted_bytes += byte_count + counter.cas_version += 1 + if charge is None: + charge = await self._repo.add_admission_charge( + ArtifactAdmissionCharge( + id=str(uuid4()), + scope_type=scope.scope_type, + scope_id=scope.scope_id, + sha256=sha256, + byte_count=byte_count, + producer_type=facts.producer_type, + producer_ref=facts.producer_ref, + creating_operation_identity=facts.operation_identity, + state="provisional", + cas_version=0, + reserved_at=database_now, + completed_at=None, + released_at=None, + ) + ) + elif charge.state == "released": + charge.state = "provisional" + charge.reserved_at = database_now + charge.released_at = None + charge.cas_version += 1 + else: + raise ArtifactAdmissionConflictError( + "artifact admission charge state is invalid" + ) + charges.append(charge) + return tuple(charges) + + async def _result( + self, attempt: ArtifactPutAttempt, *, replayed: bool + ) -> ArtifactAdmissionResult: + """Return one detached-safe internal result.""" + charge_ids = await self._repo.list_put_attempt_charge_ids(attempt.id) + return ArtifactAdmissionResult( + attempt_id=UUID(attempt.id), + status=attempt.status, + operation_identity=attempt.operation_identity, + request_digest=attempt.request_digest, + charge_ids=tuple(UUID(charge_id) for charge_id in charge_ids), + replayed=replayed, + ) async def validate_artifact_storage_namespace_at_startup( diff --git a/backend/tests/artifact_store_helpers.py b/backend/tests/artifact_store_helpers.py index bdc149a35..10a2f02c2 100644 --- a/backend/tests/artifact_store_helpers.py +++ b/backend/tests/artifact_store_helpers.py @@ -20,6 +20,18 @@ from app.modules.artifacts.sources import CommittedArtifactSource +def artifact_admission_limit_settings( + maximum_bytes: int = 1024, +) -> dict[str, int]: + """Return explicit bounded admission limits for enabled test stores.""" + return { + "artifact_admission_task_maximum_bytes": maximum_bytes, + "artifact_admission_producer_maximum_bytes": maximum_bytes, + "artifact_admission_project_maximum_bytes": maximum_bytes, + "artifact_admission_deployment_maximum_bytes": maximum_bytes, + } + + async def artifact_byte_stream(*chunks: bytes) -> AsyncIterator[bytes]: """Yield exact test bytes through the public preparation boundary.""" for chunk in chunks: diff --git a/backend/tests/test_artifact_admission.py b/backend/tests/test_artifact_admission.py new file mode 100644 index 000000000..975a878d7 --- /dev/null +++ b/backend/tests/test_artifact_admission.py @@ -0,0 +1,1924 @@ +"""PostgreSQL proofs for atomic durable-byte admission before provider I/O.""" + +from __future__ import annotations + +import asyncio +from dataclasses import replace +from datetime import UTC, datetime, timedelta +from pathlib import Path +from uuid import UUID, uuid4 + +from alembic import command +from alembic.config import Config +import pytest +from sqlalchemy import func, select, text +from sqlalchemy.exc import DBAPIError +from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine + +from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap +from app.core.config import Settings +from app.core.hashing import canonical_json_hash +from app.modules.actors.models import ActorIdentityLink, ActorProfile +from app.modules.actors.service import ActorService +from app.modules.actors.service_identities import ServiceIdentity +from app.modules.checkers.models import CheckerRun +from app.modules.artifacts.models import ( + ArtifactAdmissionCharge, + ArtifactAdmissionScope, + ArtifactContent, + ArtifactOperationReceipt, + ArtifactPutAttempt, + ArtifactPutAttemptCharge, + ArtifactReplica, + ArtifactStorageNamespace, + ArtifactUploadItem, + ArtifactUploadSession, +) +from app.modules.artifacts.repository import ArtifactRepository +from app.modules.artifacts.schemas import ( + CheckerOutputArtifactAdmissionRequest, + ContributorArtifactAdmissionRequest, + GuideArtifactAdmissionRequest, +) +from app.modules.artifacts.service import ( + ArtifactAdmissionCapacityError, + ArtifactAdmissionConflictError, + ArtifactAdmissionRelationshipError, + ArtifactAdmissionService, + ArtifactStorageNamespaceSpec, + artifact_storage_namespace_spec, +) +from app.modules.authorization.runtime import ( + ActorKind, + ActorStatus, + AuthorizationContext, + IdentityLinkStatus, +) +from app.modules.projects.models import ( + EffectiveProjectSubmissionArtifactPolicy, + GuideSourceSnapshot, + GuideSourceSnapshotItem, + PaymentPolicy, + PostSubmitCheckerPolicy, + PreSubmitCheckerPolicy, + Project, + ProjectGuide, + ReviewPolicy, + RevisionPolicy, + SubmissionArtifactPolicy, +) +from app.modules.tasks.models import AuditEvent, Submission, WorkstreamTask +from tests.artifact_store_helpers import ( + artifact_admission_limit_settings, + minted_source, +) + + +def _alembic_config() -> Config: + root = Path(__file__).resolve().parents[1] + config = Config(str(root / "alembic.ini")) + config.set_main_option("script_location", str(root / "alembic")) + return config + + +@pytest.fixture +def admission_database_env( + isolated_database_env: str, + migration_lock, +) -> str: + """Provide the exact head schema and remove all test evidence afterward.""" + config = _alembic_config() + with migration_lock(): + asyncio.run(_reset_admission_test_schema(isolated_database_env)) + command.upgrade(config, "head") + try: + yield isolated_database_env + finally: + asyncio.run(_reset_admission_test_schema(isolated_database_env)) + + +async def _reset_admission_test_schema(database_url: str) -> None: + engine = create_async_engine(database_url) + try: + async with engine.begin() as connection: + await connection.execute(text("drop schema if exists public cascade")) + await connection.execute(text("create schema public")) + finally: + await engine.dispose() + + +def _settings(tmp_path: Path, *, maximum_bytes: int = 1024) -> Settings: + durable_root = tmp_path / "durable" + durable_root.mkdir(mode=0o700, parents=True) + return Settings( + **artifact_admission_limit_settings(maximum_bytes), + environment="test", + artifact_store_backend="local", + artifact_local_root=durable_root, + artifact_scratch_root=tmp_path / "scratch", + artifact_scratch_minimum_free_bytes=0, + ) + + +def _namespace(settings: Settings) -> ArtifactStorageNamespaceSpec: + assert settings.artifact_local_root is not None + bootstrap = LocalStorageBootstrap( + LocalStorageAdapter(root=settings.artifact_local_root) + ) + try: + return artifact_storage_namespace_spec(settings, bootstrap) + finally: + bootstrap.close() + + +def _context( + *, + actor_profile_id: UUID | None = None, + identity_link_id: UUID | None = None, + actor_kind: ActorKind = ActorKind.HUMAN, +) -> AuthorizationContext: + return AuthorizationContext( + actor_profile_id=actor_profile_id or uuid4(), + actor_kind=actor_kind, + actor_status=ActorStatus.ACTIVE, + identity_link_id=identity_link_id or uuid4(), + identity_link_status=IdentityLinkStatus.ACTIVE, + request_id=uuid4(), + correlation_id=uuid4(), + ) + + +async def _seed_human_actor( + session, + context: AuthorizationContext, +) -> None: + """Persist the exact active human actor carried by a test context.""" + actor_profile_id = str(context.actor_profile_id) + if await session.get(ActorProfile, actor_profile_id) is not None: + return + session.add( + ActorProfile( + id=actor_profile_id, + actor_kind="human", + status="active", + provisioning_method="automatic_first_access", + service_identity=None, + created_by="test", + ) + ) + await session.flush() + session.add( + ActorIdentityLink( + id=str(context.identity_link_id), + actor_profile_id=actor_profile_id, + issuer="https://issuer.example.test", + subject=f"human-{actor_profile_id}", + subject_kind="human", + status="active", + linked_by="test", + last_verified_at=datetime.now(UTC), + ) + ) + await session.flush() + + +async def _seed_guide( + session, + *, + context: AuthorizationContext, + content_hash: str, + media_type: str, +) -> tuple[str, str]: + await _seed_human_actor(session, context) + captured_by = str(context.actor_profile_id) + project_id = str(uuid4()) + guide_id = str(uuid4()) + snapshot_id = str(uuid4()) + item_id = str(uuid4()) + session.add( + Project( + id=project_id, + name="Admission project", + slug=f"admission-{project_id}", + ) + ) + await session.flush() + session.add( + ProjectGuide( + id=guide_id, + project_id=project_id, + version="v1", + status="draft", + content_markdown="# Guide", + created_by="test", + ) + ) + await session.flush() + session.add( + GuideSourceSnapshot( + id=snapshot_id, + project_id=project_id, + guide_id=guide_id, + guide_version="v1", + manifest_schema_version="v1", + manifest_json={"items": [item_id]}, + bundle_hash=canonical_json_hash({"items": [item_id]}), + captured_by=captured_by, + ) + ) + await session.flush() + session.add( + GuideSourceSnapshotItem( + id=item_id, + source_snapshot_id=snapshot_id, + item_order=0, + source_kind="inline", + durable_ref="guide.md", + ingestion_adapter="inline", + content_hash=content_hash, + media_type=media_type, + ) + ) + await session.commit() + return project_id, item_id + + +async def _seed_contributor_items( + session, + *, + context: AuthorizationContext, + commitments: tuple[tuple[str, int, str], ...], +) -> tuple[str, str, tuple[str, ...]]: + await _seed_human_actor(session, context) + actor_profile_id = str(context.actor_profile_id) + project_id = str(uuid4()) + task_id = str(uuid4()) + upload_session_id = str(uuid4()) + session.add( + Project( + id=project_id, + name="Contributor project", + slug=f"contributor-{project_id}", + ) + ) + await session.flush() + session.add( + WorkstreamTask( + id=task_id, + project_id=project_id, + title="Admission task", + description="Prove artifact admission.", + status="draft", + created_by="test", + ) + ) + await session.flush() + total_bytes = sum(byte_count for _, byte_count, _ in commitments) + session.add( + ArtifactUploadSession( + id=upload_session_id, + actor_id=actor_profile_id, + project_id=project_id, + task_id=task_id, + permitted_roles=["submission"], + state="open", + maximum_bytes=max(total_bytes, 1), + current_bytes=0, + reserved_bytes=total_bytes, + maximum_items=len(commitments), + current_items=0, + reserved_items=len(commitments), + expires_at=datetime.now(UTC) + timedelta(minutes=10), + cas_version=0, + ) + ) + await session.flush() + item_ids = [] + for index, (sha256, byte_count, media_type) in enumerate(commitments): + item_id = str(uuid4()) + item_ids.append(item_id) + session.add( + ArtifactUploadItem( + id=item_id, + session_id=upload_session_id, + logical_role=f"submission-{index}", + display_name=f"result-{index}.bin", + media_type=media_type, + reserved_bytes=byte_count, + expected_sha256=sha256, + expected_size=byte_count, + idempotency_key=f"put-{item_id}", + request_digest=canonical_json_hash( + { + "sha256": sha256, + "byte_count": byte_count, + "media_type": media_type, + } + ), + state="reserved", + cas_version=0, + ) + ) + await session.commit() + return project_id, task_id, tuple(item_ids) + + +async def _seed_checker_output_relationships(session) -> tuple[str, str, str]: + """Persist one complete checker-run ownership chain for admission proof.""" + project_id = str(uuid4()) + guide_id = str(uuid4()) + snapshot_id = str(uuid4()) + submission_policy_id = str(uuid4()) + effective_policy_id = str(uuid4()) + pre_submit_policy_id = str(uuid4()) + post_submit_policy_id = str(uuid4()) + task_id = str(uuid4()) + submission_id = str(uuid4()) + contributor_id = str(uuid4()) + contributor_link_id = str(uuid4()) + checker_run_id = str(uuid4()) + guide_version = "v1" + snapshot_hash = canonical_json_hash({"items": []}) + submission_policy_body = {"required_artifacts": []} + submission_policy_hash = canonical_json_hash(submission_policy_body) + effective_policy_body = {"required_artifacts": [], "artifact_hash_algorithm": "sha256"} + effective_policy_hash = canonical_json_hash(effective_policy_body) + pre_submit_bundle = {"schema_version": "v1", "rules": []} + pre_submit_bundle_hash = canonical_json_hash(pre_submit_bundle) + post_submit_policy_body = {"required_checkers": []} + post_submit_policy_hash = canonical_json_hash(post_submit_policy_body) + now = datetime.now(UTC) + + session.add(Project(id=project_id, name="Checker project", slug=f"checker-{project_id}")) + await session.flush() + session.add( + ProjectGuide( + id=guide_id, + project_id=project_id, + version=guide_version, + status="active", + content_markdown="# Checker guide", + approved_by="setup-actor", + effective_at=now, + created_by="setup-actor", + ) + ) + await session.flush() + session.add( + GuideSourceSnapshot( + id=snapshot_id, + project_id=project_id, + guide_id=guide_id, + guide_version=guide_version, + manifest_schema_version="v1", + manifest_json={"items": []}, + bundle_hash=snapshot_hash, + captured_by="setup-actor", + ) + ) + await session.flush() + session.add( + SubmissionArtifactPolicy( + id=submission_policy_id, + project_id=project_id, + guide_id=guide_id, + guide_version=guide_version, + source_snapshot_id=snapshot_id, + source_snapshot_hash=snapshot_hash, + policy_version="v1", + lifecycle_status="approved", + policy_body=submission_policy_body, + policy_hash=submission_policy_hash, + derivation_source="test", + source_material_refs=[], + created_by="setup-actor", + approved_by_role="admin", + approved_by_actor="setup-actor", + approved_at=now, + ) + ) + await session.flush() + session.add( + EffectiveProjectSubmissionArtifactPolicy( + id=effective_policy_id, + project_id=project_id, + guide_id=guide_id, + guide_version=guide_version, + source_snapshot_id=snapshot_id, + source_snapshot_hash=snapshot_hash, + submission_artifact_policy_id=submission_policy_id, + submission_artifact_policy_hash=submission_policy_hash, + lifecycle_status="approved", + merge_algorithm_version="v1", + effective_policy=effective_policy_body, + effective_policy_hash=effective_policy_hash, + created_by="setup-actor", + ) + ) + await session.flush() + session.add( + PreSubmitCheckerPolicy( + id=pre_submit_policy_id, + project_id=project_id, + guide_id=guide_id, + guide_version=guide_version, + source_snapshot_id=snapshot_id, + source_snapshot_hash=snapshot_hash, + effective_policy_id=effective_policy_id, + effective_policy_hash=effective_policy_hash, + lifecycle_status="compiled", + compiler_version="v1", + compiled_bundle=pre_submit_bundle, + compiled_bundle_hash=pre_submit_bundle_hash, + checker_names=[], + checker_configs={}, + created_by="setup-actor", + ) + ) + await session.flush() + session.add_all( + [ + PostSubmitCheckerPolicy( + id=post_submit_policy_id, + project_id=project_id, + guide_id=guide_id, + guide_version=guide_version, + source_snapshot_id=snapshot_id, + source_snapshot_hash=snapshot_hash, + effective_policy_id=effective_policy_id, + effective_policy_hash=effective_policy_hash, + pre_submit_checker_policy_id=pre_submit_policy_id, + pre_submit_checker_bundle_hash=pre_submit_bundle_hash, + required_checkers=[], + warning_checkers=[], + blocking_severities=["error"], + policy_hash=post_submit_policy_hash, + policy_body=post_submit_policy_body, + lifecycle_status="approved", + approved_by_role="admin", + approved_by_actor="setup-actor", + approved_at=now, + created_by="setup-actor", + ), + ReviewPolicy( + id=str(uuid4()), + project_id=project_id, + guide_version=guide_version, + requires_second_review=False, + allowed_decisions=["accept", "needs_revision", "reject"], + minimum_finding_fields=[], + ), + RevisionPolicy( + id=str(uuid4()), + project_id=project_id, + guide_version=guide_version, + max_revision_rounds=1, + revision_deadline_hours=24, + auto_reject_after_limit=True, + allowed_resubmission_states=["needs_revision"], + ), + PaymentPolicy( + id=str(uuid4()), + project_id=project_id, + guide_version=guide_version, + ), + ] + ) + await session.flush() + session.add( + WorkstreamTask( + id=task_id, + project_id=project_id, + locked_guide_version=guide_version, + locked_post_submit_checker_policy_id=post_submit_policy_id, + locked_post_submit_checker_policy_version=guide_version, + locked_post_submit_checker_policy_hash=post_submit_policy_hash, + locked_post_submit_checker_policy_body=post_submit_policy_body, + locked_review_policy_version=guide_version, + locked_revision_policy_version=guide_version, + locked_payment_policy_version=guide_version, + locked_guide_source_snapshot_id=snapshot_id, + locked_guide_source_snapshot_hash=snapshot_hash, + locked_effective_project_submission_artifact_policy_id=effective_policy_id, + locked_effective_project_submission_artifact_policy_hash=effective_policy_hash, + locked_pre_submit_checker_policy_id=pre_submit_policy_id, + locked_pre_submit_checker_bundle_hash=pre_submit_bundle_hash, + title="Checker admission task", + description="Prove checker output admission.", + status="draft", + created_by="setup-actor", + ) + ) + await session.flush() + session.add( + ActorProfile( + id=contributor_id, + actor_kind="human", + status="active", + provisioning_method="automatic_first_access", + service_identity=None, + created_by="test", + ) + ) + await session.flush() + session.add( + ActorIdentityLink( + id=contributor_link_id, + actor_profile_id=contributor_id, + issuer="https://issuer.example.test", + subject=f"human-{contributor_id}", + subject_kind="human", + status="active", + linked_by="test", + last_verified_at=now, + ) + ) + await session.flush() + session.add( + Submission( + id=submission_id, + task_id=task_id, + contributor_id=contributor_id, + version=1, + status="submitted", + summary="Checker source submission", + package_hash=canonical_json_hash({"submission": submission_id}), + artifact_hash_manifest=[], + worker_attestation="complete", + locked_guide_version=guide_version, + locked_post_submit_checker_policy_id=post_submit_policy_id, + locked_post_submit_checker_policy_version=guide_version, + locked_post_submit_checker_policy_hash=post_submit_policy_hash, + locked_post_submit_checker_policy_body=post_submit_policy_body, + locked_review_policy_version=guide_version, + locked_revision_policy_version=guide_version, + locked_payment_policy_version=guide_version, + locked_guide_source_snapshot_id=snapshot_id, + locked_guide_source_snapshot_hash=snapshot_hash, + locked_effective_project_submission_artifact_policy_id=effective_policy_id, + locked_effective_project_submission_artifact_policy_hash=effective_policy_hash, + locked_pre_submit_checker_policy_id=pre_submit_policy_id, + locked_pre_submit_checker_bundle_hash=pre_submit_bundle_hash, + ) + ) + await session.flush() + session.add( + CheckerRun( + id=checker_run_id, + task_id=task_id, + submission_id=submission_id, + submission_version=1, + trigger_source="submission_finalized", + status="queued", + routing_recommendation="not_evaluated", + outcome_source="none", + triggered_by="setup-actor", + triggered_by_subject="setup-subject", + triggered_by_issuer="https://issuer.example.test", + trigger_auth_source="test", + attempt_number=1, + is_current_for_submission=True, + locked_guide_version=guide_version, + locked_post_submit_checker_policy_id=post_submit_policy_id, + locked_post_submit_checker_policy_version=guide_version, + locked_post_submit_checker_policy_hash=post_submit_policy_hash, + locked_post_submit_checker_policy_body=post_submit_policy_body, + locked_review_policy_version=guide_version, + locked_revision_policy_version=guide_version, + locked_payment_policy_version=guide_version, + package_hash=canonical_json_hash({"submission": submission_id}), + artifact_hash_manifest=[], + artifact_manifest_hash=canonical_json_hash([]), + ) + ) + await session.commit() + return project_id, task_id, checker_run_id + + +async def _count(session, model: type) -> int: + value = await session.scalar(select(func.count()).select_from(model)) + assert value is not None + return value + + +async def test_guide_admission_derives_three_scopes_without_provider_evidence( + admission_database_env: str, + tmp_path: Path, +) -> None: + settings = _settings(tmp_path) + namespace = _namespace(settings) + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + context = _context() + async with minted_source( + tmp_path / "scratch-source", + b"guide", + media_type="text/markdown", + ) as source: + project_id, item_id = await _seed_guide( + session, + context=context, + content_hash=source.commitment.sha256, + media_type=source.commitment.media_type, + ) + with pytest.raises( + ArtifactAdmissionRelationshipError, + match="artifact admission human identity is unavailable", + ): + await ArtifactAdmissionService( + session, + settings, + namespace, + ).admit( + GuideArtifactAdmissionRequest( + authorization_context=_context(), + guide_source_item_id=UUID(item_id), + source=source, + ) + ) + assert await _count(session, ArtifactStorageNamespace) == 0 + assert await _count(session, ArtifactAdmissionScope) == 0 + assert await _count(session, ArtifactAdmissionCharge) == 0 + assert await _count(session, ArtifactPutAttempt) == 0 + await session.rollback() + result = await ArtifactAdmissionService( + session, + settings, + namespace, + ).admit( + GuideArtifactAdmissionRequest( + authorization_context=context, + guide_source_item_id=UUID(item_id), + source=source, + ) + ) + + async with minted_source( + tmp_path / "wrong-source", + b"different guide bytes", + media_type="text/markdown", + ) as wrong_source: + with pytest.raises( + ArtifactAdmissionRelationshipError, + match="guide source item relationship is unavailable", + ): + await ArtifactAdmissionService( + session, + settings, + namespace, + ).admit( + GuideArtifactAdmissionRequest( + authorization_context=context, + guide_source_item_id=UUID(item_id), + source=wrong_source, + ) + ) + + attempt = await session.get(ArtifactPutAttempt, str(result.attempt_id)) + scopes = ( + await session.execute( + select(ArtifactAdmissionScope).order_by( + ArtifactAdmissionScope.scope_type + ) + ) + ).scalars().all() + assert attempt is not None + assert attempt.status == "prepared" + assert attempt.project_id == project_id + assert attempt.task_id is None + assert attempt.executor_id is None + assert attempt.lease_expires_at is None + assert attempt.next_run_at is None + assert attempt.execution_generation == 0 + assert {scope.scope_type for scope in scopes} == { + "deployment", + "producer", + "project", + } + assert len(result.charge_ids) == 3 + assert await _count(session, ArtifactPutAttempt) == 1 + assert await _count(session, ArtifactContent) == 0 + assert await _count(session, ArtifactReplica) == 0 + assert await _count(session, ArtifactOperationReceipt) == 0 + finally: + await engine.dispose() + + +async def test_human_admission_revalidates_exact_active_profile_and_link( + admission_database_env: str, + tmp_path: Path, +) -> None: + settings = _settings(tmp_path) + namespace = _namespace(settings) + context = _context() + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + async with minted_source( + tmp_path / "guide-source", + b"guide", + media_type="text/markdown", + ) as guide_source: + _, guide_item_id = await _seed_guide( + session, + context=context, + content_hash=guide_source.commitment.sha256, + media_type=guide_source.commitment.media_type, + ) + async with minted_source( + tmp_path / "contributor-source", + b"work", + ) as contributor_source: + _, _, upload_item_ids = await _seed_contributor_items( + session, + context=context, + commitments=( + ( + contributor_source.commitment.sha256, + contributor_source.commitment.byte_count, + contributor_source.commitment.media_type, + ), + ), + ) + requests = ( + GuideArtifactAdmissionRequest( + authorization_context=context, + guide_source_item_id=UUID(guide_item_id), + source=guide_source, + ), + ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(upload_item_ids[0]), + source=contributor_source, + ), + ) + forged_context = context.model_copy( + update={"identity_link_id": uuid4()} + ) + for request in requests: + with pytest.raises( + ArtifactAdmissionRelationshipError, + match="artifact admission human identity is unavailable", + ): + await ArtifactAdmissionService( + session, settings, namespace + ).admit( + replace( + request, + authorization_context=forged_context, + ) + ) + + link = await session.get( + ActorIdentityLink, + str(context.identity_link_id), + ) + assert link is not None + link.status = "revoked" + link.revoked_by = "test" + link.revoked_at = datetime.now(UTC) + link.revoked_reason = "test revocation" + await session.commit() + for request in requests: + with pytest.raises( + ArtifactAdmissionRelationshipError, + match="artifact admission human identity is unavailable", + ): + await ArtifactAdmissionService( + session, settings, namespace + ).admit(request) + + link.status = "active" + link.revoked_by = None + link.revoked_at = None + link.revoked_reason = None + link.reactivated_by = "test" + link.reactivated_at = datetime.now(UTC) + link.reactivation_reason = "test reactivation" + profile = await session.get( + ActorProfile, + str(context.actor_profile_id), + ) + assert profile is not None + profile.status = "suspended" + profile.suspended_by = "test" + profile.suspended_at = datetime.now(UTC) + profile.suspension_reason = "test suspension" + await session.commit() + for request in requests: + with pytest.raises( + ArtifactAdmissionRelationshipError, + match="artifact admission human identity is unavailable", + ): + await ArtifactAdmissionService( + session, settings, namespace + ).admit(request) + + assert await _count(session, ArtifactStorageNamespace) == 0 + assert await _count(session, ArtifactAdmissionScope) == 0 + assert await _count(session, ArtifactAdmissionCharge) == 0 + assert await _count(session, ArtifactPutAttempt) == 0 + finally: + await engine.dispose() + + +async def test_guide_admission_facts_lock_snapshot_and_item( + admission_database_env: str, + tmp_path: Path, +) -> None: + context = _context() + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as seed_session: + _, item_id = await _seed_guide( + seed_session, + context=context, + content_hash="sha256:" + "a" * 64, + media_type="text/markdown", + ) + + async with factory() as lock_session: + async with lock_session.begin(): + facts = await ArtifactRepository( + lock_session + ).get_guide_admission_facts(item_id) + assert facts is not None + + mutations = ( + ( + "update guide_source_snapshot_items " + "set media_type = 'application/json' where id = :item_id", + {"item_id": item_id}, + ), + ( + "update guide_source_snapshots set captured_by = :captured_by " + "where id = (select source_snapshot_id " + "from guide_source_snapshot_items where id = :item_id)", + { + "captured_by": str(uuid4()), + "item_id": item_id, + }, + ), + ) + for statement, parameters in mutations: + async with factory() as mutation_session: + with pytest.raises(DBAPIError, match="lock timeout"): + async with mutation_session.begin(): + await mutation_session.execute( + text("set local lock_timeout = '200ms'") + ) + await mutation_session.execute( + text(statement), + parameters, + ) + + async with factory() as assertion_session: + assert await _count(assertion_session, ArtifactAdmissionScope) == 0 + assert await _count(assertion_session, ArtifactAdmissionCharge) == 0 + assert await _count(assertion_session, ArtifactPutAttempt) == 0 + finally: + await engine.dispose() + + +async def test_actor_admission_proof_locks_exact_profile_then_link( + admission_database_env: str, +) -> None: + context = _context() + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as seed_session: + await _seed_human_actor(seed_session, context) + await seed_session.commit() + + async with factory() as lock_session: + async with lock_session.begin(): + proof = await ActorService(lock_session).lock_admission_proof( + context.actor_profile_id, + context.identity_link_id, + ) + assert proof is not None + assert proof.actor_profile_id == str(context.actor_profile_id) + assert proof.identity_link_id == str(context.identity_link_id) + + mutations = ( + ( + "update actor_profiles set status = 'suspended' where id = :id", + str(context.actor_profile_id), + ), + ( + "update actor_identity_links set status = 'revoked' where id = :id", + str(context.identity_link_id), + ), + ) + for statement, row_id in mutations: + async with factory() as mutation_session: + with pytest.raises(DBAPIError, match="lock timeout"): + async with mutation_session.begin(): + await mutation_session.execute( + text("set local lock_timeout = '200ms'") + ) + await mutation_session.execute( + text(statement), + {"id": row_id}, + ) + finally: + await engine.dispose() + + +async def test_exact_replay_returns_one_attempt_and_one_charge_set( + admission_database_env: str, + tmp_path: Path, +) -> None: + settings = _settings(tmp_path) + namespace = _namespace(settings) + context = _context() + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + async with minted_source(tmp_path / "scratch-source", b"same") as source: + _, _, item_ids = await _seed_contributor_items( + session, + context=context, + commitments=( + ( + source.commitment.sha256, + source.commitment.byte_count, + source.commitment.media_type, + ), + ), + ) + request = ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_ids[0]), + source=source, + ) + first = await ArtifactAdmissionService( + session, + settings, + namespace, + ).admit(request) + replay = await ArtifactAdmissionService( + session, + settings, + namespace, + ).admit(request) + + assert replay.replayed is True + assert replay.attempt_id == first.attempt_id + assert replay.charge_ids == first.charge_ids + assert await _count(session, ArtifactPutAttempt) == 1 + assert await _count(session, ArtifactAdmissionCharge) == 4 + assert await _count(session, ArtifactPutAttemptCharge) == 4 + finally: + await engine.dispose() + + +async def test_exact_replay_reacquires_released_charges_under_capacity( + admission_database_env: str, + tmp_path: Path, +) -> None: + settings = _settings(tmp_path, maximum_bytes=4) + namespace = _namespace(settings) + context = _context() + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + async with minted_source(tmp_path / "first-source", b"aaaa") as first_source: + async with minted_source( + tmp_path / "second-source", b"bbbb" + ) as second_source: + first_sha256 = first_source.commitment.sha256 + second_sha256 = second_source.commitment.sha256 + _, _, item_ids = await _seed_contributor_items( + session, + context=context, + commitments=( + ( + first_source.commitment.sha256, + first_source.commitment.byte_count, + first_source.commitment.media_type, + ), + ( + second_source.commitment.sha256, + second_source.commitment.byte_count, + second_source.commitment.media_type, + ), + ), + ) + first_request = ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_ids[0]), + source=first_source, + ) + second_request = ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_ids[1]), + source=second_source, + ) + first = await ArtifactAdmissionService( + session, settings, namespace + ).admit(first_request) + first_attempt = await session.get( + ArtifactPutAttempt, + str(first.attempt_id), + ) + assert first_attempt is not None + first_attempt.status = "absent_replay_required" + counters = ( + await session.execute(select(ArtifactAdmissionScope)) + ).scalars().all() + first_charges = ( + await session.execute( + select(ArtifactAdmissionCharge).where( + ArtifactAdmissionCharge.sha256 + == first_sha256 + ) + ) + ).scalars().all() + released_at = datetime.now(UTC) + for charge in first_charges: + charge.state = "released" + charge.released_at = released_at + charge.cas_version += 1 + for counter in counters: + counter.counted_bytes = 0 + counter.cas_version += 1 + await session.commit() + + await ArtifactAdmissionService(session, settings, namespace).admit( + second_request + ) + with pytest.raises(ArtifactAdmissionCapacityError): + await ArtifactAdmissionService( + session, settings, namespace + ).admit(first_request) + + second_charges = ( + await session.execute( + select(ArtifactAdmissionCharge).where( + ArtifactAdmissionCharge.sha256 + == second_sha256 + ) + ) + ).scalars().all() + counters = ( + await session.execute(select(ArtifactAdmissionScope)) + ).scalars().all() + for charge in second_charges: + charge.state = "released" + charge.released_at = datetime.now(UTC) + charge.cas_version += 1 + for counter in counters: + counter.counted_bytes = 0 + counter.cas_version += 1 + await session.commit() + + replay = await ArtifactAdmissionService( + session, settings, namespace + ).admit(first_request) + + assert replay.replayed is True + assert replay.attempt_id == first.attempt_id + refreshed_first_charges = ( + await session.execute( + select(ArtifactAdmissionCharge).where( + ArtifactAdmissionCharge.sha256 + == first_sha256 + ) + ) + ).scalars().all() + refreshed_counters = ( + await session.execute(select(ArtifactAdmissionScope)) + ).scalars().all() + assert {charge.state for charge in refreshed_first_charges} == { + "provisional" + } + assert {charge.released_at for charge in refreshed_first_charges} == {None} + assert {counter.counted_bytes for counter in refreshed_counters} == {4} + assert await _count(session, ArtifactPutAttempt) == 2 + finally: + await engine.dispose() + + +async def test_contributor_admission_rejects_cross_project_task_relationship( + admission_database_env: str, + tmp_path: Path, +) -> None: + settings = _settings(tmp_path) + namespace = _namespace(settings) + context = _context() + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + async with minted_source(tmp_path / "scratch-source", b"contributor") as source: + _, _, item_ids = await _seed_contributor_items( + session, + context=context, + commitments=( + ( + source.commitment.sha256, + source.commitment.byte_count, + source.commitment.media_type, + ), + ), + ) + item = await session.get(ArtifactUploadItem, item_ids[0]) + assert item is not None + upload_session = await session.get(ArtifactUploadSession, item.session_id) + assert upload_session is not None + unrelated_project_id = str(uuid4()) + session.add( + Project( + id=unrelated_project_id, + name="Unrelated admission project", + slug=f"unrelated-{unrelated_project_id}", + ) + ) + await session.flush() + upload_session.project_id = unrelated_project_id + await session.commit() + + with pytest.raises( + ArtifactAdmissionRelationshipError, + match="contributor upload item relationship is unavailable", + ): + await ArtifactAdmissionService(session, settings, namespace).admit( + ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_ids[0]), + source=source, + ) + ) + + assert await _count(session, ArtifactStorageNamespace) == 0 + assert await _count(session, ArtifactAdmissionScope) == 0 + assert await _count(session, ArtifactAdmissionCharge) == 0 + assert await _count(session, ArtifactPutAttempt) == 0 + assert await _count(session, AuditEvent) == 0 + finally: + await engine.dispose() + + +async def test_same_content_distinct_operations_deduplicate_scope_bytes( + admission_database_env: str, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + settings = _settings(tmp_path) + namespace = _namespace(settings) + contexts = (_context(), _context()) + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + original_reserve = ArtifactRepository.ensure_and_lock_admission_scopes + ready_count = 0 + ready_lock = asyncio.Lock() + start = asyncio.Event() + + async def synchronized_reserve(repository, scopes): + nonlocal ready_count + async with ready_lock: + ready_count += 1 + if ready_count == 2: + start.set() + await start.wait() + return await original_reserve(repository, scopes) + + monkeypatch.setattr( + ArtifactRepository, + "ensure_and_lock_admission_scopes", + synchronized_reserve, + ) + try: + async with minted_source(tmp_path / "scratch-source", b"same") as source: + async with factory() as seed_session: + commitment = ( + source.commitment.sha256, + source.commitment.byte_count, + source.commitment.media_type, + ) + item_ids = [] + for context in contexts: + _, _, context_item_ids = await _seed_contributor_items( + seed_session, + context=context, + commitments=(commitment,), + ) + item_ids.append(context_item_ids[0]) + seed_session.add( + ArtifactStorageNamespace( + id="primary", + backend=namespace.backend, + adapter=namespace.adapter, + provider_profile=namespace.provider_profile, + namespace_descriptor=namespace.namespace_descriptor, + namespace_fingerprint=namespace.namespace_fingerprint, + ) + ) + await seed_session.commit() + + async def admit(item_id: str, context: AuthorizationContext): + async with factory() as session: + return await ArtifactAdmissionService( + session, + settings, + namespace, + ).admit( + ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_id), + source=source, + ) + ) + + results = await asyncio.gather( + *(admit(item_id, context) for item_id, context in zip(item_ids, contexts)) + ) + + async with factory() as session: + assert all(result.replayed is False for result in results) + counters = ( + await session.execute(select(ArtifactAdmissionScope)) + ).scalars().all() + assert {counter.counted_bytes for counter in counters} == {4} + assert await _count(session, ArtifactAdmissionCharge) == 7 + assert await _count(session, ArtifactPutAttempt) == 2 + assert await _count(session, ArtifactPutAttemptCharge) == 8 + finally: + await engine.dispose() + + +async def test_completed_charge_deduplicates_and_released_charge_is_reacquired( + admission_database_env: str, + tmp_path: Path, +) -> None: + settings = _settings(tmp_path) + namespace = _namespace(settings) + context = _context() + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with minted_source(tmp_path / "scratch-source", b"same") as source: + async with factory() as session: + commitment = ( + source.commitment.sha256, + source.commitment.byte_count, + source.commitment.media_type, + ) + _, _, item_ids = await _seed_contributor_items( + session, + context=context, + commitments=(commitment, commitment, commitment), + ) + service = ArtifactAdmissionService(session, settings, namespace) + await service.admit( + ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_ids[0]), + source=source, + ) + ) + + charges = ( + await session.execute(select(ArtifactAdmissionCharge)) + ).scalars().all() + completed_at = datetime.now(UTC) + for charge in charges: + charge.state = "completed" + charge.completed_at = completed_at + await session.commit() + + await service.admit( + ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_ids[1]), + source=source, + ) + ) + counters = ( + await session.execute(select(ArtifactAdmissionScope)) + ).scalars().all() + assert {counter.counted_bytes for counter in counters} == {4} + + released_at = datetime.now(UTC) + for charge in charges: + charge.state = "released" + charge.completed_at = None + charge.released_at = released_at + for counter in counters: + counter.counted_bytes = 0 + counter.cas_version += 1 + await session.commit() + + await service.admit( + ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_ids[2]), + source=source, + ) + ) + + refreshed_charges = ( + await session.execute(select(ArtifactAdmissionCharge)) + ).scalars().all() + refreshed_counters = ( + await session.execute(select(ArtifactAdmissionScope)) + ).scalars().all() + assert {charge.state for charge in refreshed_charges} == {"provisional"} + assert {charge.released_at for charge in refreshed_charges} == {None} + assert {charge.cas_version for charge in refreshed_charges} == {1} + assert {counter.counted_bytes for counter in refreshed_counters} == {4} + assert await _count(session, ArtifactAdmissionCharge) == 4 + assert await _count(session, ArtifactPutAttempt) == 3 + assert await _count(session, ArtifactPutAttemptCharge) == 12 + finally: + await engine.dispose() + + +async def test_concurrent_distinct_content_cannot_oversubscribe_any_scope( + admission_database_env: str, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + settings = _settings(tmp_path, maximum_bytes=6) + namespace = _namespace(settings) + contexts = (_context(), _context()) + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + original_reserve = ArtifactRepository.ensure_and_lock_admission_scopes + ready_count = 0 + ready_lock = asyncio.Lock() + start = asyncio.Event() + + async def synchronized_reserve(repository, scopes): + nonlocal ready_count + async with ready_lock: + ready_count += 1 + if ready_count == 2: + start.set() + await start.wait() + return await original_reserve(repository, scopes) + + monkeypatch.setattr( + ArtifactRepository, + "ensure_and_lock_admission_scopes", + synchronized_reserve, + ) + try: + async with minted_source(tmp_path / "scratch-a", b"aaaa") as first_source: + async with minted_source(tmp_path / "scratch-b", b"bbbb") as second_source: + async with factory() as seed_session: + item_ids = [] + for context, source in zip( + contexts, + (first_source, second_source), + ): + _, _, context_item_ids = await _seed_contributor_items( + seed_session, + context=context, + commitments=( + ( + source.commitment.sha256, + source.commitment.byte_count, + source.commitment.media_type, + ), + ), + ) + item_ids.append(context_item_ids[0]) + seed_session.add( + ArtifactStorageNamespace( + id="primary", + backend=namespace.backend, + adapter=namespace.adapter, + provider_profile=namespace.provider_profile, + namespace_descriptor=namespace.namespace_descriptor, + namespace_fingerprint=namespace.namespace_fingerprint, + ) + ) + await seed_session.commit() + + async def admit(item_id: str, source, context: AuthorizationContext): + async with factory() as session: + return await ArtifactAdmissionService( + session, + settings, + namespace, + ).admit( + ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_id), + source=source, + ) + ) + + outcomes = await asyncio.gather( + admit(item_ids[0], first_source, contexts[0]), + admit(item_ids[1], second_source, contexts[1]), + return_exceptions=True, + ) + + assert sum(not isinstance(value, BaseException) for value in outcomes) == 1 + assert sum( + isinstance(value, ArtifactAdmissionCapacityError) for value in outcomes + ) == 1 + async with factory() as session: + counters = ( + await session.execute(select(ArtifactAdmissionScope)) + ).scalars().all() + assert {counter.counted_bytes for counter in counters} == {4} + assert await _count(session, ArtifactPutAttempt) == 1 + assert await _count(session, ArtifactAdmissionCharge) == 4 + finally: + await engine.dispose() + + +async def test_capacity_failure_rolls_back_namespace_scopes_charges_and_attempt( + admission_database_env: str, + tmp_path: Path, +) -> None: + settings = _settings(tmp_path, maximum_bytes=3) + namespace = _namespace(settings) + context = _context() + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + async with minted_source(tmp_path / "scratch-source", b"four") as source: + _, _, item_ids = await _seed_contributor_items( + session, + context=context, + commitments=( + ( + source.commitment.sha256, + source.commitment.byte_count, + source.commitment.media_type, + ), + ), + ) + with pytest.raises(ArtifactAdmissionCapacityError): + await ArtifactAdmissionService( + session, + settings, + namespace, + ).admit( + ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_ids[0]), + source=source, + ) + ) + + assert await _count(session, ArtifactStorageNamespace) == 0 + assert await _count(session, ArtifactAdmissionScope) == 0 + assert await _count(session, ArtifactAdmissionCharge) == 0 + assert await _count(session, ArtifactPutAttempt) == 0 + finally: + await engine.dispose() + + +async def test_changed_input_for_existing_operation_fails_closed( + admission_database_env: str, + tmp_path: Path, +) -> None: + settings = _settings(tmp_path) + namespace = _namespace(settings) + context = _context() + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + async with minted_source(tmp_path / "scratch-a", b"aaaa") as first_source: + _, _, item_ids = await _seed_contributor_items( + session, + context=context, + commitments=( + ( + first_source.commitment.sha256, + first_source.commitment.byte_count, + first_source.commitment.media_type, + ), + ), + ) + await ArtifactAdmissionService(session, settings, namespace).admit( + ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_ids[0]), + source=first_source, + ) + ) + + async with minted_source(tmp_path / "scratch-b", b"bbbb") as changed_source: + item = await session.get(ArtifactUploadItem, item_ids[0]) + assert item is not None + item.expected_sha256 = changed_source.commitment.sha256 + item.expected_size = changed_source.commitment.byte_count + await session.commit() + with pytest.raises(ArtifactAdmissionConflictError): + await ArtifactAdmissionService(session, settings, namespace).admit( + ContributorArtifactAdmissionRequest( + authorization_context=context, + upload_item_id=UUID(item_ids[0]), + source=changed_source, + ) + ) + finally: + await engine.dispose() + + +async def test_checker_output_requires_exact_active_fixed_service_identity( + admission_database_env: str, + tmp_path: Path, +) -> None: + settings = _settings(tmp_path) + namespace = _namespace(settings) + actor_id = uuid4() + link_id = uuid4() + context = _context( + actor_profile_id=actor_id, + identity_link_id=link_id, + actor_kind=ActorKind.SERVICE, + ) + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + project_id, task_id, checker_run_id = await _seed_checker_output_relationships( + session + ) + session.add( + ActorProfile( + id=str(actor_id), + actor_kind="service", + status="active", + provisioning_method="manual_service_provisioning", + service_identity=ServiceIdentity.ARTIFACT_CHECKER_OUTPUT.value, + created_by="test", + ) + ) + session.add( + ActorIdentityLink( + id=str(link_id), + actor_profile_id=str(actor_id), + issuer="https://issuer.example.test", + subject="checker-output-service", + subject_kind="service", + status="active", + linked_by="test", + ) + ) + await session.commit() + canonical_task = await session.get(WorkstreamTask, task_id) + assert canonical_task is not None + unrelated_task_id = str(uuid4()) + session.add( + WorkstreamTask( + id=unrelated_task_id, + project_id=project_id, + locked_guide_version=canonical_task.locked_guide_version, + locked_post_submit_checker_policy_id=( + canonical_task.locked_post_submit_checker_policy_id + ), + locked_post_submit_checker_policy_version=( + canonical_task.locked_post_submit_checker_policy_version + ), + locked_post_submit_checker_policy_hash=( + canonical_task.locked_post_submit_checker_policy_hash + ), + locked_post_submit_checker_policy_body=( + canonical_task.locked_post_submit_checker_policy_body + ), + locked_review_policy_version=canonical_task.locked_review_policy_version, + locked_revision_policy_version=( + canonical_task.locked_revision_policy_version + ), + locked_payment_policy_version=( + canonical_task.locked_payment_policy_version + ), + locked_guide_source_snapshot_id=( + canonical_task.locked_guide_source_snapshot_id + ), + locked_guide_source_snapshot_hash=( + canonical_task.locked_guide_source_snapshot_hash + ), + locked_effective_project_submission_artifact_policy_id=( + canonical_task.locked_effective_project_submission_artifact_policy_id + ), + locked_effective_project_submission_artifact_policy_hash=( + canonical_task.locked_effective_project_submission_artifact_policy_hash + ), + locked_pre_submit_checker_policy_id=( + canonical_task.locked_pre_submit_checker_policy_id + ), + locked_pre_submit_checker_bundle_hash=( + canonical_task.locked_pre_submit_checker_bundle_hash + ), + title="Unrelated checker task", + description="Must not own the checker output.", + status="draft", + created_by="setup-actor", + ) + ) + await session.flush() + checker_run = await session.get(CheckerRun, checker_run_id) + assert checker_run is not None + checker_run.task_id = unrelated_task_id + await session.commit() + + async with minted_source(tmp_path / "scratch-source", b"checker") as source: + service = ArtifactAdmissionService(session, settings, namespace) + forged = context.model_copy(update={"identity_link_id": uuid4()}) + with pytest.raises( + ArtifactAdmissionRelationshipError, + match="service identity is unavailable", + ): + await service.admit( + CheckerOutputArtifactAdmissionRequest( + authorization_context=forged, + checker_run_id=UUID(checker_run_id), + logical_role="platform-review", + source=source, + ) + ) + assert await _count(session, ArtifactStorageNamespace) == 0 + assert await _count(session, ArtifactAdmissionScope) == 0 + assert await _count(session, ArtifactAdmissionCharge) == 0 + assert await _count(session, ArtifactPutAttempt) == 0 + await session.rollback() + + with pytest.raises( + ArtifactAdmissionRelationshipError, + match="checker run relationship is unavailable", + ): + await service.admit( + CheckerOutputArtifactAdmissionRequest( + authorization_context=context, + checker_run_id=UUID(checker_run_id), + logical_role="platform-review", + source=source, + ) + ) + assert await _count(session, ArtifactStorageNamespace) == 0 + assert await _count(session, ArtifactAdmissionScope) == 0 + assert await _count(session, ArtifactAdmissionCharge) == 0 + assert await _count(session, ArtifactPutAttempt) == 0 + await session.rollback() + + checker_run = await session.get(CheckerRun, checker_run_id) + assert checker_run is not None + checker_run.task_id = task_id + await session.commit() + + request = CheckerOutputArtifactAdmissionRequest( + authorization_context=context, + checker_run_id=UUID(checker_run_id), + logical_role="platform-review", + source=source, + ) + result = await service.admit(request) + replay = await service.admit(request) + + attempt = await session.get(ArtifactPutAttempt, str(result.attempt_id)) + scopes = ( + await session.execute( + select(ArtifactAdmissionScope).order_by( + ArtifactAdmissionScope.scope_type, + ArtifactAdmissionScope.scope_id, + ) + ) + ).scalars().all() + links = ( + await session.execute( + select(ArtifactPutAttemptCharge).where( + ArtifactPutAttemptCharge.attempt_id == str(result.attempt_id) + ) + ) + ).scalars().all() + assert attempt is not None + assert replay.attempt_id == result.attempt_id + assert replay.charge_ids == result.charge_ids + assert attempt.status == "prepared" + assert attempt.producer_request_type == "checker_output" + assert attempt.producer_type == "service_identity" + assert attempt.producer_ref == ServiceIdentity.ARTIFACT_CHECKER_OUTPUT.value + assert attempt.project_id == project_id + assert attempt.task_id == task_id + assert attempt.checker_run_id == checker_run_id + assert attempt.logical_role == "platform-review" + assert attempt.executor_id is None + assert attempt.lease_expires_at is None + assert attempt.next_run_at is None + assert attempt.execution_generation == 0 + assert {scope.scope_type for scope in scopes} == { + "deployment", + "producer", + "project", + "task", + } + assert len(result.charge_ids) == 4 + assert len(links) == 4 + assert await _count(session, ArtifactPutAttempt) == 1 + assert await _count(session, ArtifactAdmissionCharge) == 4 + assert await _count(session, ArtifactContent) == 0 + assert await _count(session, ArtifactReplica) == 0 + assert await _count(session, ArtifactOperationReceipt) == 0 + finally: + await engine.dispose() + + +async def test_invalid_checker_role_precedes_namespace_drift( + admission_database_env: str, + tmp_path: Path, +) -> None: + settings = _settings(tmp_path) + namespace = _namespace(settings) + engine = create_async_engine(admission_database_env) + factory = async_sessionmaker(engine, expire_on_commit=False) + try: + async with factory() as session: + session.add( + ArtifactStorageNamespace( + id="primary", + backend=namespace.backend, + adapter=namespace.adapter, + provider_profile=namespace.provider_profile, + namespace_descriptor=namespace.namespace_descriptor, + namespace_fingerprint="sha256:" + "f" * 64, + ) + ) + await session.commit() + async with minted_source(tmp_path / "scratch-source", b"checker") as source: + with pytest.raises( + ArtifactAdmissionRelationshipError, + match="logical role is invalid", + ): + await ArtifactAdmissionService(session, settings, namespace).admit( + CheckerOutputArtifactAdmissionRequest( + authorization_context=_context(actor_kind=ActorKind.SERVICE), + checker_run_id=uuid4(), + logical_role="é" * 100, + source=source, + ) + ) + assert await _count(session, ArtifactStorageNamespace) == 1 + assert await _count(session, ArtifactAdmissionScope) == 0 + assert await _count(session, ArtifactAdmissionCharge) == 0 + assert await _count(session, ArtifactPutAttempt) == 0 + finally: + await engine.dispose() + + +def test_artifact_admission_migration_preserves_prior_rows_and_round_trips_empty( + isolated_database_env: str, + migration_lock, +) -> None: + config = _alembic_config() + namespace_fingerprint = "sha256:" + "a" * 64 + + async def seed_prior_namespace() -> None: + engine = create_async_engine(isolated_database_env) + try: + async with engine.begin() as connection: + await connection.execute( + text( + "insert into artifact_storage_namespaces " + "(id,backend,adapter,provider_profile,namespace_descriptor," + "namespace_fingerprint) values " + "('primary','local','local','local-v2','{}',:fingerprint)" + ), + {"fingerprint": namespace_fingerprint}, + ) + finally: + await engine.dispose() + + async def state() -> tuple[int, bool]: + engine = create_async_engine(isolated_database_env) + try: + async with engine.connect() as connection: + count = await connection.scalar( + text("select count(*) from artifact_storage_namespaces") + ) + table_exists = await connection.scalar( + text("select to_regclass('artifact_put_attempts') is not null") + ) + return int(count or 0), bool(table_exists) + finally: + await engine.dispose() + + async def cleanup() -> None: + engine = create_async_engine(isolated_database_env) + try: + async with engine.begin() as connection: + await connection.execute( + text("truncate table artifact_storage_namespaces cascade") + ) + finally: + await engine.dispose() + + with migration_lock(): + try: + asyncio.run(_reset_admission_test_schema(isolated_database_env)) + command.upgrade(config, "0026_actor_profile_lifecycle") + asyncio.run(seed_prior_namespace()) + command.upgrade(config, "0028_artifact_admission") + assert asyncio.run(state()) == (1, True) + command.downgrade(config, "0026_actor_profile_lifecycle") + assert asyncio.run(state()) == (1, False) + command.upgrade(config, "0028_artifact_admission") + assert asyncio.run(state()) == (1, True) + asyncio.run(cleanup()) + finally: + asyncio.run(_reset_admission_test_schema(isolated_database_env)) + + +def test_artifact_admission_migration_refuses_populated_downgrade( + isolated_database_env: str, + migration_lock, +) -> None: + config = _alembic_config() + + async def seed_attempt_only() -> None: + engine = create_async_engine(isolated_database_env) + try: + factory = async_sessionmaker(engine, expire_on_commit=False) + async with factory() as session: + context = _context() + project_id, item_id = await _seed_guide( + session, + context=context, + content_hash="sha256:" + "b" * 64, + media_type="text/markdown", + ) + namespace_fingerprint = "sha256:" + "c" * 64 + await session.execute( + text( + "insert into artifact_storage_namespaces " + "(id,backend,adapter,provider_profile,namespace_descriptor," + "namespace_fingerprint) values " + "('primary','local','local','local-v2','{}',:fingerprint)" + ), + {"fingerprint": namespace_fingerprint}, + ) + await session.execute( + text( + "insert into artifact_put_attempts " + "(id,producer_request_type,producer_type,producer_ref," + "project_id,guide_source_item_id,sha256,byte_count,media_type," + "storage_namespace_id,namespace_fingerprint,canonical_target," + "operation_identity,request_digest,status," + "execution_generation,cas_version,prepared_at) values " + "(:id,'guide','actor_profile',:producer_ref,:project_id," + ":item_id,:sha256,1,'text/markdown','primary',:fingerprint," + ":target,:operation_identity,:request_digest,'prepared'," + "0,0,now())" + ), + { + "id": str(uuid4()), + "producer_ref": str(context.actor_profile_id), + "project_id": project_id, + "item_id": item_id, + "sha256": "sha256:" + "b" * 64, + "fingerprint": namespace_fingerprint, + "target": "sha256/bb/" + "b" * 62, + "operation_identity": "sha256:" + "d" * 64, + "request_digest": "sha256:" + "e" * 64, + }, + ) + await session.commit() + finally: + await engine.dispose() + + async def cleanup() -> None: + engine = create_async_engine(isolated_database_env) + try: + async with engine.begin() as connection: + await connection.execute( + text( + "truncate table artifact_put_attempt_charges, " + "artifact_put_attempts, artifact_admission_charges, " + "artifact_admission_scopes cascade" + ) + ) + finally: + await engine.dispose() + + with migration_lock(): + try: + asyncio.run(_reset_admission_test_schema(isolated_database_env)) + command.upgrade(config, "0028_artifact_admission") + asyncio.run(seed_attempt_only()) + with pytest.raises( + RuntimeError, + match="cannot downgrade populated artifact admission ledger", + ): + command.downgrade(config, "0026_actor_profile_lifecycle") + asyncio.run(cleanup()) + command.downgrade(config, "0026_actor_profile_lifecycle") + finally: + asyncio.run(_reset_admission_test_schema(isolated_database_env)) diff --git a/backend/tests/test_artifact_architecture.py b/backend/tests/test_artifact_architecture.py index 65aed7da1..ea389a79c 100644 --- a/backend/tests/test_artifact_architecture.py +++ b/backend/tests/test_artifact_architecture.py @@ -35,6 +35,13 @@ "ArtifactBindingResourceType", } RAW_TYPES = {"ArtifactStore", "ArtifactStorageOrchestrator"} +INTERNAL_ADMISSION_TYPES = { + "ArtifactAdmissionService", + "ArtifactAdmissionResult", + "CheckerOutputArtifactAdmissionRequest", + "ContributorArtifactAdmissionRequest", + "GuideArtifactAdmissionRequest", +} PROVIDER_METHODS = {"put", "observe_put_result", "open", "head"} CONCRETE_ADAPTER_MODULES = { "app.adapters.artifacts.local", @@ -76,7 +83,7 @@ def test_product_api_and_workers_cannot_import_or_inject_raw_artifact_types() -> for node in ast.walk(tree): if isinstance(node, ast.ImportFrom): imported = {alias.name for alias in node.names} - forbidden = imported & RAW_TYPES + forbidden = imported & (RAW_TYPES | INTERNAL_ADMISSION_TYPES) if forbidden: violations.append(f"{path.relative_to(BACKEND_ROOT)} imports {sorted(forbidden)}") if ( @@ -104,6 +111,42 @@ def test_product_api_and_workers_cannot_import_or_inject_raw_artifact_types() -> assert violations == [] +def test_artifact_domain_has_no_provider_execution_during_admission_foundation() -> None: + """Keep 02C1 free of all provider write and acknowledgement execution.""" + violations: list[str] = [] + for path in _python_files(APP_ROOT / "modules" / "artifacts"): + for node in ast.walk(_tree(path)): + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and node.func.attr in {"put", "observe_put_result"} + ): + violations.append( + f"{path.relative_to(BACKEND_ROOT)} calls {node.func.attr}" + ) + assert violations == [] + + +def test_artifact_domain_does_not_import_adapter_modules() -> None: + """Keep provider-neutral artifact rules independent from adapters.""" + violations: list[str] = [] + for path in _python_files(APP_ROOT / "modules" / "artifacts"): + for node in ast.walk(_tree(path)): + if isinstance(node, ast.ImportFrom) and ( + node.module or "" + ).startswith("app.adapters.artifacts"): + violations.append( + f"{path.relative_to(BACKEND_ROOT)} imports {node.module}" + ) + if isinstance(node, ast.Import): + for alias in node.names: + if alias.name.startswith("app.adapters.artifacts"): + violations.append( + f"{path.relative_to(BACKEND_ROOT)} imports {alias.name}" + ) + assert violations == [] + + def test_concrete_adapter_construction_has_one_composition_path() -> None: factory_calls: list[Path] = [] adapter_calls: list[Path] = [] @@ -296,3 +339,18 @@ def test_scratch_cleanup_worker_has_no_product_or_database_state() -> None: assert not any( module.startswith(forbidden_import_prefixes) for module in imported_modules ) + + +def test_artifact_repository_does_not_own_actor_persistence() -> None: + """Keep canonical actor models and queries behind the actors-owned proof API.""" + path = APP_ROOT / "modules" / "artifacts" / "repository.py" + tree = _tree(path) + imported_modules: set[str] = set() + for node in ast.walk(tree): + if isinstance(node, ast.ImportFrom) and node.module is not None: + imported_modules.add(node.module) + elif isinstance(node, ast.Import): + imported_modules.update(alias.name for alias in node.names) + assert not any(module.startswith("app.modules.actors") for module in imported_modules) + assert "actor_profiles" not in path.read_text() + assert "actor_identity_links" not in path.read_text() diff --git a/backend/tests/test_artifact_cleanup_wiring.py b/backend/tests/test_artifact_cleanup_wiring.py index 1dd1c501f..20535a527 100644 --- a/backend/tests/test_artifact_cleanup_wiring.py +++ b/backend/tests/test_artifact_cleanup_wiring.py @@ -12,10 +12,12 @@ from app.core.config import Settings, get_settings from app.main import create_app from app.interfaces.artifacts import ArtifactProviderLiveProofRequiredError +from tests.artifact_store_helpers import artifact_admission_limit_settings def _enabled_settings(tmp_path: Path, **changes: object) -> Settings: values: dict[str, object] = { + **artifact_admission_limit_settings(), "environment": "test", "artifact_store_backend": "local", "artifact_local_root": tmp_path / "store", @@ -81,6 +83,7 @@ async def test_production_auth_validation_precedes_artifact_startup( events: list[str] = [] settings = Settings( + **artifact_admission_limit_settings(), environment="production", artifact_store_backend="s3_compatible", artifact_scratch_root=tmp_path / "scratch", @@ -164,6 +167,7 @@ async def test_aws_s3_startup_requires_live_provider_proof( monkeypatch.setattr(main_module, "build_auth_verifier", lambda _settings: None) app = create_app( Settings( + **artifact_admission_limit_settings(), environment="production", artifact_store_backend="s3_compatible", artifact_scratch_root=tmp_path / "scratch", @@ -351,6 +355,7 @@ def test_aws_s3_is_runtime_ineligible_for_celery_and_cleanup_task( """Apply the same inactive-provider guard to API and worker entry points.""" celery_module, artifacts_module = _load_worker_modules(monkeypatch, request) settings = Settings( + **artifact_admission_limit_settings(), environment="production", celery_task_always_eager=True, artifact_store_backend="s3_compatible", diff --git a/backend/tests/test_artifacts.py b/backend/tests/test_artifacts.py index 47a2086af..67ccfcaee 100644 --- a/backend/tests/test_artifacts.py +++ b/backend/tests/test_artifacts.py @@ -1,1107 +1,108 @@ -"""PostgreSQL integration tests for the ArtifactStore v2 orchestration boundary.""" +"""Focused tests for artifact namespace fencing after direct-write removal.""" from __future__ import annotations -import asyncio -from collections.abc import AsyncIterator -from datetime import UTC, datetime, timedelta from pathlib import Path -from unittest.mock import AsyncMock -from uuid import uuid4 -from alembic import command -from alembic.config import Config import pytest -from sqlalchemy import select, text -from sqlalchemy.exc import IntegrityError -from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap from app.core.config import Settings -from app.core.hashing import canonical_json_hash -from app.interfaces.artifacts import ( - ArtifactByteRange, - ArtifactConfigurationError, - ArtifactInputMismatchError, - ArtifactIntegrityError, - ArtifactObjectHead, - ArtifactPutObservation, - ArtifactPutResult, - ArtifactStoreUnavailableError, - ArtifactStoreNamespaceIdentity, -) from app.interfaces.external_services import ExternalServiceAdapterIdentity from app.modules.artifacts.models import ( - ArtifactContent, ArtifactOperationReceipt, ArtifactReplica, ArtifactStorageNamespace, - ArtifactUploadItem, - ArtifactUploadSession, ) from app.modules.artifacts.service import ( - ArtifactIngestStateError, ArtifactStorageNamespaceError, - ArtifactStorageNamespaceSpec, - ArtifactStorageOrchestrator, - ProviderAttemptFence, artifact_storage_namespace_spec, - validate_artifact_storage_namespace_at_startup, -) -import app.modules.artifacts.service as artifact_service_module -from app.modules.artifacts.sources import ArtifactCommitment, CommittedArtifactSource -from app.modules.projects.models import Project -from tests.artifact_store_helpers import ( - local_namespace_claim, - minted_source as _minted_source, ) - - -def _alembic_config() -> Config: - root = Path(__file__).resolve().parents[1] - config = Config(str(root / "alembic.ini")) - config.set_main_option("script_location", str(root / "alembic")) - return config - - -@pytest.fixture -def artifact_database_env(isolated_database_env: str, migration_lock) -> str: - """Provide one fully migrated empty PostgreSQL database per test.""" - config = _alembic_config() - with migration_lock(): - asyncio.run(_truncate_artifacts_if_present(isolated_database_env)) - command.downgrade(config, "base") - command.upgrade(config, "head") - yield isolated_database_env - with migration_lock(): - asyncio.run(_truncate_artifacts(isolated_database_env)) - command.downgrade(config, "base") - - -async def _truncate_artifacts(database_url: str) -> None: - engine = create_async_engine(database_url) - try: - async with engine.begin() as connection: - await connection.execute( - text( - "truncate table artifact_storage_namespaces, artifact_upload_sessions, " - "artifact_contents cascade" - ) - ) - finally: - await engine.dispose() - - -async def _truncate_artifacts_if_present(database_url: str) -> None: - """Clear only artifact tables that exist at the database's current revision.""" - engine = create_async_engine(database_url) - candidates = ( - "artifact_storage_namespaces", - "artifact_upload_sessions", - "artifact_contents", - ) - try: - async with engine.begin() as connection: - existing = [ - table_name - for table_name in candidates - if await connection.scalar( - text("select to_regclass(:table_name) is not null"), - {"table_name": f"public.{table_name}"}, - ) - ] - if existing: - await connection.execute(text(f"truncate table {', '.join(existing)} cascade")) - finally: - await engine.dispose() +from tests.artifact_store_helpers import artifact_admission_limit_settings def _settings(tmp_path: Path) -> Settings: - (tmp_path / "store").mkdir(mode=0o700, exist_ok=True) + root = tmp_path / "store" + root.mkdir(mode=0o700, parents=True) return Settings( + **artifact_admission_limit_settings(), environment="test", artifact_store_backend="local", - artifact_local_root=tmp_path / "store", + artifact_local_root=root, artifact_scratch_root=tmp_path / "scratch", artifact_scratch_minimum_free_bytes=0, ) -async def _seed_reserved_item( - session, - commitment: ArtifactCommitment, -) -> str: - project_id = str(uuid4()) - session_id = str(uuid4()) - item_id = str(uuid4()) - session.add(Project(id=project_id, name="Artifact project", slug=f"artifact-{project_id}")) - await session.flush() - session.add( - ArtifactUploadSession( - id=session_id, - actor_id="actor-1", - project_id=project_id, - permitted_roles=["submission"], - state="open", - maximum_bytes=1024, - current_bytes=0, - reserved_bytes=commitment.byte_count, - maximum_items=1, - current_items=0, - reserved_items=1, - expires_at=datetime.now(UTC) + timedelta(minutes=5), - cas_version=0, - ) - ) - await session.flush() - session.add( - ArtifactUploadItem( - id=item_id, - session_id=session_id, - logical_role="submission", - display_name="result.bin", - media_type=commitment.media_type, - reserved_bytes=commitment.byte_count, - expected_sha256=commitment.sha256, - expected_size=commitment.byte_count, - idempotency_key=f"put-{item_id}", - request_digest=canonical_json_hash( - { - "sha256": commitment.sha256, - "byte_count": commitment.byte_count, - "media_type": commitment.media_type, - } - ), - state="reserved", - cas_version=0, - ) - ) - await session.commit() - return item_id - - -async def _seed_reserved_items_in_one_session( - session, - commitments: tuple[ArtifactCommitment, ...], -) -> tuple[str, ...]: - """Create multiple independently fenced items under one aggregate ledger.""" - project_id = str(uuid4()) - session_id = str(uuid4()) - item_ids = tuple(str(uuid4()) for _ in commitments) - total_bytes = sum(commitment.byte_count for commitment in commitments) - session.add(Project(id=project_id, name="Artifact project", slug=f"artifact-{project_id}")) - await session.flush() - session.add( - ArtifactUploadSession( - id=session_id, - actor_id="actor-1", - project_id=project_id, - permitted_roles=["submission"], - state="open", - maximum_bytes=1024, - current_bytes=0, - reserved_bytes=total_bytes, - maximum_items=len(commitments), - current_items=0, - reserved_items=len(commitments), - expires_at=datetime.now(UTC) + timedelta(minutes=5), - cas_version=0, +def _bootstrap(settings: Settings) -> LocalStorageBootstrap: + assert settings.artifact_local_root is not None + return LocalStorageBootstrap( + LocalStorageAdapter( + root=settings.artifact_local_root, + buffer_bytes=settings.artifact_stream_buffer_bytes, + lock_timeout_seconds=settings.artifact_operation_lock_timeout_seconds, ) ) - await session.flush() - for item_id, commitment in zip(item_ids, commitments, strict=True): - session.add( - ArtifactUploadItem( - id=item_id, - session_id=session_id, - logical_role="submission", - display_name=f"{item_id}.bin", - media_type=commitment.media_type, - reserved_bytes=commitment.byte_count, - expected_sha256=commitment.sha256, - expected_size=commitment.byte_count, - idempotency_key=f"put-{item_id}", - request_digest=canonical_json_hash( - { - "sha256": commitment.sha256, - "byte_count": commitment.byte_count, - "media_type": commitment.media_type, - } - ), - state="reserved", - cas_version=0, - ) - ) - await session.commit() - return item_ids -@pytest.mark.asyncio -async def test_put_acknowledgement_stops_at_pending_verification( - artifact_database_env: str, +def test_namespace_descriptor_is_canonical_and_excludes_local_path( tmp_path: Path, ) -> None: - content = b"artifact-v2" - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) settings = _settings(tmp_path) - bootstrap = LocalStorageBootstrap(LocalStorageAdapter(root=tmp_path / "store", buffer_bytes=2)) - namespace = artifact_storage_namespace_spec(settings, bootstrap) - store = bootstrap.initialize_after_namespace_claim(local_namespace_claim(bootstrap)) + bootstrap = _bootstrap(settings) try: - async with _minted_source(tmp_path / "scratch", content) as source: - async with factory() as session: - item_id = await _seed_reserved_item(session, source.commitment) - result = await ArtifactStorageOrchestrator( - session, store, namespace - ).put_reserved_item(item_id, source) - async with _minted_source(tmp_path / "scratch-replay", content) as replay_source: - async with factory() as session: - replay_item_id = await _seed_reserved_item(session, replay_source.commitment) - replay = await ArtifactStorageOrchestrator( - session, store, namespace - ).put_reserved_item(replay_item_id, replay_source) - assert replay.replayed is True - - async with factory() as session: - item = await session.get(ArtifactUploadItem, item_id) - replica = (await session.execute(select(ArtifactReplica))).scalar_one() - receipts = (await session.execute(select(ArtifactOperationReceipt))).scalars().all() - receipt = next(value for value in receipts if value.upload_item_id == item_id) - replay_receipt = next( - value for value in receipts if value.upload_item_id == replay_item_id - ) - namespace_row = (await session.execute(select(ArtifactStorageNamespace))).scalar_one() - assert item is not None - assert item.state == "stored_pending_verification" - assert item.provider_object_ref == result.provider_object_ref - assert replica.verification_state == "pending" - assert replica.availability_state == "unknown" - assert replica.integrity_state == "unknown" - assert replica.namespace_fingerprint == namespace_row.namespace_fingerprint - assert receipt.operation == "put" - assert receipt.outcome == "stored_pending_verification" - assert receipt.provider_object_ref == result.provider_object_ref - assert receipt.replayed is False - assert len(receipts) == 2 - assert replay_receipt.replayed is True - assert await store.head(result.provider_object_ref) == ArtifactObjectHead( - result.provider_object_ref, - exists=True, - byte_count=len(content), - media_type=None, - ) - finally: - store.close() - await engine.dispose() - - -class _UnavailableStore: - """Count provider calls and fail with one sanitized retryable error.""" - - identity = ExternalServiceAdapterIdentity("artifact_store", "local") - namespace_identity = ArtifactStoreNamespaceIdentity( - provider_profile="local-v2", - descriptor_items=( - ("private_prefix", "objects/sha256"), - ("private_root_identity", "sha256:" + "0" * 64), - ), - ) - - def __init__(self) -> None: - self.put_calls = 0 - - async def put(self, _source: CommittedArtifactSource) -> ArtifactPutResult: - self.put_calls += 1 - raise ArtifactStoreUnavailableError() - - async def observe_put_result(self, _commitment: ArtifactCommitment) -> ArtifactPutObservation: - raise NotImplementedError - - def open( - self, _provider_object_ref: str, _byte_range: ArtifactByteRange | None = None - ) -> AsyncIterator[bytes]: - raise NotImplementedError - - async def head(self, _provider_object_ref: str) -> ArtifactObjectHead: - raise NotImplementedError - - -class _TerminalStore(_UnavailableStore): - async def put(self, _source: CommittedArtifactSource) -> ArtifactPutResult: - self.put_calls += 1 - raise ArtifactInputMismatchError() - - -class _AcknowledgementUnknownThenReplayStore(_UnavailableStore): - """Lose the first acknowledgement and return an exact replay next.""" - - async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: - self.put_calls += 1 - if self.put_calls == 1: - raise RuntimeError("provider acknowledgement lost") - digest_hex = source.commitment.sha256[7:] - return ArtifactPutResult( - f"sha256/{digest_hex[:2]}/{digest_hex[2:]}", - replayed=True, - ) - - -class _CancelledStore(_UnavailableStore): - async def put(self, _source: CommittedArtifactSource) -> ArtifactPutResult: - self.put_calls += 1 - raise asyncio.CancelledError("provider put cancelled") - - -class _AcknowledgingStore(_UnavailableStore): - async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: - self.put_calls += 1 - digest_hex = source.commitment.sha256[7:] - return ArtifactPutResult( - f"sha256/{digest_hex[:2]}/{digest_hex[2:]}", - replayed=False, - ) - - -class _NamespaceObservingAcknowledgingStore(_AcknowledgingStore): - """Assert the committed namespace exists before the provider is invoked.""" - - def __init__(self, factory: async_sessionmaker[AsyncSession]) -> None: - super().__init__() - self._factory = factory - self.observed_namespace_fingerprint: str | None = None - - async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: - async with self._factory() as session: - namespace = await session.scalar(select(ArtifactStorageNamespace)) - assert namespace is not None - self.observed_namespace_fingerprint = namespace.namespace_fingerprint - return await super().put(source) - - -class _ConcurrentAcknowledgingStore(_AcknowledgingStore): - """Release two same-object acknowledgements into finalization together.""" - - def __init__(self) -> None: - super().__init__() - self._arrivals = 0 - self._release = asyncio.Event() - - async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: - self.put_calls += 1 - self._arrivals += 1 - arrival = self._arrivals - if self._arrivals == 2: - self._release.set() - await asyncio.wait_for(self._release.wait(), timeout=2) - digest_hex = source.commitment.sha256[7:] - return ArtifactPutResult( - f"sha256/{digest_hex[:2]}/{digest_hex[2:]}", - replayed=arrival > 1, - ) - - -class _ConcurrentDistinctAcknowledgingStore(_ConcurrentAcknowledgingStore): - """Release two distinct initial publications into finalization together.""" - - async def put(self, source: CommittedArtifactSource) -> ArtifactPutResult: - result = await super().put(source) - return ArtifactPutResult(result.provider_object_ref, replayed=False) - - -@pytest.mark.asyncio -async def test_concurrent_same_object_finalization_reuses_one_replica( - artifact_database_env: str, - tmp_path: Path, -) -> None: - """Finalize two exact replays with one replica and two receipts.""" - content = b"concurrent-artifact-v2" - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - store = _ConcurrentAcknowledgingStore() - namespace = artifact_storage_namespace_spec(_settings(tmp_path), store) - try: - async with _minted_source(tmp_path / "scratch-first", content) as first_source: - async with _minted_source(tmp_path / "scratch-second", content) as second_source: - async with factory() as session: - first_item_id = await _seed_reserved_item(session, first_source.commitment) - second_item_id = await _seed_reserved_item(session, second_source.commitment) - await ArtifactStorageOrchestrator( - session, store, namespace - ).ensure_storage_namespace() - async with session.begin(): - session.add( - ArtifactContent( - id=str(uuid4()), - sha256=first_source.commitment.sha256, - byte_count=first_source.commitment.byte_count, - media_type=first_source.commitment.media_type, - normalized_display_name="preexisting.bin", - ) - ) - - async def finalize(item_id: str, source: CommittedArtifactSource) -> None: - async with factory() as session: - await ArtifactStorageOrchestrator( - session, store, namespace - ).put_reserved_item(item_id, source) - - await asyncio.gather( - finalize(first_item_id, first_source), - finalize(second_item_id, second_source), - ) - - async with factory() as session: - items = ( - ( - await session.execute( - select(ArtifactUploadItem).where( - ArtifactUploadItem.id.in_((first_item_id, second_item_id)) - ) - ) - ) - .scalars() - .all() - ) - replicas = (await session.execute(select(ArtifactReplica))).scalars().all() - receipts = (await session.execute(select(ArtifactOperationReceipt))).scalars().all() - - assert {item.state for item in items} == {"stored_pending_verification"} - assert len(items) == 2 - assert len(replicas) == 1 - assert replicas[0].verification_state == "pending" - assert replicas[0].availability_state == "unknown" - assert replicas[0].integrity_state == "unknown" - assert len(receipts) == 2 - assert {receipt.replica_id for receipt in receipts} == {replicas[0].id} - finally: - await engine.dispose() - - -@pytest.mark.asyncio -async def test_independent_items_in_one_session_finalize_without_shared_cas_replay( - artifact_database_env: str, - tmp_path: Path, -) -> None: - """Fence provider acknowledgements per item while accounting one session.""" - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - store = _ConcurrentDistinctAcknowledgingStore() - namespace = artifact_storage_namespace_spec(_settings(tmp_path), store) - try: - async with _minted_source(tmp_path / "scratch-first", b"first-item") as first: - async with _minted_source(tmp_path / "scratch-second", b"second-item") as second: - async with factory() as session: - item_ids = await _seed_reserved_items_in_one_session( - session, - (first.commitment, second.commitment), - ) - await ArtifactStorageOrchestrator( - session, store, namespace - ).ensure_storage_namespace() - - async def finalize(item_id: str, source: CommittedArtifactSource) -> None: - async with factory() as candidate_session: - await ArtifactStorageOrchestrator( - candidate_session, store, namespace - ).put_reserved_item(item_id, source) - - await asyncio.gather( - finalize(item_ids[0], first), - finalize(item_ids[1], second), - ) - - async with factory() as session: - items = ( - ( - await session.execute( - select(ArtifactUploadItem).where(ArtifactUploadItem.id.in_(item_ids)) - ) - ) - .scalars() - .all() - ) - upload_session = await session.get(ArtifactUploadSession, items[0].session_id) - receipts = (await session.execute(select(ArtifactOperationReceipt))).scalars().all() - - assert {item.state for item in items} == {"stored_pending_verification"} - assert upload_session is not None - assert upload_session.reserved_bytes == 0 - assert upload_session.reserved_items == 0 - assert upload_session.current_bytes == len(b"first-item") + len(b"second-item") - assert upload_session.current_items == 2 - assert len(receipts) == 2 - finally: - await engine.dispose() - - -@pytest.mark.asyncio -async def test_database_rejects_partial_upload_result_metadata( - artifact_database_env: str, - tmp_path: Path, -) -> None: - """Require both stored-result references or neither in every item state.""" - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - try: - async with _minted_source(tmp_path / "scratch", b"constraint") as source: - async with factory() as session: - item_id = await _seed_reserved_item(session, source.commitment) - content_id = str(uuid4()) - session.add( - ArtifactContent( - id=content_id, - sha256=source.commitment.sha256, - byte_count=source.commitment.byte_count, - media_type=source.commitment.media_type, - normalized_display_name="constraint.bin", - ) - ) - await session.commit() - - with pytest.raises(IntegrityError): - async with engine.begin() as connection: - await connection.execute( - text( - "update artifact_upload_items set content_id = :content_id " - "where id = :item_id" - ), - {"content_id": content_id, "item_id": item_id}, - ) - with pytest.raises(IntegrityError): - async with engine.begin() as connection: - await connection.execute( - text( - "update artifact_upload_items " - "set provider_object_ref = 'sha256/00/object' where id = :item_id" - ), - {"item_id": item_id}, - ) - - async with factory() as session: - item = await session.get(ArtifactUploadItem, item_id) - assert item is not None - assert item.content_id is None - assert item.provider_object_ref is None - finally: - await engine.dispose() - - -def test_reservation_accounting_rejects_drift_without_clamping() -> None: - """Never consume capacity belonging to a different upload item.""" - upload_session = ArtifactUploadSession(reserved_bytes=3, reserved_items=1) - item = ArtifactUploadItem(reserved_bytes=4) - - with pytest.raises(ArtifactIntegrityError, match="reservation accounting"): - ArtifactStorageOrchestrator._apply_committed_accounting(upload_session, item, 4) - - assert upload_session.reserved_bytes == 3 - assert upload_session.reserved_items == 1 - - -class _FinalizationCancelledOrchestrator(ArtifactStorageOrchestrator): - async def _finalize_put( - self, - item_id: str, - commitment: ArtifactCommitment, - result: ArtifactPutResult, - fence: ProviderAttemptFence, - *, - correlation_id: str, - ) -> None: - del item_id, commitment, result, fence, correlation_id - raise asyncio.CancelledError("put finalization cancelled") - - -async def _assert_replay_required_without_durable_facts( - factory: async_sessionmaker, - item_id: str, -) -> None: - async with factory() as session: - item = await session.get(ArtifactUploadItem, item_id) - assert item is not None - upload_session = await session.get(ArtifactUploadSession, item.session_id) - assert upload_session is not None - assert item.state == "replay_required" - assert item.error_code == "artifact_put_acknowledgement_unknown" - assert upload_session.reserved_bytes == item.reserved_bytes - assert upload_session.reserved_items == 1 - assert await session.scalar(select(ArtifactContent)) is None - assert await session.scalar(select(ArtifactReplica)) is None - assert await session.scalar(select(ArtifactOperationReceipt)) is None - - -@pytest.mark.asyncio -async def test_retryable_put_marks_existing_item_replay_required( - artifact_database_env: str, - tmp_path: Path, -) -> None: - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - store = _UnavailableStore() - try: - async with _minted_source(tmp_path / "scratch", b"retry") as source: - async with factory() as session: - item_id = await _seed_reserved_item(session, source.commitment) - orchestrator = ArtifactStorageOrchestrator( - session, - store, - artifact_storage_namespace_spec(_settings(tmp_path), store), - ) - with pytest.raises(ArtifactStoreUnavailableError): - await orchestrator.put_reserved_item(item_id, source) - - async with factory() as session: - item = await session.get(ArtifactUploadItem, item_id) - assert item is not None - assert item.state == "replay_required" - assert item.error_code == "artifact_put_acknowledgement_unknown" - assert await session.scalar(select(ArtifactReplica)) is None - assert await session.scalar(select(ArtifactOperationReceipt)) is None - assert store.put_calls == 1 + spec = artifact_storage_namespace_spec(settings, bootstrap) finally: - await engine.dispose() + bootstrap.close() - -@pytest.mark.asyncio -async def test_same_item_replay_finalizes_once_without_double_accounting( - artifact_database_env: str, - tmp_path: Path, -) -> None: - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - store = _AcknowledgementUnknownThenReplayStore() - content = b"same-item-replay" - try: - async with _minted_source(tmp_path / "scratch-first", content) as source: - async with factory() as session: - item_id = await _seed_reserved_item(session, source.commitment) - orchestrator = ArtifactStorageOrchestrator( - session, - store, - artifact_storage_namespace_spec(_settings(tmp_path), store), - ) - with pytest.raises(RuntimeError, match="acknowledgement lost"): - await orchestrator.put_reserved_item(item_id, source) - - async with _minted_source(tmp_path / "scratch-replay", content) as replay: - async with factory() as session: - result = await ArtifactStorageOrchestrator( - session, - store, - artifact_storage_namespace_spec(_settings(tmp_path), store), - ).put_reserved_item(item_id, replay) - - assert result.replayed is True - async with factory() as session: - item = await session.get(ArtifactUploadItem, item_id) - assert item is not None - upload_session = await session.get(ArtifactUploadSession, item.session_id) - assert upload_session is not None - receipts = (await session.execute(select(ArtifactOperationReceipt))).scalars().all() - replicas = (await session.execute(select(ArtifactReplica))).scalars().all() - contents = (await session.execute(select(ArtifactContent))).scalars().all() - - assert item.state == "stored_pending_verification" - assert item.error_code is None - assert upload_session.reserved_bytes == 0 - assert upload_session.reserved_items == 0 - assert upload_session.current_bytes == len(content) - assert upload_session.current_items == 1 - assert len(contents) == 1 - assert len(replicas) == 1 - assert len(receipts) == 1 - assert receipts[0].upload_item_id == item_id - assert receipts[0].replayed is True - assert store.put_calls == 2 - finally: - await engine.dispose() - - -@pytest.mark.asyncio -async def test_provider_cancellation_persists_replay_required_without_facts( - artifact_database_env: str, - tmp_path: Path, -) -> None: - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - store = _CancelledStore() - try: - async with _minted_source(tmp_path / "scratch", b"provider cancellation") as source: - async with factory() as session: - item_id = await _seed_reserved_item(session, source.commitment) - orchestrator = ArtifactStorageOrchestrator( - session, - store, - artifact_storage_namespace_spec(_settings(tmp_path), store), - ) - with pytest.raises(asyncio.CancelledError, match="provider put cancelled"): - await orchestrator.put_reserved_item(item_id, source) - - await _assert_replay_required_without_durable_facts(factory, item_id) - assert store.put_calls == 1 - finally: - await engine.dispose() + assert spec.backend == "local" + assert spec.adapter == "local" + assert spec.provider_profile == "local-v2" + assert spec.namespace_descriptor["provider_profile"] == "local-v2" + assert str(tmp_path) not in repr(spec.namespace_descriptor) + assert spec.namespace_fingerprint.startswith("sha256:") -@pytest.mark.asyncio -async def test_finalization_cancellation_persists_replay_required_without_facts( - artifact_database_env: str, - tmp_path: Path, -) -> None: - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - store = _AcknowledgingStore() +def test_namespace_fingerprint_changes_when_pinned_root_changes(tmp_path: Path) -> None: + first = _settings(tmp_path / "first") + second = _settings(tmp_path / "second") + first_bootstrap = _bootstrap(first) + second_bootstrap = _bootstrap(second) try: - async with _minted_source(tmp_path / "scratch", b"finalization cancellation") as source: - async with factory() as session: - item_id = await _seed_reserved_item(session, source.commitment) - orchestrator = _FinalizationCancelledOrchestrator( - session, - store, - artifact_storage_namespace_spec(_settings(tmp_path), store), - ) - with pytest.raises(asyncio.CancelledError, match="finalization cancelled"): - await orchestrator.put_reserved_item(item_id, source) - - await _assert_replay_required_without_durable_facts(factory, item_id) - assert store.put_calls == 1 + first_fingerprint = artifact_storage_namespace_spec( + first, + first_bootstrap, + ).namespace_fingerprint + second_fingerprint = artifact_storage_namespace_spec( + second, + second_bootstrap, + ).namespace_fingerprint finally: - await engine.dispose() - + first_bootstrap.close() + second_bootstrap.close() -@pytest.mark.asyncio -async def test_terminal_put_failure_releases_reservation_and_fails_item( - artifact_database_env: str, - tmp_path: Path, -) -> None: - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - store = _TerminalStore() - try: - async with _minted_source(tmp_path / "scratch", b"terminal") as source: - async with factory() as session: - item_id = await _seed_reserved_item(session, source.commitment) - with pytest.raises(ArtifactInputMismatchError): - await ArtifactStorageOrchestrator( - session, - store, - artifact_storage_namespace_spec(_settings(tmp_path), store), - ).put_reserved_item(item_id, source) + assert first_fingerprint != second_fingerprint - async with factory() as session: - item = await session.get(ArtifactUploadItem, item_id) - assert item is not None - upload_session = await session.get(ArtifactUploadSession, item.session_id) - assert item.state == "failed" - assert item.error_code == "artifact_input_mismatch" - assert upload_session is not None - assert upload_session.reserved_bytes == 0 - assert upload_session.reserved_items == 0 - assert store.put_calls == 1 - finally: - await engine.dispose() - -@pytest.mark.asyncio -async def test_changed_commitment_is_rejected_before_provider_io( - artifact_database_env: str, - tmp_path: Path, -) -> None: - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - store = _UnavailableStore() - try: - async with _minted_source(tmp_path / "scratch-first", b"first") as first: - async with factory() as session: - item_id = await _seed_reserved_item(session, first.commitment) - async with _minted_source(tmp_path / "scratch-second", b"second") as second: - async with factory() as session: - with pytest.raises(ArtifactIngestStateError, match="commitment changed"): - await ArtifactStorageOrchestrator( - session, - store, - artifact_storage_namespace_spec(_settings(tmp_path), store), - ).put_reserved_item(item_id, second) - assert store.put_calls == 0 - finally: - await engine.dispose() - - -@pytest.mark.asyncio -async def test_namespace_mismatch_fails_before_provider_io( - artifact_database_env: str, - tmp_path: Path, -) -> None: - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - store = _UnavailableStore() +def test_namespace_spec_rejects_adapter_identity_drift(tmp_path: Path) -> None: settings = _settings(tmp_path) - canonical = artifact_storage_namespace_spec(settings, store) - conflicting_descriptor = dict(canonical.namespace_descriptor) - conflicting_descriptor["private_root_identity"] = "sha256:" + "f" * 64 - conflicting = ArtifactStorageNamespaceSpec( - backend=canonical.backend, - adapter=canonical.adapter, - provider_profile=canonical.provider_profile, - namespace_descriptor=conflicting_descriptor, - namespace_fingerprint=canonical_json_hash(conflicting_descriptor), - ) - try: - async with factory() as session: - await ArtifactStorageOrchestrator(session, store, canonical).ensure_storage_namespace() - async with _minted_source(tmp_path / "scratch", b"blocked") as source: - async with factory() as session: - item_id = await _seed_reserved_item(session, source.commitment) - with pytest.raises(ArtifactStorageNamespaceError): - await ArtifactStorageOrchestrator( - session, store, conflicting - ).put_reserved_item(item_id, source) - assert store.put_calls == 0 - finally: - await engine.dispose() - -@pytest.mark.asyncio -async def test_concurrent_different_first_namespace_writers_have_one_winner( - artifact_database_env: str, - tmp_path: Path, -) -> None: - engine = create_async_engine(artifact_database_env) - factory = async_sessionmaker(engine, expire_on_commit=False) - first_store = _NamespaceObservingAcknowledgingStore(factory) - second_store = _NamespaceObservingAcknowledgingStore(factory) - first = artifact_storage_namespace_spec(_settings(tmp_path), first_store) - second_descriptor = dict(first.namespace_descriptor) - second_descriptor["private_root_identity"] = "sha256:" + "a" * 64 - second = ArtifactStorageNamespaceSpec( - backend=first.backend, - adapter=first.adapter, - provider_profile=first.provider_profile, - namespace_descriptor=second_descriptor, - namespace_fingerprint=canonical_json_hash(second_descriptor), - ) - - async def publish( - item_id: str, - source: CommittedArtifactSource, - store: _NamespaceObservingAcknowledgingStore, - spec: ArtifactStorageNamespaceSpec, - ) -> object: - async with factory() as session: - try: - return await ArtifactStorageOrchestrator(session, store, spec).put_reserved_item( - item_id, source - ) - except ArtifactStorageNamespaceError as exc: - return exc - - try: - async with _minted_source(tmp_path / "scratch-first", b"first") as first_source: - async with _minted_source(tmp_path / "scratch-second", b"second") as second_source: - async with factory() as session: - first_item_id = await _seed_reserved_item(session, first_source.commitment) - second_item_id = await _seed_reserved_item(session, second_source.commitment) - outcomes = await asyncio.gather( - publish(first_item_id, first_source, first_store, first), - publish(second_item_id, second_source, second_store, second), - ) - - assert sum(isinstance(value, ArtifactPutResult) for value in outcomes) == 1 - assert sum(isinstance(value, ArtifactStorageNamespaceError) for value in outcomes) == 1 - winning_index = next( - index for index, value in enumerate(outcomes) if isinstance(value, ArtifactPutResult) - ) - stores = (first_store, second_store) - specs = (first, second) - assert stores[winning_index].put_calls == 1 - assert ( - stores[winning_index].observed_namespace_fingerprint - == specs[winning_index].namespace_fingerprint + class _DriftedBootstrap: + identity = ExternalServiceAdapterIdentity( + "artifact_store", + "s3_compatible", ) - assert stores[1 - winning_index].put_calls == 0 - assert stores[1 - winning_index].observed_namespace_fingerprint is None - async with factory() as session: - namespace = await session.scalar(select(ArtifactStorageNamespace)) - assert namespace is not None - assert namespace.namespace_fingerprint == specs[winning_index].namespace_fingerprint - finally: - await engine.dispose() - - -def test_namespace_descriptor_is_canonical_and_does_not_store_local_path( - tmp_path: Path, -) -> None: - settings = _settings(tmp_path) - store = _UnavailableStore() - spec = artifact_storage_namespace_spec(settings, store) - serialized = repr(spec.namespace_descriptor) - - assert spec.backend == "local" - assert spec.provider_profile == "local-v2" - assert spec.namespace_fingerprint == canonical_json_hash(spec.namespace_descriptor) - assert str(settings.artifact_local_root) not in serialized - assert spec.namespace_descriptor["private_root_identity"].startswith("sha256:") - - -def test_namespace_descriptor_changes_when_local_root_is_replaced(tmp_path: Path) -> None: - settings = _settings(tmp_path) - first_store = LocalStorageBootstrap(LocalStorageAdapter(root=settings.artifact_local_root)) - original = artifact_storage_namespace_spec(settings, first_store) - first_store.close() - - root = settings.artifact_local_root - assert root is not None - root.rename(tmp_path / "replaced-store") - root.mkdir(mode=0o700) - - replacement_store = LocalStorageBootstrap(LocalStorageAdapter(root=root)) - replacement = artifact_storage_namespace_spec(settings, replacement_store) - replacement_store.close() - assert replacement.namespace_descriptor != original.namespace_descriptor - assert replacement.namespace_fingerprint != original.namespace_fingerprint - - -def test_store_bootstrap_requires_preprovisioned_private_root(tmp_path: Path) -> None: - with pytest.raises(ArtifactConfigurationError, match="storage is unavailable"): - LocalStorageAdapter(root=tmp_path / "missing-store") - - -def test_namespace_spec_rejects_adapter_drift(tmp_path: Path) -> None: - local_store = _UnavailableStore() - with pytest.raises(ArtifactStorageNamespaceError, match="identity"): - artifact_storage_namespace_spec(Settings(), local_store) - - -@pytest.mark.asyncio -@pytest.mark.parametrize( - "provider_failure", - [asyncio.CancelledError("cancelled"), RuntimeError("provider crashed")], -) -async def test_unexpected_or_cancelled_provider_failure_requires_replay( - tmp_path: Path, - provider_failure: BaseException, -) -> None: - store = _UnavailableStore() - store.put = AsyncMock(side_effect=provider_failure) # type: ignore[method-assign] - orchestrator = ArtifactStorageOrchestrator( - None, # type: ignore[arg-type] - store, - artifact_storage_namespace_spec(_settings(tmp_path), store), - ) - orchestrator._start_put = AsyncMock( # type: ignore[method-assign] - return_value=ProviderAttemptFence(1) - ) - orchestrator._mark_replay_required = AsyncMock() # type: ignore[method-assign] - - async with _minted_source(tmp_path / "scratch", b"provider") as source: - with pytest.raises(type(provider_failure)): - await orchestrator.put_reserved_item("item", source) - orchestrator._mark_replay_required.assert_awaited_once() # type: ignore[attr-defined] - - -@pytest.mark.asyncio -@pytest.mark.parametrize( - "finalization_failure", - [ - ArtifactInputMismatchError("mismatch"), - asyncio.CancelledError("cancelled"), - RuntimeError("database unavailable"), - ], -) -async def test_finalization_failure_never_leaves_uploading_unresolved( - tmp_path: Path, - finalization_failure: BaseException, -) -> None: - store = _UnavailableStore() - store.put = AsyncMock( # type: ignore[method-assign] - return_value=ArtifactPutResult("sha256/00/" + "0" * 62, replayed=False) - ) - orchestrator = ArtifactStorageOrchestrator( - None, # type: ignore[arg-type] - store, - artifact_storage_namespace_spec(_settings(tmp_path), store), - ) - orchestrator._start_put = AsyncMock( # type: ignore[method-assign] - return_value=ProviderAttemptFence(1) - ) - orchestrator._finalize_put = AsyncMock( # type: ignore[method-assign] - side_effect=finalization_failure - ) - orchestrator._fail_put = AsyncMock() # type: ignore[method-assign] - orchestrator._mark_replay_required = AsyncMock() # type: ignore[method-assign] - - async with _minted_source(tmp_path / "scratch", b"finalize") as source: - with pytest.raises(type(finalization_failure)): - await orchestrator.put_reserved_item("item", source) - if isinstance(finalization_failure, ArtifactInputMismatchError): - orchestrator._fail_put.assert_awaited_once() # type: ignore[attr-defined] - orchestrator._mark_replay_required.assert_not_awaited() # type: ignore[attr-defined] - else: - orchestrator._mark_replay_required.assert_awaited_once() # type: ignore[attr-defined] - orchestrator._fail_put.assert_not_awaited() # type: ignore[attr-defined] - - -@pytest.mark.asyncio -async def test_startup_namespace_validation_uses_canonical_session_factory( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - settings = _settings(tmp_path) - store = _UnavailableStore() - - class _TransactionContext: - async def __aenter__(self) -> None: - return None - - async def __aexit__(self, *_args: object) -> None: - return None - - class _Session: - def begin(self) -> _TransactionContext: - return _TransactionContext() - - session = _Session() - - class _SessionContext: - async def __aenter__(self) -> object: - return session - - async def __aexit__(self, *_args: object) -> None: - return None - - repository = object() - ensured = AsyncMock() - - def create_repository(candidate_session: object) -> object: - assert candidate_session is session - return repository - - monkeypatch.setattr( - artifact_service_module, - "get_session_factory", - lambda: lambda: _SessionContext(), - ) - monkeypatch.setattr(artifact_service_module, "ArtifactRepository", create_repository) - monkeypatch.setattr( - artifact_service_module, - "_claim_and_validate_storage_namespace", - ensured, - ) - claim = await validate_artifact_storage_namespace_at_startup(store, settings) - assert isinstance(ensured.await_args.args[1], ArtifactStorageNamespaceSpec) - assert ensured.await_args.args[0] is repository - assert claim.namespace_identity == store.namespace_identity + with pytest.raises( + ArtifactStorageNamespaceError, + match="does not match configuration", + ): + artifact_storage_namespace_spec(settings, _DriftedBootstrap()) # type: ignore[arg-type] -def test_models_remove_v1_provider_retention_and_receipt_fields() -> None: - assert not hasattr(ArtifactReplica, "retention_state") - assert not hasattr(ArtifactReplica, "provider_artifact_id") - assert not hasattr(ArtifactReplica, "provider_manifest_id") - assert not hasattr(ArtifactOperationReceipt, "provider_receipt_id") - assert not hasattr(ArtifactOperationReceipt, "retention_reference") - assert not hasattr(ArtifactOperationReceipt, "service_principal") - assert not hasattr(ArtifactUploadItem, "provider_operation_reference") - assert hasattr(ArtifactReplica, "provider_object_ref") - assert hasattr(ArtifactStorageNamespace, "namespace_fingerprint") +def test_models_retain_only_v2_provider_evidence_fields() -> None: + assert "provider_manifest_id" not in ArtifactReplica.__table__.columns + assert "retention_state" not in ArtifactReplica.__table__.columns + assert "provider_receipt_id" not in ArtifactOperationReceipt.__table__.columns + assert "retention_reference" not in ArtifactOperationReceipt.__table__.columns + assert "namespace_fingerprint" in ArtifactStorageNamespace.__table__.columns diff --git a/backend/tests/test_aws_credential_isolation.py b/backend/tests/test_aws_credential_isolation.py index 40449d6c8..89da07a32 100644 --- a/backend/tests/test_aws_credential_isolation.py +++ b/backend/tests/test_aws_credential_isolation.py @@ -13,12 +13,14 @@ from app.adapters.artifacts import s3_compatible from app.core.config import Settings +from tests.artifact_store_helpers import artifact_admission_limit_settings from app.interfaces.artifacts import ArtifactConfigurationError from tests.assertion_helpers import assert_secret_not_retained def _aws_settings(tmp_path: Path, method: str = "container-role") -> Settings: return Settings( + **artifact_admission_limit_settings(), environment="production", artifact_store_backend="s3_compatible", artifact_scratch_root=tmp_path / "scratch", diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py index 8aec41452..04e65ce97 100644 --- a/backend/tests/test_config.py +++ b/backend/tests/test_config.py @@ -34,6 +34,7 @@ ) from app.main import create_app from tests.assertion_helpers import assert_secret_not_retained +from tests.artifact_store_helpers import artifact_admission_limit_settings def test_default_settings_are_fail_closed(monkeypatch: pytest.MonkeyPatch) -> None: @@ -386,6 +387,7 @@ def _flow_settings(**overrides) -> Settings: def _minio_setting_values(tmp_path: Path, **overrides: object) -> dict[str, object]: values: dict[str, object] = { + **artifact_admission_limit_settings(), "environment": "test", "artifact_store_backend": "s3_compatible", "artifact_scratch_root": tmp_path / "scratch", @@ -409,6 +411,7 @@ def _minio_settings(tmp_path: Path, **overrides: object) -> Settings: def _aws_settings(tmp_path: Path, **overrides: object) -> Settings: values: dict[str, object] = { + **artifact_admission_limit_settings(), "environment": "production", "artifact_store_backend": "s3_compatible", "artifact_scratch_root": tmp_path / "scratch", @@ -529,6 +532,7 @@ def test_settings_reject_local_artifacts_outside_development(environment: str) - """Keep filesystem storage out of production-like deployments.""" with pytest.raises(ValidationError, match="local artifact storage"): Settings( + **artifact_admission_limit_settings(), environment=environment, artifact_store_backend="local", artifact_local_root="/tmp/workstream-artifacts", @@ -539,11 +543,38 @@ def test_settings_reject_local_artifacts_outside_development(environment: str) - def test_settings_require_scratch_root_for_enabled_artifacts() -> None: with pytest.raises(ValidationError, match="artifact scratch root"): Settings( + **artifact_admission_limit_settings(), environment="test", artifact_store_backend="s3_compatible", ) +def test_settings_require_every_durable_byte_admission_limit(tmp_path: Path) -> None: + """Enabled stores must never inherit an implicit capacity policy.""" + common = { + "environment": "test", + "artifact_store_backend": "local", + "artifact_local_root": tmp_path / "artifacts", + "artifact_scratch_root": tmp_path / "scratch", + } + with pytest.raises( + ValidationError, + match="requires all durable-byte admission limits", + ): + Settings(**common) + + partial = artifact_admission_limit_settings() + partial.pop("artifact_admission_task_maximum_bytes") + with pytest.raises( + ValidationError, + match="requires all durable-byte admission limits", + ): + Settings(**common, **partial) + + settings = Settings(**common, **artifact_admission_limit_settings()) + assert settings.artifact_admission_task_maximum_bytes == 1024 + + @pytest.mark.parametrize("interval", [0, -1, 86_401]) def test_artifact_scratch_cleanup_interval_is_positive_and_bounded(interval: int) -> None: with pytest.raises(ValidationError): @@ -565,6 +596,7 @@ def test_artifact_backend_enum_is_exact_and_flow_node_is_rejected(tmp_path: Path def test_local_artifact_settings_and_factory(tmp_path: Path) -> None: """Construct local storage only from complete development configuration.""" settings = Settings( + **artifact_admission_limit_settings(), environment="test", artifact_store_backend="local", artifact_local_root=tmp_path / "artifacts", @@ -992,6 +1024,7 @@ def test_minio_secret_values_from_env_and_dotenv_are_absent_from_errors( with pytest.raises(ValidationError) as env_error: Settings( + **artifact_admission_limit_settings(), environment="test", artifact_store_backend="s3_compatible", artifact_scratch_root=tmp_path / "scratch", @@ -1009,6 +1042,7 @@ def test_minio_secret_values_from_env_and_dotenv_are_absent_from_errors( monkeypatch.delenv("WORKSTREAM_ARTIFACT_S3_SECRET_ACCESS_KEY") with pytest.raises(ValidationError) as dotenv_error: Settings( + **artifact_admission_limit_settings(), environment="test", artifact_store_backend="s3_compatible", artifact_scratch_root=tmp_path / "scratch", @@ -1026,6 +1060,7 @@ def test_minio_secret_values_from_env_and_dotenv_are_absent_from_errors( def test_minio_endpoint_is_normalized_before_namespace_identity(tmp_path: Path) -> None: """Equivalent MinIO origins must produce one endpoint and namespace identity.""" common = { + **artifact_admission_limit_settings(), "environment": "test", "artifact_store_backend": "s3_compatible", "artifact_scratch_root": tmp_path / "scratch", diff --git a/backend/tests/test_local_artifact_store.py b/backend/tests/test_local_artifact_store.py index 72bacc12e..30852212b 100644 --- a/backend/tests/test_local_artifact_store.py +++ b/backend/tests/test_local_artifact_store.py @@ -14,10 +14,6 @@ import pytest from app.adapters.artifacts.local import LocalStorageAdapter, LocalStorageBootstrap -from app.adapters.artifacts.references import ( - artifact_provider_object_ref, - parse_artifact_provider_object_ref, -) from app.interfaces.artifacts import ( ArtifactByteRange, ArtifactConfigurationError, @@ -26,6 +22,8 @@ ArtifactOperationConflictError, ArtifactStoreUnavailableError, ArtifactStoreNamespaceClaim, + artifact_provider_object_ref, + parse_artifact_provider_object_ref, ) from app.modules.artifacts.sources import ArtifactCommitment, CommittedArtifactSource from tests.artifact_store_helpers import ( diff --git a/backend/tests/test_s3_artifact_store.py b/backend/tests/test_s3_artifact_store.py index 054f601c7..336ccb653 100644 --- a/backend/tests/test_s3_artifact_store.py +++ b/backend/tests/test_s3_artifact_store.py @@ -37,7 +37,10 @@ from app.modules.artifacts.preparation import ArtifactPreparationService from app.modules.artifacts.sources import ArtifactCommitment from tests.assertion_helpers import assert_secret_not_retained -from tests.artifact_store_helpers import minted_source +from tests.artifact_store_helpers import ( + artifact_admission_limit_settings, + minted_source, +) from tests.test_artifact_store_conformance import ArtifactStoreConformanceTests @@ -91,6 +94,7 @@ def minio_settings( ) -> Settings: """Return one complete local-only MinIO configuration.""" return Settings( + **artifact_admission_limit_settings(), environment="test", artifact_store_backend="s3_compatible", artifact_scratch_root=Path("/tmp/workstream-test-artifact-scratch"), diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 13f9d3c0a..9f73bad0f 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -724,10 +724,11 @@ in the v2 clean cut. No compatibility adapter or dual format remains. provider call. 3. Transaction A atomically claims or validates the deployment storage namespace, derives and reserves every applicable durable-storage scope - charge as `provisional`, reserves the item, and commits the server-computed - digest, size, media type, operation identity, canonical request digest, - limits, one `ArtifactPutAttempt`, and CAS. Any quota or namespace failure - rolls back all reservations. + charge as `provisional`, links the authoritative producer source reference, + and commits the server-computed digest, size, media type, operation identity, + canonical request digest, exact limits, one `ArtifactPutAttempt`, and CAS. + Any quota, relationship, or namespace failure rolls back the complete + transaction. 4. Workstream passes the sealed `CommittedArtifactSource` from the artifact orchestration service to the adapter outside the transaction. @@ -746,6 +747,12 @@ in the v2 clean cut. No compatibility adapter or dual format remains. No provider call occurs inside a PostgreSQL transaction. No product binding is created before independent verification. +Chunk 02C1 implements step 3 only. It creates a `prepared` attempt with no +executor, lease, receipt, replica, or execution generation and exposes no +product route. Steps 4 through 9 remain inactive until their separately +approved owning chunks. In particular, 02C1 contains no provider `put`, +`observe_put_result`, verification publication, recovery, or product cutover. + Acknowledgement loss leaves the durable put attempt and charges provisional. The attempt scanner publishes resolution after an ambiguous outcome or expired execution lease. Resolution performs a fresh read-only observation and full @@ -808,6 +815,20 @@ The preparation settings use the standard `WORKSTREAM_` environment prefix: | `WORKSTREAM_ARTIFACT_STREAM_BUFFER_BYTES` | `1048576` | Bounded streaming buffer, limited to at most 1 MiB. | | `WORKSTREAM_ARTIFACT_OPERATION_LOCK_TIMEOUT_SECONDS` | `1800` | Maximum wait for a private cross-process artifact-store operation lock before failing closed. | +Enabled artifact storage also requires an explicit durable-byte policy. None +of these limits has a default, and startup fails unless all four are positive: + +| Environment variable | Default | Contract | +|---|---:|---| +| `WORKSTREAM_ARTIFACT_ADMISSION_TASK_MAXIMUM_BYTES` | unset | Maximum cumulative unique provisional/completed bytes charged to one task. Guide bytes have no task scope. | +| `WORKSTREAM_ARTIFACT_ADMISSION_PRODUCER_MAXIMUM_BYTES` | unset | Maximum cumulative unique provisional/completed bytes charged to one canonical human actor or fixed service identity. | +| `WORKSTREAM_ARTIFACT_ADMISSION_PROJECT_MAXIMUM_BYTES` | unset | Maximum cumulative unique provisional/completed bytes charged to one project. | +| `WORKSTREAM_ARTIFACT_ADMISSION_DEPLOYMENT_MAXIMUM_BYTES` | unset | Maximum cumulative unique provisional/completed bytes charged to the deployment namespace. | + +The first committed scope row pins its configured limit. A later process whose +configuration disagrees fails admission instead of silently changing the +persisted capacity contract. + The S3-compatible provider settings use the same prefix: | Environment variable | Default | Contract | @@ -1131,6 +1152,11 @@ Implementation is a clean cut: Operator must reprovision an empty database/storage namespace out of band and reingest authoritative bytes through v2; records whose authoritative bytes are unavailable are not migrated. +- migration `0028_artifact_admission` installs the durable admission ledger and prepared + put-attempt tables. Its downgrade locks every owned table and refuses to + remove the foundation when any admission scope, charge, attempt, or + attempt-charge link exists; downgrade is permitted only while all four + tables are empty. Every migration proves fresh upgrade, prior-head upgrade, populated-state preservation or explicit refusal, empty downgrade/re-upgrade, and no artifact diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index 60eb26719..c631c5744 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -4154,7 +4154,7 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: ).read_text(encoding="utf-8") artifact_contract = Path( ".agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/" - "WS-ART-001-02B1-s3-compatible-minio-aws.md" + "WS-ART-001-02C1-admission-put-attempt-foundation.md" ).read_text(encoding="utf-8") work_queue = Path(".agent-loop/WORK_QUEUE.md").read_text(encoding="utf-8") loop_state = Path(".agent-loop/LOOP_STATE.md").read_text(encoding="utf-8") @@ -4223,14 +4223,15 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: ) assert "Merged through PR #129 as `9a04434`" in artifact_map assert "Merged through PR #141 as `a10d901`" in artifact_map - assert "Active after 02A3 merged through PR #141" in artifact_map - assert "Status: Active after explicit human start" in artifact_contract + assert "Merged through PR #151 as `1b5422fc`" in artifact_map + assert "Active after PR #151 and explicit user start" in artifact_map + assert "Status: Active after explicit user start" in artifact_contract assert ( "AUTH's owner reconciliation merged through PR #140 as\n" "`d541521`" in artifact_status ) - assert "`WS-ART-001-02B1` is active" in artifact_status - assert "The current gate is deterministic 02B1 proof followed by all nine" in ( + assert "`WS-ART-001-02C1` is active" in artifact_status + assert "The current gate is deterministic 02C1 proof followed by all nine" in ( artifact_status.replace("\n", " ") ) assert "No later artifact chunk starts automatically" in artifact_status.replace( @@ -4240,22 +4241,12 @@ def test_parallel_initiative_status_matches_trusted_main() -> None: "| `WS-AUTH-001-09C` | Actor And Identity-Link Administration Reads | L1 | " "Merged through PR #146 as `0ffdabf`" in work_queue ) - assert ( - "| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | " - "Merged through PR #151 as `1b5422f` on 2026-07-19" in work_queue - ) - assert ( - "PR #151 then merged `WS-ART-001-02B1` as `1b5422f` on 2026-07-19" - in loop_state - ) - assert ( - "ART-02C1 remains inactive pending signed memory and a separate explicit start" - in loop_state.replace("\n", " ") - ) - assert "Current ART gate: integrate trusted `main`" not in loop_state - assert "| `WS-ART-001-02B1` | S3-Compatible MinIO And AWS | L1 | Active" not in ( + assert "| `WS-ART-001-02C1` | Admission And Put-Attempt Foundation | L1 | Active" in ( work_queue ) + assert "Current ART gate: integrate trusted `main`, complete deterministic 02C1" in ( + loop_state + ) def test_stale_authorization_discovery_includes_new_untracked_docs() -> None: