From 14ee2136c42c29ee848946f99fcba7d87760ba25 Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 13:43:24 +0100 Subject: [PATCH 01/12] docs(plan): [PILOT-15] bound Markdown guide originals --- .../pilot15-markdown-guide-documents.md | 123 ++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 .commitrail/changes/pilot15-markdown-guide-documents.md diff --git a/.commitrail/changes/pilot15-markdown-guide-documents.md b/.commitrail/changes/pilot15-markdown-guide-documents.md new file mode 100644 index 000000000..0ff6ad773 --- /dev/null +++ b/.commitrail/changes/pilot15-markdown-guide-documents.md @@ -0,0 +1,123 @@ +# [PILOT-15] Admit Markdown Guide Originals + +- Initiative: None +- Durable disposition: Planned +- Intended merge outcome: A Project Manager can declare and upload a + `text/markdown` guide original that retains byte-exact ART custody and is + available through the existing setup-agent and assigned-contributor reads. + +## Intent + +Resolve F-019 / [issue #508](https://github.com/Flow-Research/workstream/issues/508). +Project Managers should not have to export a Markdown guide to PDF before using +the existing governed guide flow. Markdown remains an original guide document, +not executable input or a separate storage path. + +## Current behavior + +`GuideDocumentMediaType` and `DOCUMENT_EXTENSIONS` in +`backend/app/modules/projects/api/guide_documents.py` admit PDF, DOCX and PPTX. +The public guide declaration, ART upload format inspection, setup-agent manifest, +review-packet metadata and Go declaration client consume that closed type. The +database constraint on `review_packet_guide_items.media_type` independently +repeats the same three values. PILOT-13 PR #505 adds task-locked contributor +original reads and CLI downloads, but its open branch is not delivered behavior. + +## Bounded change + +### Allowed + +- The existing PROJECTS `GuideDocumentMediaType` and extension mapping, public + guide declaration schemas/OpenAPI, and their direct contract tests. +- ART's existing bounded guide-format inspector and upload admission path for + byte-preserving UTF-8 Markdown classification; existing digest, size, + idempotency, storage and namespace custody remain unchanged. +- The existing setup-agent manifest/workspace path and focused tests proving a + `.md` original is opened under the same run-scoped grant, disabled-network + container and exact resource cleanup. +- Review-packet typed metadata/model constraint plus one forward-only migration + after the actual merged migration head, its graph/schema consumers and direct + PostgreSQL refusal/acceptance proof. +- The Go guide-declaration client and current CLI/API documentation for + `text/markdown`. After PR #505 merges, its task guide DTO/extension/download + consumers and tests may be reconciled without copying its implementation. +- Existing guide API, real PostgreSQL/MinIO intake fixtures, format, setup + workspace, review-packet, CLI declaration and later PILOT-13 contributor-read + tests; current README, artifact/guide specifications, roadmap and this record. + +### Not allowed + +- No HTML, docs-site, URL-fetch or generic text formats; no Markdown rendering, + conversion, sanitization or separate storage/per-task copy. +- No changes to PDF, DOCX or PPTX behavior, artifact byte limits, digests, + namespaces, authority, locks, task/guide lifecycle, setup proposal shape or + isolated-workspace policy. +- No checker catalogue/registration changes, activation refusal or recovery + behavior; F-020 belongs to PILOT-04 / issue #491. +- No duplicate contributor read, task authority, streaming or CLI download + implementation while PR #505 remains open. No compatibility variant, data + rewrite, downgrade, deployment, merge or issue closure. + +## Design and decisions + +Extend the one owner type with `text/markdown` and `.md`. ART classifies the +declared type as bounded UTF-8 text without interpreting Markdown and preserves +the original bytes for hashing, storage and reads. Known binary signatures and +malformed UTF-8 remain invalid rather than being relabelled as Markdown. Existing +typed consumers inherit the new member; the review-packet database check receives +the matching additive value in the next linear migration. The setup agent stages +the verified original through its current opaque handle and network-disabled +workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. + +## Acceptance criteria + +- [ ] Public API and Go CLI declaration accept exactly `text/markdown`; HTML, + `text/plain`, binary/malformed UTF-8 and media-type substitution remain denied. +- [ ] Real PostgreSQL/MinIO intake stores and reads back the exact Markdown + bytes with the server-computed SHA-256, byte count and declared media type. +- [ ] The existing setup-agent workspace opens the exact `.md` version through + its run-scoped grant and retains disabled network, bounded resources and cleanup. +- [ ] Review-packet metadata and PostgreSQL accept the exact Markdown member + while direct writes of unsupported media types still fail. +- [ ] After PILOT-13 merges, an assigned contributor lists and downloads the + task-locked Markdown original byte-identically through REST/CLI; foreign, + unassigned, corrupt and unsafe-destination cases retain their existing denial. +- [ ] PDF, DOCX and PPTX declaration/upload/setup/review/contributor behavior + remains covered and unchanged; no second migration head is introduced. +- [ ] Current API/CLI, artifact/guide specifications and roadmap state the + delivered boundary without claiming HTML, remote sites or deployed capability. + +## Risk and review routing + +- Risk class: L1 +- Required reviewers: architecture, reuse, security, qa, test_delta, + documentation, product_ops +- Human review focus: Strict media classification without byte rewriting; + unchanged ART/authority/isolation boundaries; one linear migration after + PILOT-13; no duplicate contributor-read implementation; honest format limits. + +## Evidence + +| Claim | Command or proof | Result | Remaining uncertainty | +|---|---|---|---| +| Bounded Markdown classification | Focused `test_guide_formats.py` and upload mismatch controls | Required before completion | Markdown semantics are not parsed or rendered | +| Durable original custody | Isolated real PostgreSQL/MinIO guide-intake journey with provider readback | Required before completion | Hosted S3 remains shared ART-provider evidence | +| Setup-agent exact read | Existing workspace/runtime tests with Markdown version and byte-bearing grant | Required before completion | No live model inference is required for byte-access proof | +| Persisted review metadata | Real PostgreSQL migration/direct-write and review-packet storage proof | Required after PR #505 establishes the predecessor | Existing rows are retained; no downgrade is supported | +| Contributor REST/CLI | PILOT-13 focused public API and built-binary tests extended after merge | Blocked on merged PR #505 | Open-branch code is inspection input, not delivered proof | + +## Review findings + +No review finding has produced a durable source change yet. + +## Reconciliation + +- Current-source reconciliation: Based on main + `3fa0dfb9eeda18888cdced494dfe813bd8dacf63`, whose migration head is + `0025_submission_dispatch`. PR #505 currently proposes the next guide-read + migration and contributor read/CLI owners; this branch will not allocate its + revision or duplicate those owners before it merges. +- Next usable boundary: Merge and verify PILOT-13, then reconcile its exact task + guide media type and CLI extension consumers and allocate the next migration. +- Remaining risks: Live provider/model inference is outside this byte-custody + change. Hosted deployment and HTML/docs-site ingestion remain unsupported. From db42a24d80b0741839a4c65d40286b3f649fdf48 Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 13:55:59 +0100 Subject: [PATCH 02/12] feat: [PILOT-15] admit Markdown guide originals --- README.md | 6 ++- .../app/modules/artifacts/guide_formats.py | 19 ++++++++ .../modules/projects/api/guide_documents.py | 2 + backend/app/modules/reviews/packet/models.py | 2 +- .../artifacts/test_review_packet_contract.py | 1 + .../tests/test_guide_document_api_contract.py | 11 ++++- backend/tests/test_guide_document_intake.py | 16 +++++-- backend/tests/test_guide_formats.py | 29 +++++++++++- backend/tests/test_guide_runtime_workspace.py | 46 ++++++++++++++++++- cli/README.md | 7 +-- cli/internal/api/guide_create.go | 2 +- .../integration/test_guide_create_http.py | 15 ++++++ docs/architecture_data_model.md | 11 +++-- ...ion_0003_project_guides_are_first_class.md | 2 +- ...ssion_artifact_policy_drives_pre_submit.md | 7 +-- docs/glossary.md | 10 ++-- docs/operations_backend_testing.md | 7 +-- docs/operations_project_operating_manual.md | 2 +- docs/product_first_user_flows.md | 2 +- docs/roadmap_status.md | 5 +- docs/spec_artifact_storage_service.md | 15 +++--- docs/spec_authorization_service.md | 5 +- docs/template_submission_artifact_policy.md | 2 +- 23 files changed, 180 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 7385141ce..0e1b1a865 100644 --- a/README.md +++ b/README.md @@ -173,8 +173,10 @@ storage, detached source facts and accepted-effects contracts. B6 commits each new hidden Submission with its initial checker reservation and outbox request; automatic request delivery, routing and acceptance remain unavailable. Project-guide ingestion stores original documents, records immutable metadata and provides authorized exact-file reads to the -unified setup agent. Guide metadata in PostgreSQL also holds at least one required -task example; the agent assesses the examples with the uploaded guide documents. +unified setup agent. Accepted originals are PDF, DOCX, PPTX and byte-preserved +UTF-8 Markdown (`.md`); HTML and documentation sites are not guide inputs. Guide +metadata in PostgreSQL also holds at least one required task example; the agent +assesses the examples with the uploaded guide documents. Findings and policy proposals retain document-access evidence. [ART-07A1](.commitrail/initiatives/WS-ART-001/WS-ART-001-07A1.md) provides strict diff --git a/backend/app/modules/artifacts/guide_formats.py b/backend/app/modules/artifacts/guide_formats.py index bd579b1ce..2c8d130c7 100644 --- a/backend/app/modules/artifacts/guide_formats.py +++ b/backend/app/modules/artifacts/guide_formats.py @@ -2,6 +2,7 @@ from __future__ import annotations +import codecs from dataclasses import dataclass from io import BytesIO from pathlib import PurePosixPath @@ -104,8 +105,26 @@ def detect( return self._classified("pdf") if header.startswith(b"PK\x03\x04") or header.startswith(b"PK\x05\x06"): return self._inspect_zip(reader) + if declared_media_type == "text/markdown": + return self._inspect_markdown(reader) return GuideFormatResult("opaque", "unsupported", {}) + @staticmethod + def _inspect_markdown(reader: BinaryIO) -> GuideFormatResult: + """Validate bounded text syntax without parsing or rewriting the original.""" + decoder = codecs.getincrementaldecoder("utf-8")(errors="strict") + try: + while chunk := reader.read(_SAMPLE_BYTES): + if "\x00" in decoder.decode(chunk): + return GuideFormatResult("markdown", "malformed", {}) + if "\x00" in decoder.decode(b"", final=True): + return GuideFormatResult("markdown", "malformed", {}) + except UnicodeDecodeError: + return GuideFormatResult("markdown", "malformed", {}) + finally: + reader.seek(0) + return GuideFormatResult("md", "classified", {}) + def _inspect_zip(self, reader: BinaryIO) -> GuideFormatResult: state = {"entries": 0, "decompressed": 0, "compressed": 0, "depth": 0} try: diff --git a/backend/app/modules/projects/api/guide_documents.py b/backend/app/modules/projects/api/guide_documents.py index f3e06188b..7580cfa85 100644 --- a/backend/app/modules/projects/api/guide_documents.py +++ b/backend/app/modules/projects/api/guide_documents.py @@ -17,11 +17,13 @@ "application/pdf", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "application/vnd.openxmlformats-officedocument.presentationml.presentation", + "text/markdown", ] DOCUMENT_EXTENSIONS: dict[str, str] = { "application/pdf": "pdf", "application/vnd.openxmlformats-officedocument.wordprocessingml.document": "docx", "application/vnd.openxmlformats-officedocument.presentationml.presentation": "pptx", + "text/markdown": "md", } diff --git a/backend/app/modules/reviews/packet/models.py b/backend/app/modules/reviews/packet/models.py index 07b654c13..ff25955a2 100644 --- a/backend/app/modules/reviews/packet/models.py +++ b/backend/app/modules/reviews/packet/models.py @@ -133,7 +133,7 @@ class ReviewPacketGuideItem(Base): CheckConstraint("item_order >= 0", name="order_nonnegative"), CheckConstraint("logical_role='guide_source_original'", name="original_guide"), CheckConstraint( - "media_type in ('application/pdf','application/vnd.openxmlformats-officedocument.wordprocessingml.document','application/vnd.openxmlformats-officedocument.presentationml.presentation')", + "media_type in ('application/pdf','application/vnd.openxmlformats-officedocument.wordprocessingml.document','application/vnd.openxmlformats-officedocument.presentationml.presentation','text/markdown')", name="guide_media", ), ) diff --git a/backend/tests/artifacts/test_review_packet_contract.py b/backend/tests/artifacts/test_review_packet_contract.py index 66a4a6431..877addc8f 100644 --- a/backend/tests/artifacts/test_review_packet_contract.py +++ b/backend/tests/artifacts/test_review_packet_contract.py @@ -147,6 +147,7 @@ def test_closed_roles_and_supported_media() -> None: "application/pdf", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "application/vnd.openxmlformats-officedocument.presentationml.presentation", + "text/markdown", ): assert ReviewGuideMember.model_validate({**guide(0), "media_type": media}).media_type == media diff --git a/backend/tests/test_guide_document_api_contract.py b/backend/tests/test_guide_document_api_contract.py index a6550cacd..5787298d9 100644 --- a/backend/tests/test_guide_document_api_contract.py +++ b/backend/tests/test_guide_document_api_contract.py @@ -26,7 +26,7 @@ def test_current_guide_response_excludes_retained_body(): @pytest.mark.parametrize("patch", [ {"source_kind": "url_doc"}, {"source_kind": "rubric"}, - {"ingestion_adapter": "manual_import"}, {"media_type": "text/markdown"}, + {"ingestion_adapter": "manual_import"}, {"media_type": "text/plain"}, {"media_type": "image/png"}, {"media_type": "audio/wav"}, {"media_type": "application/vnd.ms-powerpoint"}, ]) @@ -37,6 +37,14 @@ def test_source_metadata_rejects_superseded_or_unsupported_ingress(patch): }) +def test_source_metadata_accepts_markdown_as_an_original_document(): + document = ProjectGuideDocumentInput.model_validate( + {"label": "guide.md", "media_type": "text/markdown"} + ) + + assert document.media_type == "text/markdown" + + @pytest.mark.parametrize("method,suffix", [ ("GET", "post-submit-checker-policy/setup"), ("POST", "post-submit-checker-policy/approve"), @@ -147,6 +155,7 @@ def test_document_upload_openapi_binary_body_and_bounded_responses(): "application/pdf", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "application/vnd.openxmlformats-officedocument.presentationml.presentation", + "text/markdown", } assert all(value["schema"] == {"type": "string", "format": "binary"} for value in body["content"].values()) responses = operation["responses"] diff --git a/backend/tests/test_guide_document_intake.py b/backend/tests/test_guide_document_intake.py index 2610a9241..cce9674cf 100644 --- a/backend/tests/test_guide_document_intake.py +++ b/backend/tests/test_guide_document_intake.py @@ -257,17 +257,24 @@ def publish(*, args, task_id): "version": "initial", "task_examples": [{"content": "Review a claim."}], "documents": [ - {"label": name, "media_type": "application/pdf"} - for name in ("guide.pdf", "appendix.pdf") + {"label": "guide.md", "media_type": "text/markdown"}, + {"label": "appendix.pdf", "media_type": "application/pdf"}, ], }, ) assert created.status_code == 201, created.text guide = created.json() - originals = [b"%PDF-1.7\nGuide fixture\n%%EOF", b"%PDF-1.7\nAppendix fixture\n%%EOF"] + originals = [ + "# Guide fixture\n\nFollow the locked instructions exactly. 雪\n".encode(), + b"%PDF-1.7\nAppendix fixture\n%%EOF", + ] for index, (document, original) in enumerate(zip(guide["documents"], originals, strict=True)): path = f"/api/v1/projects/{project['id']}/guides/{guide['id']}/documents/{document['document_id']}/content" - headers = auth_headers() | {"Content-Type": ("Application/PDF", "application/pdf; name=appendix.pdf")[index]} + headers = auth_headers() | { + "Content-Type": ("Text/Markdown; charset=utf-8", "application/pdf; name=appendix.pdf")[ + index + ] + } real_callback = internal_workers.continue_guide_setup_after_stored_document if index == 1 and recover_callback: @@ -326,6 +333,7 @@ async def recover(attempt_id): ).all() replicas = (await session.scalars(select(ArtifactReplica))).all() assert len(attempts) == len(replicas) == 2 + assert {row.media_type for row in attempts} == {"text/markdown", "application/pdf"} bootstrap, store = _open_store(get_settings()) try: stored = [ diff --git a/backend/tests/test_guide_formats.py b/backend/tests/test_guide_formats.py index 13d917a23..82f85c4df 100644 --- a/backend/tests/test_guide_formats.py +++ b/backend/tests/test_guide_formats.py @@ -236,7 +236,6 @@ def test_fixed_v01_limits_accept_exact_values_and_reject_one_over() -> None: @pytest.mark.parametrize("payload,media_type,adapter", [ (b'{"answer":42}', "application/json", "json"), (b"a,b\n1,2\n", "text/csv", "csv"), - (b"# Guide", "text/markdown", "manual_import"), (b"Guide", "text/plain", "upload"), (b"\x89PNG\r\n\x1a\n", "image/png", "upload"), (b"ID3\x00", "audio/mpeg", "upload"), @@ -247,6 +246,34 @@ def test_unsupported_formats_and_removed_adapters_never_activate_text_fallback(d assert (result.status, result.detected_format, result.facts) == ("unsupported", "opaque", {}) +def test_markdown_classification_streams_valid_utf8_without_rewriting(detector): + payload = b"a" * (64 * 1024 - 1) + "é\n# Guide\n".encode() + source = BytesIO(payload) + + result = detector.detect(source, declared_media_type="text/markdown", ingestion_adapter="upload") + + assert (result.status, result.detected_format, result.facts) == ("classified", "md", {}) + assert source.read() == payload + + +@pytest.mark.parametrize("payload", [b"guide\xff", b"guide\x00body"]) +def test_markdown_rejects_non_utf8_and_nul_bytes(detector, payload): + result = detector.detect(BytesIO(payload), declared_media_type="text/markdown") + + assert (result.status, result.detected_format) == ("malformed", "markdown") + + +@pytest.mark.parametrize( + ("payload", "detected"), + [(b"%PDF-1.7\n", "pdf"), (_zip({"document.txt": b"guide"}).getvalue(), "zip")], +) +def test_markdown_declaration_does_not_relabel_known_binary_formats(detector, payload, detected): + result = detector.detect(BytesIO(payload), declared_media_type="text/markdown") + + assert result.detected_format == detected + assert result.detected_format != "md" + + def test_pdf_signature_is_classified_without_extracting_text(detector): result = detector.detect(BytesIO(b"%PDF-1.7\n"), declared_media_type="application/pdf") assert (result.status, result.detected_format) == ("classified", "pdf") diff --git a/backend/tests/test_guide_runtime_workspace.py b/backend/tests/test_guide_runtime_workspace.py index 54c6e854e..a34bce35b 100644 --- a/backend/tests/test_guide_runtime_workspace.py +++ b/backend/tests/test_guide_runtime_workspace.py @@ -7,6 +7,7 @@ from unittest.mock import AsyncMock from uuid import uuid4 import asyncio +import hashlib import httpx2 from openai import NotFoundError @@ -61,8 +62,9 @@ async def record_cleanup_failed(self, identifier): class Grant: - def __init__(self, manifest): + def __init__(self, manifest, source_bytes=SOURCE_BYTES): self.manifest = manifest + self.source_bytes = source_bytes self.closed = False self.denied = False self.calls = 0 @@ -73,7 +75,7 @@ async def open(self, handle): if self.closed or self.denied: raise RuntimeError("grant unavailable") document = next(item for item in self.manifest.documents if self.manifest.handle_for(item) == handle) - with BytesIO(SOURCE_BYTES) as reader: + with BytesIO(self.source_bytes) as reader: yield OpenGuideDocument(document, reader) async def close(self): @@ -119,6 +121,46 @@ async def test_duplicate_opens_revalidate_grant_but_stage_once(workspace): assert grant.closed and len(custody.deleted) == 3 +@pytest.mark.asyncio +async def test_markdown_original_is_staged_byte_exact_in_isolated_workspace(workspace): + instance, client, grant, custody = workspace + markdown = "# Exact guide\n\nRead the retained original. 雪\n".encode() + current = instance.manifest.documents[0] + document = current.model_copy( + update={ + "media_type": "text/markdown", + "sha256": "sha256:" + hashlib.sha256(markdown).hexdigest(), + "byte_count": len(markdown), + } + ) + manifest = instance.manifest.model_copy(update={"documents": (document,)}) + instance.manifest = manifest + grant.manifest = manifest + grant.source_bytes = markdown + staged = [] + + async def stage(*, file, **_kwargs): + filename, reader, media_type = file + staged.append((filename, reader.read(), media_type)) + return SimpleNamespace( + id="file-owned", + created_at=1_800_000_000, + expires_at=1_800_000_000 + instance.configuration.file_expiry_seconds, + ) + + client.files.create.side_effect = stage + await instance.start() + handle = manifest.handle_for(document) + opened = await instance.open_document(handle) + + assert staged == [(f"{document.source_item_id}.md", markdown, "text/markdown")] + assert opened["sha256"] == document.sha256 + assert client.containers.create.call_args.kwargs["network_policy"] == {"type": "disabled"} + assert custody.opened == {handle} + await instance.close() + assert grant.closed and custody.deleted == set(custody.known) + + @pytest.mark.asyncio @pytest.mark.parametrize("handle", ["foreign", "s3://bucket/private.pdf", "../../guide.pdf"]) async def test_unknown_handle_has_no_source_or_provider_access(workspace, handle): diff --git a/cli/README.md b/cli/README.md index 2537d74e4..bed2aff10 100644 --- a/cli/README.md +++ b/cli/README.md @@ -119,7 +119,7 @@ guide document's contents or a storage URL. For example: {"content": "Evaluate the supplied experiment's evidence.", "title": "Evidence evaluation", "labels": ["research"]} ], "documents": [ - {"label": "Guide.pdf", "media_type": "application/pdf"} + {"label": "Guide.md", "media_type": "text/markdown"} ] } ``` @@ -128,8 +128,9 @@ guide document's contents or a storage URL. For example: example `title` may be omitted or null; example `labels` defaults to an empty array. Unknown/duplicate members, malformed JSON and null required fields or array members are rejected. The file is sent unchanged; the API owns semantic -limits and validation. PDF, DOCX and PPTX declarations use the media types in -OpenAPI. The API normalizes document-label whitespace; example text is preserved. +limits and validation. PDF, DOCX, PPTX and UTF-8 Markdown (`.md`) declarations +use the media types in OpenAPI. The API normalizes document-label whitespace; +example text is preserved. The CLI bounds this input to 1MiB and this response to 2MiB because declarations include example text and document selectors. These are client wire envelopes, diff --git a/cli/internal/api/guide_create.go b/cli/internal/api/guide_create.go index b77f8a727..68bdabdd3 100644 --- a/cli/internal/api/guide_create.go +++ b/cli/internal/api/guide_create.go @@ -193,7 +193,7 @@ func normalizedGuideLabel(label string) string { func guideMediaType(value string) bool { return value == "application/pdf" || value == "application/vnd.openxmlformats-officedocument.wordprocessingml.document" || - value == "application/vnd.openxmlformats-officedocument.presentationml.presentation" + value == "application/vnd.openxmlformats-officedocument.presentationml.presentation" || value == "text/markdown" } func guideCreateFailure(err error) error { diff --git a/cli/tests/integration/test_guide_create_http.py b/cli/tests/integration/test_guide_create_http.py index 7ccde066f..e0fde0657 100644 --- a/cli/tests/integration/test_guide_create_http.py +++ b/cli/tests/integration/test_guide_create_http.py @@ -126,6 +126,21 @@ def test_guide_create_preserves_body_selectors_defaults_and_safe_text(cli, tmp_p assert len(requests) == 36 +def test_guide_create_accepts_markdown_document_declaration(cli, tmp_path): + body = deepcopy(DECLARATION) + body["documents"] = [{"label": "Guide.md", "media_type": "text/markdown"}] + path = write_input(tmp_path, body) + with http_fixture() as (origin, response, requests): + response.update(status=201, body=json.dumps(receipt(body)).encode()) + + result = invoke(cli, origin, path) + + assert result.returncode == 0 and result.stderr == "" + assert json.loads(result.stdout)["documents"] == receipt(body)["documents"] + assert response["commands"] == [("application/json", [KEY], body)] + assert len(requests) == 1 + + def test_guide_create_input_bounds_and_large_valid_response(cli, tmp_path): body = deepcopy(DECLARATION) body["task_examples"] = [{"content": "é" * 32768}] diff --git a/docs/architecture_data_model.md b/docs/architecture_data_model.md index afd0710cd..dccffd74c 100644 --- a/docs/architecture_data_model.md +++ b/docs/architecture_data_model.md @@ -208,9 +208,10 @@ Status: ## ProjectGuide -Current guide content is the versioned PDF/DOCX/PPTX original-document manifest -and private ArtifactStore objects. Guide metadata writes do not accept inline -Markdown; PostgreSQL stores no newly extracted document bodies. +Current guide content is the versioned PDF/DOCX/PPTX or UTF-8 Markdown (`.md`) +original-document manifest and private ArtifactStore objects. Guide metadata +writes do not accept embedded document bodies; PostgreSQL stores no newly +extracted document bodies. Fields: @@ -246,8 +247,8 @@ Fields: - `selected_revision_policy_generation` - `selected_revision_policy_hash` -The guide is versioned and human-facing. Uploaded PDF/DOCX/PPTX originals live -in private ArtifactStore/S3 objects; PostgreSQL stores their metadata and the +The guide is versioned and human-facing. Uploaded PDF/DOCX/PPTX or UTF-8 Markdown +(`.md`) originals live in private ArtifactStore/S3 objects; PostgreSQL stores their metadata and the ordinary task-example text. The examples are representative inference inputs, not selected assignments, and do not create Workstream Tasks. One project-level policy proposal covers the project task set. diff --git a/docs/decision_0003_project_guides_are_first_class.md b/docs/decision_0003_project_guides_are_first_class.md index 0dbb9637e..10a2280b8 100644 --- a/docs/decision_0003_project_guides_are_first_class.md +++ b/docs/decision_0003_project_guides_are_first_class.md @@ -14,7 +14,7 @@ If project rules live only in chat or loose markdown, operators and reviewers wi Every active Workstream project must have a versioned project guide attached to its configuration. -The project guide is the human-facing project source of truth. The current v0.1 input is uploaded PDF/DOCX/PPTX documents in ArtifactStore/S3, with guide metadata and at least one ordinary-text task example in PostgreSQL. Contributors and reviewers use it to understand the project purpose, quality bar, instructions, examples, reviewer rubric, and common rejection reasons. +The project guide is the human-facing project source of truth. The current v0.1 input is uploaded PDF/DOCX/PPTX or UTF-8 Markdown (`.md`) documents in ArtifactStore/S3, with guide metadata and at least one ordinary-text task example in PostgreSQL. Contributors and reviewers use it to understand the project purpose, quality bar, instructions, examples, reviewer rubric, and common rejection reasons. Runtime enforcement uses approved machine-readable policies attached to the guide version. Workstream must not reread guide prose at submission time and guess what to enforce. diff --git a/docs/decision_0011_submission_artifact_policy_drives_pre_submit.md b/docs/decision_0011_submission_artifact_policy_drives_pre_submit.md index e1bd5d034..cfd1ef354 100644 --- a/docs/decision_0011_submission_artifact_policy_drives_pre_submit.md +++ b/docs/decision_0011_submission_artifact_policy_drives_pre_submit.md @@ -8,9 +8,10 @@ Accepted Project guides are human-facing. They explain the project, task expectations, examples, reviewer rubric, and quality bar. Current v0.1 guide files are uploaded -PDF/DOCX/PPTX originals stored through ArtifactStore/S3. PostgreSQL stores guide -metadata and a required ordered list of ordinary-text task examples. Markdown -bodies and URL/repository ingestion are not current input paths. +PDF/DOCX/PPTX or UTF-8 Markdown (`.md`) originals stored through ArtifactStore/S3. +PostgreSQL stores guide metadata and a required ordered list of ordinary-text +task examples. Embedded document bodies and URL/repository ingestion are not +current input paths. Submission intake needs a deterministic machine contract. If artifact requirements live only as guide prose, each project can drift into a different interpretation of what a valid submission packet must contain. diff --git a/docs/glossary.md b/docs/glossary.md index f24318a55..dc998b18d 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -28,8 +28,9 @@ published contribution policy, and queue. The external or internal organization that provides open-ended project material and business terms, including rubrics, task instructions and compensation -expectations. Current guide inputs are uploaded PDF/DOCX/PPTX originals, plus -at least one ordinary-text task example stored with guide metadata in PostgreSQL. +expectations. Current guide inputs are uploaded PDF/DOCX/PPTX or UTF-8 Markdown +(`.md`) originals, plus at least one ordinary-text task example stored with guide +metadata in PostgreSQL. The project owner does not author or approve Workstream's machine-readable internal policy schema. @@ -178,8 +179,9 @@ A future external task source that can submit tasks into Workstream through an a ## Project Guide -The human-facing operating guide for a project. Uploaded PDF, DOCX or supported -PPTX files contain project instructions, quality requirements and review rules. +The human-facing operating guide for a project. Uploaded PDF, DOCX, PPTX or +UTF-8 Markdown (`.md`) files contain project instructions, quality requirements +and review rules. ArtifactStore/S3 holds those originals; PostgreSQL holds their metadata and the guide version's required task-example list. At least one nonblank example is required. An example may be a starting idea or fuller description and need not diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index ab85af6f4..e2afe7515 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -2,9 +2,10 @@ ## Guide document runtime -Guide uploads preserve original files in ArtifactStore. Bounded PDF/OOXML -format admission runs locally; the agent reads documents in its isolated -provider workspace. There is no local guide extractor or Pillow/PDF +Guide uploads preserve original files in ArtifactStore. Bounded PDF/OOXML and +UTF-8 Markdown format admission runs locally; the agent reads documents in its +isolated provider workspace. Markdown is not parsed, rendered or rewritten. +There is no local guide extractor or Pillow/PDF parser dependency. Use the [Developer Quickstart](../README.md#developer-quickstart) for the supported backend environment. Scripted runtime tests prove contracts; real-provider document reading requires the explicit live probe and credentials diff --git a/docs/operations_project_operating_manual.md b/docs/operations_project_operating_manual.md index 9887c8c8b..b1e1011f5 100644 --- a/docs/operations_project_operating_manual.md +++ b/docs/operations_project_operating_manual.md @@ -29,7 +29,7 @@ remaining setup/activation work: - project name and slug exist - project guide metadata created with at least one ordinary-text task example and - the complete PDF/DOCX/PPTX document list + the complete PDF/DOCX/PPTX or UTF-8 Markdown (`.md`) document list - every declared original uploaded to ArtifactStore/S3 using its returned document ID - automatic setup starts after all declared documents have committed bytes - project owner setup material captured diff --git a/docs/product_first_user_flows.md b/docs/product_first_user_flows.md index 69a626a0a..7ce996a6d 100644 --- a/docs/product_first_user_flows.md +++ b/docs/product_first_user_flows.md @@ -35,7 +35,7 @@ activation flow below describes the target lifecycle. 2. Project owner provides open-ended guide material and business terms. 3. An authorized covered Project Manager adds guide metadata with at least one ordinary-text task example in PostgreSQL and uploads the assigned - PDF/DOCX/PPTX guide originals to ArtifactStore/S3. + PDF/DOCX/PPTX or UTF-8 Markdown (`.md`) guide originals to ArtifactStore/S3. 4. Once ART commits every assigned original guide document, Workstream automatically queues its authorized unified compilation in Celery. 5. One unified compilation assesses sufficiency and proposes artifact, diff --git a/docs/roadmap_status.md b/docs/roadmap_status.md index 7fe1cd9ac..0a0aed2dc 100644 --- a/docs/roadmap_status.md +++ b/docs/roadmap_status.md @@ -432,9 +432,10 @@ cannot be reused as post-submission review-gate evidence. See the response supplies document IDs for the public binary upload route; no separate source-snapshot creation call remains. A starting idea is sufficient and optional example fields do not repeat requirements from the guide. Each snapshot/run - binds the exact version's examples. Upload admission checks bounded format, digest and size. + binds the exact version's examples. Upload admission accepts PDF, DOCX, PPTX + and byte-preserved UTF-8 Markdown (`.md`) and checks bounded format, digest and size. Committed originals do not bypass the separate verification required for - submission ZIPs. Inline markdown setup and extractors are removed. + submission ZIPs. Embedded document bodies and extractors are removed. - Committed original-document readiness automatically runs one unified compilation through Celery. It persists sufficiency findings and separate pre-submit and post-submit proposals, then projects the sufficiency and artifact-policy diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 67d3e1796..e4df47900 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -835,8 +835,10 @@ one; then seal the computed commitment and second-pass stream together as `CommittedArtifactSource`. Guide ingest never permits a caller digest or size commitment. The raw upload -must match the snapshot item's server-bound declared PDF/DOCX/PPTX media type; -format inspection validates the actual file instead of trusting a header alone. +must match the snapshot item's server-bound declared PDF/DOCX/PPTX or +`text/markdown` media type; format inspection validates the actual file instead +of trusting a header alone. Markdown admission requires byte-preserved UTF-8 +without NUL and does not parse, render or rewrite the original. Preparation computes the content digest and byte count. Durable admission locks the exact project/guide/snapshot/item, requires a draft guide, and revalidates `artifact.guide_source.ingest`. A committed `document_stored` receipt, or exact @@ -1461,10 +1463,11 @@ provider internals. Guide-source ingestion stores immutable original documents through ArtifactStore. PostgreSQL records the source manifest, versions, exact content identity, upload receipts and run custody. It does not store extracted guide bodies. Current -uploads accept PDF, DOCX and PPTX. Task examples are required ordinary PostgreSQL -text inputs alongside the guide metadata; they do not create Workstream Tasks. -Markdown-only setup, separate extraction jobs and the three-agent setup APIs -are removed. Submission ZIP verification remains a separate workflow. +uploads accept PDF, DOCX, PPTX and UTF-8 Markdown (`.md`). Task examples are +required ordinary PostgreSQL text inputs alongside the guide metadata; they do +not create Workstream Tasks. Embedded Markdown bodies, separate extraction jobs +and the three-agent setup APIs are removed. Submission ZIP verification remains +a separate workflow. Upload admission inspects the bounded declared file format and computes size and SHA-256. A committed `document_stored` receipt establishes guide readiness; diff --git a/docs/spec_authorization_service.md b/docs/spec_authorization_service.md index f72358c04..18bc93186 100644 --- a/docs/spec_authorization_service.md +++ b/docs/spec_authorization_service.md @@ -1231,8 +1231,9 @@ payout/economic, or contribution-record configuration fields. Guide create requires the complete document declarations and task examples stored as immutable PostgreSQL metadata. AUTH receives the request digest, example hash and count; the guide and exact replay response are committed together. Document/upload snapshots -bind that commitment and receive original PDF/DOCX/PPTX files through ART. -Inline Markdown and URL/repository ingestion are unavailable. Only bounded +bind that commitment and receive original PDF/DOCX/PPTX or UTF-8 Markdown +(`.md`) files through ART. Embedded Markdown bodies and URL/repository ingestion +are unavailable. Only bounded metadata such as `change_summary` remains editable while the guide is draft. For guide creation and document upload, exact committed retries return the recorded response without another mutation, setup run, or dispatch. Changed, concurrent-pending, cross-project, diff --git a/docs/template_submission_artifact_policy.md b/docs/template_submission_artifact_policy.md index 43d607908..9eafe8154 100644 --- a/docs/template_submission_artifact_policy.md +++ b/docs/template_submission_artifact_policy.md @@ -44,7 +44,7 @@ Source snapshot items: | Item ID | Item Order | Source Kind | Source Label | Ingestion Adapter | Media Type | | --- | --- | --- | --- | --- | --- | -| `` | `` | `document` | `` | `upload` | `` | +| `` | `` | `document` | `` | `upload` | `` | Guide documents are uploaded originals in ArtifactStore/S3. Source labels must not store query strings, signed URLs, credentials, token-bearing refs, local From 34b6cd9c39904743acff21f008ce5702cf4dd632 Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 13:57:25 +0100 Subject: [PATCH 03/12] test: preserve guide media on new-key replay --- backend/tests/test_guide_document_intake.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/tests/test_guide_document_intake.py b/backend/tests/test_guide_document_intake.py index cce9674cf..8cb274971 100644 --- a/backend/tests/test_guide_document_intake.py +++ b/backend/tests/test_guide_document_intake.py @@ -317,7 +317,7 @@ async def recover(attempt_id): changed = await project_client.post(path, headers=headers, content=original + b"changed") assert changed.status_code == 409, changed.text another_key = await project_client.post( - path, headers=auth_headers() | {"Content-Type": "application/pdf"}, content=original, + path, headers=auth_headers() | {"Content-Type": headers["Content-Type"]}, content=original, ) assert another_key.status_code == 409, another_key.text assert await _stored_keys(get_settings()) == before_keys From 307f31a8a0dff3e544d53e6afeb16474f4cc4da1 Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 13:59:31 +0100 Subject: [PATCH 04/12] test: reject HTML guide declarations --- backend/tests/test_guide_document_api_contract.py | 2 +- backend/tests/test_guide_formats.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/backend/tests/test_guide_document_api_contract.py b/backend/tests/test_guide_document_api_contract.py index 5787298d9..d91a9eec8 100644 --- a/backend/tests/test_guide_document_api_contract.py +++ b/backend/tests/test_guide_document_api_contract.py @@ -27,7 +27,7 @@ def test_current_guide_response_excludes_retained_body(): @pytest.mark.parametrize("patch", [ {"source_kind": "url_doc"}, {"source_kind": "rubric"}, {"ingestion_adapter": "manual_import"}, - {"media_type": "text/plain"}, {"media_type": "image/png"}, + {"media_type": "text/plain"}, {"media_type": "text/html"}, {"media_type": "image/png"}, {"media_type": "audio/wav"}, {"media_type": "application/vnd.ms-powerpoint"}, ]) def test_source_metadata_rejects_superseded_or_unsupported_ingress(patch): diff --git a/backend/tests/test_guide_formats.py b/backend/tests/test_guide_formats.py index 82f85c4df..fa58a5fd3 100644 --- a/backend/tests/test_guide_formats.py +++ b/backend/tests/test_guide_formats.py @@ -237,6 +237,7 @@ def test_fixed_v01_limits_accept_exact_values_and_reject_one_over() -> None: (b'{"answer":42}', "application/json", "json"), (b"a,b\n1,2\n", "text/csv", "csv"), (b"Guide", "text/plain", "upload"), + (b"Guide", "text/html", "upload"), (b"\x89PNG\r\n\x1a\n", "image/png", "upload"), (b"ID3\x00", "audio/mpeg", "upload"), (b"\xff\xfe", "text/plain", "upload"), From bde7a120ace6912fb2803da90d5f509c8bc0c7b6 Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 15:08:02 +0100 Subject: [PATCH 05/12] feat: [PILOT-15] complete Markdown guide reads --- .../pilot15-markdown-guide-documents.md | 68 +++++++++++-------- backend/alembic/env.py | 4 +- .../versions/0027_markdown_guide_media.py | 51 ++++++++++++++ .../app/modules/tasks/api/guide_documents.py | 1 + backend/tests/committed_guide_fixtures.py | 4 +- backend/tests/conftest.py | 2 +- .../tests/reviews/packet/test_migration.py | 53 +++++++++++++++ .../tests/tasks/guide_document_fixtures.py | 22 ++++-- backend/tests/tasks/test_guide_documents.py | 13 ++-- backend/tests/test_alembic.py | 2 +- cli/README.md | 4 +- cli/internal/api/task_guide.go | 2 + cli/tests/integration/test_task_guide_http.py | 26 +++++++ .../integration/test_task_guide_public_api.py | 6 +- 14 files changed, 207 insertions(+), 51 deletions(-) create mode 100644 backend/alembic/versions/0027_markdown_guide_media.py diff --git a/.commitrail/changes/pilot15-markdown-guide-documents.md b/.commitrail/changes/pilot15-markdown-guide-documents.md index 0ff6ad773..38bc3eaa0 100644 --- a/.commitrail/changes/pilot15-markdown-guide-documents.md +++ b/.commitrail/changes/pilot15-markdown-guide-documents.md @@ -1,7 +1,7 @@ # [PILOT-15] Admit Markdown Guide Originals - Initiative: None -- Durable disposition: Planned +- Durable disposition: Complete - Intended merge outcome: A Project Manager can declare and upload a `text/markdown` guide original that retains byte-exact ART custody and is available through the existing setup-agent and assigned-contributor reads. @@ -20,8 +20,8 @@ not executable input or a separate storage path. The public guide declaration, ART upload format inspection, setup-agent manifest, review-packet metadata and Go declaration client consume that closed type. The database constraint on `review_packet_guide_items.media_type` independently -repeats the same three values. PILOT-13 PR #505 adds task-locked contributor -original reads and CLI downloads, but its open branch is not delivered behavior. +repeats the same three values. Delivered PILOT-13 provides task-locked +contributor original reads and CLI downloads with the same closed media type. ## Bounded change @@ -35,12 +35,13 @@ original reads and CLI downloads, but its open branch is not delivered behavior. - The existing setup-agent manifest/workspace path and focused tests proving a `.md` original is opened under the same run-scoped grant, disabled-network container and exact resource cleanup. -- Review-packet typed metadata/model constraint plus one forward-only migration - after the actual merged migration head, its graph/schema consumers and direct - PostgreSQL refusal/acceptance proof. +- Review-packet typed metadata/model constraint plus forward-only revision + `0027_markdown_guide_media` after delivered `0026_task_guide_read`, its + graph/schema consumers and direct PostgreSQL refusal/acceptance proof. - The Go guide-declaration client and current CLI/API documentation for - `text/markdown`. After PR #505 merges, its task guide DTO/extension/download - consumers and tests may be reconciled without copying its implementation. + `text/markdown`. The merged PILOT-13 task guide DTO/extension/download consumers, + their real upload/read fixtures and direct tests are reconciled without copying + its implementation. - Existing guide API, real PostgreSQL/MinIO intake fixtures, format, setup workspace, review-packet, CLI declaration and later PILOT-13 contributor-read tests; current README, artifact/guide specifications, roadmap and this record. @@ -55,8 +56,8 @@ original reads and CLI downloads, but its open branch is not delivered behavior. - No checker catalogue/registration changes, activation refusal or recovery behavior; F-020 belongs to PILOT-04 / issue #491. - No duplicate contributor read, task authority, streaming or CLI download - implementation while PR #505 remains open. No compatibility variant, data - rewrite, downgrade, deployment, merge or issue closure. + implementation. No compatibility variant, data rewrite, downgrade, + deployment, merge or issue closure. ## Design and decisions @@ -71,20 +72,20 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. ## Acceptance criteria -- [ ] Public API and Go CLI declaration accept exactly `text/markdown`; HTML, +- [x] Public API and Go CLI declaration accept exactly `text/markdown`; HTML, `text/plain`, binary/malformed UTF-8 and media-type substitution remain denied. -- [ ] Real PostgreSQL/MinIO intake stores and reads back the exact Markdown +- [x] Real PostgreSQL/MinIO intake stores and reads back the exact Markdown bytes with the server-computed SHA-256, byte count and declared media type. -- [ ] The existing setup-agent workspace opens the exact `.md` version through +- [x] The existing setup-agent workspace opens the exact `.md` version through its run-scoped grant and retains disabled network, bounded resources and cleanup. -- [ ] Review-packet metadata and PostgreSQL accept the exact Markdown member +- [x] Review-packet metadata and PostgreSQL accept the exact Markdown member while direct writes of unsupported media types still fail. -- [ ] After PILOT-13 merges, an assigned contributor lists and downloads the +- [x] An assigned contributor lists and downloads the task-locked Markdown original byte-identically through REST/CLI; foreign, unassigned, corrupt and unsafe-destination cases retain their existing denial. -- [ ] PDF, DOCX and PPTX declaration/upload/setup/review/contributor behavior +- [x] PDF, DOCX and PPTX declaration/upload/setup/review/contributor behavior remains covered and unchanged; no second migration head is introduced. -- [ ] Current API/CLI, artifact/guide specifications and roadmap state the +- [x] Current API/CLI, artifact/guide specifications and roadmap state the delivered boundary without claiming HTML, remote sites or deployed capability. ## Risk and review routing @@ -100,24 +101,31 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. | Claim | Command or proof | Result | Remaining uncertainty | |---|---|---|---| -| Bounded Markdown classification | Focused `test_guide_formats.py` and upload mismatch controls | Required before completion | Markdown semantics are not parsed or rendered | -| Durable original custody | Isolated real PostgreSQL/MinIO guide-intake journey with provider readback | Required before completion | Hosted S3 remains shared ART-provider evidence | -| Setup-agent exact read | Existing workspace/runtime tests with Markdown version and byte-bearing grant | Required before completion | No live model inference is required for byte-access proof | -| Persisted review metadata | Real PostgreSQL migration/direct-write and review-packet storage proof | Required after PR #505 establishes the predecessor | Existing rows are retained; no downgrade is supported | -| Contributor REST/CLI | PILOT-13 focused public API and built-binary tests extended after merge | Blocked on merged PR #505 | Open-branch code is inspection input, not delivered proof | +| Bounded Markdown classification | Focused format/API/review-packet/setup batch plus Ruff | 126 passed; strict streaming UTF-8, binary substitution and HTML/plain-text negatives passed | Markdown semantics are not parsed or rendered | +| Durable original custody | Isolated PostgreSQL 16/MinIO intake journey with provider readback, replay and recovery | Both journey variants passed; runner verified database and MinIO cleanup | Hosted S3 remains shared ART-provider evidence | +| Setup-agent exact read | Workspace/runtime batch with Markdown version, byte-bearing grant and disabled network | Exact `.md` bytes, media type, run handle and cleanup passed | No live model inference is required for byte-access proof | +| Persisted review metadata | PostgreSQL 16 predecessor/head direct-write and schema-fingerprint proofs | Predecessor rejected Markdown; 0027 admitted it past the media check; HTML remained rejected; graph and regenerated fingerprint passed | Existing rows are retained; no downgrade is supported | +| Contributor REST/CLI | Real PostgreSQL/MinIO locked-read batch and built-CLI HTTP/public-API journeys | Six backend cases, 34 HTTP cases and the real built-CLI API journey passed, including concealment, successor locking, corrupt originals and `.md` publication | Deployment remains outside this change | ## Review findings -No review finding has produced a durable source change yet. +- Real PostgreSQL/MinIO replay exposed a PDF-specific alternate-key fixture; + the control now reuses each declared media type while preserving conflict proof. +- The merged PILOT-13 policy setup fixture assumed every already-committed source + was PDF. It now accepts an existing ingest only when its media matches the + snapshot item, while its synthetic fallback remains PDF-only. +- FastAPI correctly appends the UTF-8 charset parameter to a Markdown response; + the REST test compares its parsed base media type while the CLI independently + parses and requires the advertised `text/markdown` type. ## Reconciliation -- Current-source reconciliation: Based on main - `3fa0dfb9eeda18888cdced494dfe813bd8dacf63`, whose migration head is - `0025_submission_dispatch`. PR #505 currently proposes the next guide-read - migration and contributor read/CLI owners; this branch will not allocate its - revision or duplicate those owners before it merges. -- Next usable boundary: Merge and verify PILOT-13, then reconcile its exact task - guide media type and CLI extension consumers and allocate the next migration. +- Current-source reconciliation: Rebased on main + `72b83ffc2f0fcb29efdc68f6babaee884ab7d4dd`, where PILOT-13 is delivered as + revision `0026_task_guide_read`. Open PRs #509 and #510 add no migration, so + this branch owns the next linear revision, `0027_markdown_guide_media`. +- Delivered dependency reconciliation: PILOT-13's exact task-locked authority, + concealment, verified streaming and safe CLI publication are unchanged; only + its closed media type and extension maps admit Markdown. - Remaining risks: Live provider/model inference is outside this byte-custody change. Hosted deployment and HTML/docs-site ingestion remain unsupported. diff --git a/backend/alembic/env.py b/backend/alembic/env.py index 58f65e70d..4033591fa 100644 --- a/backend/alembic/env.py +++ b/backend/alembic/env.py @@ -21,7 +21,7 @@ target_metadata = Base.metadata _BASELINE_REVISION = "0001_uuid7_v01" -_CURRENT_HEAD_REVISION = "0026_task_guide_read" +_CURRENT_HEAD_REVISION = "0027_markdown_guide_media" _RECREATE_GUIDANCE = ( "Workstream v0.1 requires a fresh database; recreate this database before " "running the 0001_uuid7_v01 migration" @@ -52,7 +52,7 @@ def do_run_migrations(connection: Connection) -> None: .scalars() .all() ) - if revisions not in ((), (_BASELINE_REVISION,), ("0002_task_queue_authority",), ("0003_task_read_authority",), ("0004_task_context_authority",), ("0005_task_evidence_authority",), ("0006_history_read_authority",), ("0007_checker_output_custody",), ("0008_checker_execution",), ("0009_checker_material_lineage",), ("0010_post_submit_authority",), ("0011_task_routing_source",), ("0012_review_packet",), ("0013_review_source",), ("0014_final_acceptance",), ("0015_contribution_awards",), ("0016_review_lifecycle_fence",), ("0017_acceptance_source_contracts",), ("0018_task_routing_request",), ("0019_submitter_awards",), ("0020_review_admission_lock_order",), ("0021_submission_manifest",), ("0022_submission_packet_custody",), ("0023_remove_task_payment_policy",), ("0024_require_second_review_false",), ("0025_submission_dispatch",), (_CURRENT_HEAD_REVISION,)): + if revisions not in ((), (_BASELINE_REVISION,), ("0002_task_queue_authority",), ("0003_task_read_authority",), ("0004_task_context_authority",), ("0005_task_evidence_authority",), ("0006_history_read_authority",), ("0007_checker_output_custody",), ("0008_checker_execution",), ("0009_checker_material_lineage",), ("0010_post_submit_authority",), ("0011_task_routing_source",), ("0012_review_packet",), ("0013_review_source",), ("0014_final_acceptance",), ("0015_contribution_awards",), ("0016_review_lifecycle_fence",), ("0017_acceptance_source_contracts",), ("0018_task_routing_request",), ("0019_submitter_awards",), ("0020_review_admission_lock_order",), ("0021_submission_manifest",), ("0022_submission_packet_custody",), ("0023_remove_task_payment_policy",), ("0024_require_second_review_false",), ("0025_submission_dispatch",), ("0026_task_guide_read",), (_CURRENT_HEAD_REVISION,)): raise RuntimeError(_RECREATE_GUIDANCE) # The read-only preflight autobegins a SQLAlchemy transaction. End that # transaction before Alembic establishes the migration transaction; diff --git a/backend/alembic/versions/0027_markdown_guide_media.py b/backend/alembic/versions/0027_markdown_guide_media.py new file mode 100644 index 000000000..f6ae37c79 --- /dev/null +++ b/backend/alembic/versions/0027_markdown_guide_media.py @@ -0,0 +1,51 @@ +"""Admit Markdown to the persisted review-packet guide media invariant.""" + +from alembic import op +import sqlalchemy as sa + +revision = "0027_markdown_guide_media" +down_revision = "0026_task_guide_read" +branch_labels = None +depends_on = None + +_CONSTRAINT = "ck_review_packet_guide_items_guide_media" +_PREDECESSOR_DEFINITION = ( + "CHECK (((media_type)::text = ANY ((ARRAY['application/pdf'::character varying, " + "'application/vnd.openxmlformats-officedocument.wordprocessingml.document'::character varying, " + "'application/vnd.openxmlformats-officedocument.presentationml.presentation'::character varying])::text[])))" +) +_CURRENT_EXPRESSION = ( + "media_type in ('application/pdf'," + "'application/vnd.openxmlformats-officedocument.wordprocessingml.document'," + "'application/vnd.openxmlformats-officedocument.presentationml.presentation'," + "'text/markdown')" +) + + +def upgrade(): + op.execute("set local search_path = pg_catalog, public, pg_temp") + op.execute("lock table public.review_packet_guide_items in access exclusive mode") + row = ( + op.get_bind() + .execute( + sa.text( + "select pg_get_constraintdef(oid), convalidated " + "from pg_catalog.pg_constraint " + "where conrelid='public.review_packet_guide_items'::regclass " + "and conname=:name" + ), + {"name": _CONSTRAINT}, + ) + .one() + ) + if row[0] != _PREDECESSOR_DEFINITION or row[1] is not True: + raise RuntimeError("review packet guide media constraint shape changed") + op.execute(f"alter table public.review_packet_guide_items drop constraint {_CONSTRAINT}") + op.execute( + f"alter table public.review_packet_guide_items add constraint {_CONSTRAINT} " + f"check ({_CURRENT_EXPRESSION})" + ) + + +def downgrade(): + raise RuntimeError("Workstream v0.1 migrations cannot be downgraded; recreate the database") diff --git a/backend/app/modules/tasks/api/guide_documents.py b/backend/app/modules/tasks/api/guide_documents.py index dc9b4db37..e77b2e52e 100644 --- a/backend/app/modules/tasks/api/guide_documents.py +++ b/backend/app/modules/tasks/api/guide_documents.py @@ -31,6 +31,7 @@ class ContributorGuideDocument(BaseModel): "application/pdf", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "application/vnd.openxmlformats-officedocument.presentationml.presentation", + "text/markdown", ] byte_count: int = Field(gt=0) sha256: str = Field(pattern=r"^sha256:[0-9a-f]{64}$") diff --git a/backend/tests/committed_guide_fixtures.py b/backend/tests/committed_guide_fixtures.py index aac68ef18..315b9a7b1 100644 --- a/backend/tests/committed_guide_fixtures.py +++ b/backend/tests/committed_guide_fixtures.py @@ -52,10 +52,12 @@ async def create_committed_document_fixture(source_snapshot_id: str, *, sessions session.add(replica) await session.flush() for item in items: - assert (item.source_kind, item.ingestion_adapter, item.media_type) == ("document", "upload", "application/pdf") + assert (item.source_kind, item.ingestion_adapter) == ("document", "upload") existing = await session.scalar(select(GuideSourceArtifactIngest).where(GuideSourceArtifactIngest.source_item_id == item.id)) if existing is not None: + assert existing.media_type == item.media_type continue + assert item.media_type == "application/pdf" session.add(GuideSourceArtifactIngest(id=str(new_record_id()), source_item_id=item.id, actor_profile_id=actor.id, sha256=SOURCE_SHA256, byte_count=len(SOURCE_BYTES), media_type="application/pdf")) put = ArtifactPutAttempt(id=str(new_record_id()), producer_request_type="guide", producer_type="actor_profile", producer_ref=actor.id, project_id=snapshot.project_id, guide_source_item_id=item.id, sha256=SOURCE_SHA256, byte_count=len(SOURCE_BYTES), media_type="application/pdf", storage_namespace_id=namespace.id, namespace_fingerprint=namespace.namespace_fingerprint, canonical_target=f"sha256/{SOURCE_SHA256[7:9]}/{SOURCE_SHA256[9:]}", operation_identity=sha256_hash(item.id), request_digest=sha256_hash("request:" + item.id), status="object_confirmed", terminal_result_code="document_stored", replica_id=replica.id, terminal_at=datetime.now(timezone.utc)) session.add(put) diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py index 02dd7a90b..1782ebe73 100644 --- a/backend/tests/conftest.py +++ b/backend/tests/conftest.py @@ -23,7 +23,7 @@ DDL_LOCK_DIRECTORY = Path("/tmp") # Match the PostgreSQL 16 engine used by Backend CI. Catalog identity rendering # differs across major versions; regenerate only after comparing actual objects. -EXPECTED_PUBLIC_SCHEMA_SHA256 = "27ba211f2e2daded2c41e661163a73ed173f53612ee03a8d922a6845a81d0be8" +EXPECTED_PUBLIC_SCHEMA_SHA256 = "07aeeac4912292e382127fbbf4c33357b3a5546e50b8c7ecd3465248aa95d3da" PROTECTED_TEST_TABLES = ( "joint_lifecycle_release_control", "actor_profile_migration_state", diff --git a/backend/tests/reviews/packet/test_migration.py b/backend/tests/reviews/packet/test_migration.py index ebe584a5c..afb5ce105 100644 --- a/backend/tests/reviews/packet/test_migration.py +++ b/backend/tests/reviews/packet/test_migration.py @@ -1,6 +1,7 @@ """Populated predecessor upgrade preserves canonical owners without inventing packets.""" import asyncio +from uuid import uuid4 import asyncpg import pytest @@ -10,6 +11,7 @@ from app.db import session as db_session from tests.migration_fixtures import add_current_art_seed_column, restore_predecessor_evidence_schema from tests.migration_fixtures import _config +from tests.migration_fixtures import current_schema_revision from tests.reviews.packet.support import packet_source pytestmark = pytest.mark.postgres_schema_contract @@ -77,3 +79,54 @@ async def test_packet_upgrade_preserves_existing_owners( ) finally: await connection.close() + + +async def test_markdown_media_upgrade_changes_only_the_closed_packet_check( + isolated_database_env, migration_lock +): + url = isolated_database_env.replace("+asyncpg", "") + insert = ( + "INSERT INTO public.review_packet_guide_items " + "(packet_id,source_item_id,ingest_id,item_order,logical_role,media_type) " + "VALUES ($1,$2,$3,0,'guide_source_original',$4)" + ) + with migration_lock(): + await db_session.dispose_engine() + connection = await asyncpg.connect(url) + try: + await connection.execute("DROP SCHEMA public CASCADE; CREATE SCHEMA public") + finally: + await connection.close() + await asyncio.to_thread(command.upgrade, _config(), "0026_task_guide_read") + connection = await asyncpg.connect(url) + try: + selectors = tuple(uuid4() for _ in range(3)) + with pytest.raises(asyncpg.CheckViolationError) as predecessor: + await connection.execute(insert, *selectors, "text/markdown") + assert predecessor.value.constraint_name == ( + "ck_review_packet_guide_items_guide_media" + ) + finally: + await connection.close() + + await asyncio.to_thread(command.upgrade, _config(), current_schema_revision()) + connection = await asyncpg.connect(url) + try: + with pytest.raises(asyncpg.ForeignKeyViolationError) as accepted_media: + await connection.execute(insert, *selectors, "text/markdown") + assert accepted_media.value.constraint_name.startswith( + "fk_review_packet_guide_items_" + ) + with pytest.raises(asyncpg.CheckViolationError) as unsupported_media: + await connection.execute(insert, *selectors, "text/html") + assert unsupported_media.value.constraint_name == ( + "ck_review_packet_guide_items_guide_media" + ) + assert ( + await connection.fetchval( + "SELECT count(*) FROM public.review_packet_guide_items" + ) + == 0 + ) + finally: + await connection.close() diff --git a/backend/tests/tasks/guide_document_fixtures.py b/backend/tests/tasks/guide_document_fixtures.py index 614a0c0fe..22b7f9766 100644 --- a/backend/tests/tasks/guide_document_fixtures.py +++ b/backend/tests/tasks/guide_document_fixtures.py @@ -86,18 +86,23 @@ async def activate_uploaded_guide(client, project_id, monkeypatch, *, version="v "version": version, "task_examples": [{"content": "PRIVATE SETUP EXAMPLE"}], "documents": [ - {"label": label, "media_type": "application/pdf"} - for label in ("guide.pdf", "rubric.pdf") + {"label": "guide.md", "media_type": "text/markdown"}, + {"label": "rubric.pdf", "media_type": "application/pdf"}, ], }, ) assert created.status_code == 201, created.text guide = created.json() - originals = tuple(f"%PDF-1.7\n{version} original {index}\n%%EOF".encode() for index in range(2)) - for document, original in zip(guide["documents"], originals, strict=True): + originals = ( + f"# {version} guide\n\nFollow the locked original. 雪\n".encode(), + f"%PDF-1.7\n{version} rubric original\n%%EOF".encode(), + ) + for document, original, media_type in zip( + guide["documents"], originals, ("text/markdown", "application/pdf"), strict=True + ): stored = await client.post( f"/api/v1/projects/{project_id}/guides/{guide['id']}/documents/{document['document_id']}/content", - headers=auth_headers() | {"Content-Type": "application/pdf"}, + headers=auth_headers() | {"Content-Type": media_type}, content=original, ) assert stored.status_code == 202, stored.text @@ -109,7 +114,9 @@ async def activate_uploaded_guide(client, project_id, monkeypatch, *, version="v ) async with db_session.get_session_factory()() as session: assert (await session.get(ProjectGuide, guide["id"])).status == "active" - for document, original in zip(guide["documents"], originals, strict=True): + for document, original, media_type in zip( + guide["documents"], originals, ("text/markdown", "application/pdf"), strict=True + ): put = ( await session.scalars( select(ArtifactPutAttempt).where( @@ -124,9 +131,10 @@ async def activate_uploaded_guide(client, project_id, monkeypatch, *, version="v receipt.replica_id == replica.id and receipt.provider_object_ref == replica.provider_object_ref ) - assert (put.sha256, put.byte_count) == ( + assert (put.sha256, put.byte_count, put.media_type) == ( "sha256:" + hashlib.sha256(original).hexdigest(), len(original), + media_type, ) from tests.test_guide_document_intake import _open_store diff --git a/backend/tests/tasks/test_guide_documents.py b/backend/tests/tasks/test_guide_documents.py index e889213ff..f73a25a50 100644 --- a/backend/tests/tasks/test_guide_documents.py +++ b/backend/tests/tasks/test_guide_documents.py @@ -70,9 +70,9 @@ async def test_assigned_originals_are_ordered_and_verified(task_client, guide_wo "read_reference", } assert document["document_id"] == guide_world.guide["documents"][index]["document_id"] - assert ( - document["order"] == index and document["label"] == ("guide.pdf", "rubric.pdf")[index] - ) + assert document["order"] == index + assert document["label"] == ("guide.md", "rubric.pdf")[index] + assert document["media_type"] == ("text/markdown", "application/pdf")[index] assert (document["sha256"], document["byte_count"]) == ( "sha256:" + hashlib.sha256(original).hexdigest(), len(original), @@ -81,7 +81,10 @@ async def test_assigned_originals_are_ordered_and_verified(task_client, guide_wo assert read.status_code == 200, read.text assert read.content == original and read.headers["cache-control"] == "private, no-store" assert int(read.headers["content-length"]) == len(original) - assert read.headers["content-type"] == "application/pdf" + assert read.headers["content-type"].partition(";")[0] == ( + "text/markdown", + "application/pdf", + )[index] async def test_unassigned_and_foreign_actors_are_concealed(task_client, guide_world, monkeypatch): @@ -510,7 +513,7 @@ async def test_damaged_stored_original_never_serves_partial_bytes(task_client, g response = await task_client.get(content_path(world), headers=auth_headers()) assert response.status_code == 503, response.text assert response.json()["error"]["code"] == "guide_document_integrity_unavailable" - assert b"%PDF" not in response.content and key not in response.text + assert world.originals[0] not in response.content and key not in response.text finally: store.close() bootstrap.close() diff --git a/backend/tests/test_alembic.py b/backend/tests/test_alembic.py index 0cf9ce18f..0f4c93835 100644 --- a/backend/tests/test_alembic.py +++ b/backend/tests/test_alembic.py @@ -79,7 +79,7 @@ def test_v01_graph_has_one_root_and_head() -> None: script = ScriptDirectory.from_config(config) revisions = list(script.walk_revisions()) - assert [revision.revision for revision in revisions] == [HEAD_REVISION, "0025_submission_dispatch", "0024_require_second_review_false", "0023_remove_task_payment_policy", "0022_submission_packet_custody", "0021_submission_manifest", "0020_review_admission_lock_order", "0019_submitter_awards", "0018_task_routing_request", "0017_acceptance_source_contracts", "0016_review_lifecycle_fence", "0015_contribution_awards", "0014_final_acceptance", "0013_review_source", "0012_review_packet", "0011_task_routing_source", "0010_post_submit_authority", "0009_checker_material_lineage", "0008_checker_execution", "0007_checker_output_custody", "0006_history_read_authority", "0005_task_evidence_authority", "0004_task_context_authority", "0003_task_read_authority", "0002_task_queue_authority", BASELINE_REVISION] + assert [revision.revision for revision in revisions] == [HEAD_REVISION, "0026_task_guide_read", "0025_submission_dispatch", "0024_require_second_review_false", "0023_remove_task_payment_policy", "0022_submission_packet_custody", "0021_submission_manifest", "0020_review_admission_lock_order", "0019_submitter_awards", "0018_task_routing_request", "0017_acceptance_source_contracts", "0016_review_lifecycle_fence", "0015_contribution_awards", "0014_final_acceptance", "0013_review_source", "0012_review_packet", "0011_task_routing_source", "0010_post_submit_authority", "0009_checker_material_lineage", "0008_checker_execution", "0007_checker_output_custody", "0006_history_read_authority", "0005_task_evidence_authority", "0004_task_context_authority", "0003_task_read_authority", "0002_task_queue_authority", BASELINE_REVISION] assert revisions[-1].down_revision is None assert script.get_heads() == [HEAD_REVISION] diff --git a/cli/README.md b/cli/README.md index bed2aff10..68f37b178 100644 --- a/cli/README.md +++ b/cli/README.md @@ -313,8 +313,8 @@ responding. Missing, corrupt or wrong-namespace originals fail with are private/no-store; setup-agent run-scoped access remains unchanged. The CLI reconstructs fixed same-origin paths and verifies size/SHA-256 again. -Files use canonical document UUID names with `.pdf`, `.docx` or `.pptx`, never -labels as paths. The destination must already exist and must not be a symlink. +Files use canonical document UUID names with `.pdf`, `.docx`, `.pptx` or `.md`, +never labels as paths. The destination must already exist and must not be a symlink. Downloads use private bounded temporary files and atomic no-overwrite publication; existing targets/symlinks are refused and failed unpublished files are removed. Documents completed before a later document fails remain valid local files. diff --git a/cli/internal/api/task_guide.go b/cli/internal/api/task_guide.go index bc469d28b..d2f8b3bbf 100644 --- a/cli/internal/api/task_guide.go +++ b/cli/internal/api/task_guide.go @@ -46,6 +46,8 @@ func (d TaskGuideDocument) Extension() string { return ".docx" case "application/vnd.openxmlformats-officedocument.presentationml.presentation": return ".pptx" + case "text/markdown": + return ".md" default: return "" } diff --git a/cli/tests/integration/test_task_guide_http.py b/cli/tests/integration/test_task_guide_http.py index b8b308c28..f2e41e6f2 100644 --- a/cli/tests/integration/test_task_guide_http.py +++ b/cli/tests/integration/test_task_guide_http.py @@ -135,6 +135,32 @@ def test_assigned_guide_list_and_verified_private_download(cli, tmp_path): assert len(requests) == before + 1 # No second download attempt or overwrite. +def test_markdown_guide_download_uses_canonical_md_name_and_exact_bytes(cli, tmp_path): + markdown = "# Locked guide\n\nRead this exact original. 雪\n".encode() + with guide_http() as (origin, state, requests): + document = state["context"]["guide_documents"][0] + document.update( + label="../../untrusted-name.html", + media_type="text/markdown", + byte_count=len(markdown), + sha256="sha256:" + hashlib.sha256(markdown).hexdigest(), + ) + state["bytes"] = markdown + state["headers"] = {"Content-Type": "text/markdown; charset=utf-8"} + destination = tmp_path / "markdown" + destination.mkdir() + + downloaded = cli( + origin, TOKEN, "task", "guide", TASK, "--download", str(destination), "-o", "json" + ) + + assert downloaded.returncode == 0, downloaded.stderr + target = destination / f"{ACTOR}.md" + assert target.read_bytes() == markdown and target.stat().st_mode & 0o777 == 0o600 + assert list(destination.iterdir()) == [target] + assert requests[-1] == (document["read_reference"], "Bearer " + TOKEN) + + @pytest.mark.parametrize("phase", ("headers", "body", "context")) def test_guide_download_has_its_own_deadline_but_json_keeps_twelve_seconds( cli, diff --git a/cli/tests/integration/test_task_guide_public_api.py b/cli/tests/integration/test_task_guide_public_api.py index ff9b0658d..eb671954a 100644 --- a/cli/tests/integration/test_task_guide_public_api.py +++ b/cli/tests/integration/test_task_guide_public_api.py @@ -109,9 +109,11 @@ async def test_installed_cli_reads_and_downloads_real_assigned_originals( assert result.returncode == 0, result.stderr documents = json.loads(result.stdout) assert len(documents) == len(guide_world.originals) == 2 - for document, original in zip(documents, guide_world.originals, strict=True): + for document, original, extension in zip( + documents, guide_world.originals, ("md", "pdf"), strict=True + ): assert ( - directory / f"{document['document_id']}.pdf" + directory / f"{document['document_id']}.{extension}" ).read_bytes() == original assert ( "task_examples" not in result.stdout From 58f4c642b0329893d04ce7b84f8fed0d13454a5a Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 16:10:07 +0100 Subject: [PATCH 06/12] test: reconcile Markdown guide CI gates --- .../pilot15-markdown-guide-documents.md | 7 +++ backend/tests/test_guide_document_intake.py | 47 +++++++++++-------- cli/tests/integration/test_task_guide_http.py | 14 +++++- 3 files changed, 46 insertions(+), 22 deletions(-) diff --git a/.commitrail/changes/pilot15-markdown-guide-documents.md b/.commitrail/changes/pilot15-markdown-guide-documents.md index 38bc3eaa0..08e02bd60 100644 --- a/.commitrail/changes/pilot15-markdown-guide-documents.md +++ b/.commitrail/changes/pilot15-markdown-guide-documents.md @@ -117,6 +117,13 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. - FastAPI correctly appends the UTF-8 charset parameter to a Markdown response; the REST test compares its parsed base media type while the CLI independently parses and requires the advertised `text/markdown` type. +- Hosted boundary validation found that the MinIO intake proof had grown past + the frozen 120-line test limit. A focused helper now performs only the stored + original lookup; every replica and byte assertion remains in the same primary + test node, which falls below the hard limit and adds no structural-debt entry. +- The CLI public-contract formatter found the Markdown download journey was not + in canonical Ruff form. The focused integration source is now formatted and + remains covered by the built-CLI HTTP journey. ## Reconciliation diff --git a/backend/tests/test_guide_document_intake.py b/backend/tests/test_guide_document_intake.py index 8cb274971..aa483ad14 100644 --- a/backend/tests/test_guide_document_intake.py +++ b/backend/tests/test_guide_document_intake.py @@ -193,6 +193,29 @@ async def _stored_put_state(project_id): ).where(ArtifactPutAttempt.project_id == project_id).order_by(ArtifactPutAttempt.id))).all() +async def _stored_originals(project_id): + from app.core.config import get_settings + from app.modules.artifacts.models import ArtifactReplica, ArtifactPutAttempt + + async with db_session.get_session_factory()() as session: + attempts = ( + await session.scalars( + select(ArtifactPutAttempt).where(ArtifactPutAttempt.project_id == project_id) + ) + ).all() + replicas = (await session.scalars(select(ArtifactReplica))).all() + bootstrap, store = _open_store(get_settings()) + try: + stored = [ + b"".join([chunk async for chunk in store.open(row.provider_object_ref)]) + for row in replicas + ] + finally: + store.close() + bootstrap.close() + return attempts, replicas, stored + + async def _assert_inactive_resolver_replay_denied( client, project_id, resolver_id, path, headers, original, provider_calls, deliveries, ): @@ -227,7 +250,6 @@ async def test_all_documents_stored_dispatches_once_through_minio( ): from app.core.config import get_settings from app.modules.actors.api import ServiceIdentity - from app.modules.artifacts.models import ArtifactReplica, ArtifactPutAttempt from app.workers.project_setup import run_project_guide_compilation from app.adapters.artifacts import internal_workers @@ -325,25 +347,10 @@ async def recover(attempt_id): async with db_session.get_session_factory()() as session: run = await session.get(ProjectSetupRun, guide["setup"]["id"]) assert run.status == ("awaiting_documents" if index == 0 else "queued") - async with db_session.get_session_factory()() as session: - attempts = ( - await session.scalars( - select(ArtifactPutAttempt).where(ArtifactPutAttempt.project_id == project["id"]) - ) - ).all() - replicas = (await session.scalars(select(ArtifactReplica))).all() - assert len(attempts) == len(replicas) == 2 - assert {row.media_type for row in attempts} == {"text/markdown", "application/pdf"} - bootstrap, store = _open_store(get_settings()) - try: - stored = [ - b"".join([chunk async for chunk in store.open(row.provider_object_ref)]) - for row in replicas - ] - assert set(stored) == set(originals) - finally: - store.close() - bootstrap.close() + attempts, replicas, stored = await _stored_originals(project["id"]) + assert len(attempts) == len(replicas) == 2 + assert {row.media_type for row in attempts} == {"text/markdown", "application/pdf"} + assert set(stored) == set(originals) # Even a completed put requires the fixed resolver's current authority. await _assert_inactive_resolver_replay_denied( diff --git a/cli/tests/integration/test_task_guide_http.py b/cli/tests/integration/test_task_guide_http.py index f2e41e6f2..e16731d16 100644 --- a/cli/tests/integration/test_task_guide_http.py +++ b/cli/tests/integration/test_task_guide_http.py @@ -151,12 +151,22 @@ def test_markdown_guide_download_uses_canonical_md_name_and_exact_bytes(cli, tmp destination.mkdir() downloaded = cli( - origin, TOKEN, "task", "guide", TASK, "--download", str(destination), "-o", "json" + origin, + TOKEN, + "task", + "guide", + TASK, + "--download", + str(destination), + "-o", + "json", ) assert downloaded.returncode == 0, downloaded.stderr target = destination / f"{ACTOR}.md" - assert target.read_bytes() == markdown and target.stat().st_mode & 0o777 == 0o600 + assert ( + target.read_bytes() == markdown and target.stat().st_mode & 0o777 == 0o600 + ) assert list(destination.iterdir()) == [target] assert requests[-1] == (document["read_reference"], "Bearer " + TOKEN) From 188f451ddc1c671c33c317ade0b3228f753a1a3d Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 17:41:14 +0100 Subject: [PATCH 07/12] docs: reconcile PILOT-15 with current main --- .commitrail/changes/pilot15-markdown-guide-documents.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/.commitrail/changes/pilot15-markdown-guide-documents.md b/.commitrail/changes/pilot15-markdown-guide-documents.md index 08e02bd60..cf5c1b4d0 100644 --- a/.commitrail/changes/pilot15-markdown-guide-documents.md +++ b/.commitrail/changes/pilot15-markdown-guide-documents.md @@ -127,10 +127,11 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. ## Reconciliation -- Current-source reconciliation: Rebased on main - `72b83ffc2f0fcb29efdc68f6babaee884ab7d4dd`, where PILOT-13 is delivered as - revision `0026_task_guide_read`. Open PRs #509 and #510 add no migration, so - this branch owns the next linear revision, `0027_markdown_guide_media`. +- Current-source reconciliation: Merged main + `36e8a615f01801cecd6ccf83d7411235ab1cfa8f`, where PILOT-13 is delivered as + revision `0026_task_guide_read`. The later PILOT-00 experiment and delivery + changes add no migration, so this branch retains the next linear revision, + `0027_markdown_guide_media`. - Delivered dependency reconciliation: PILOT-13's exact task-locked authority, concealment, verified streaming and safe CLI publication are unchanged; only its closed media type and extension maps admit Markdown. From cda8b9a8b8086ac942a21557f6f5e15c0da7a2c8 Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Thu, 8 Oct 2026 17:54:24 +0100 Subject: [PATCH 08/12] fix: advertise Markdown task guide downloads --- .../pilot15-markdown-guide-documents.md | 7 ++++-- backend/app/modules/tasks/router.py | 1 + .../tests/test_guide_document_api_contract.py | 22 +++++++++++++++++++ 3 files changed, 28 insertions(+), 2 deletions(-) diff --git a/.commitrail/changes/pilot15-markdown-guide-documents.md b/.commitrail/changes/pilot15-markdown-guide-documents.md index cf5c1b4d0..f9f12997c 100644 --- a/.commitrail/changes/pilot15-markdown-guide-documents.md +++ b/.commitrail/changes/pilot15-markdown-guide-documents.md @@ -40,8 +40,8 @@ contributor original reads and CLI downloads with the same closed media type. graph/schema consumers and direct PostgreSQL refusal/acceptance proof. - The Go guide-declaration client and current CLI/API documentation for `text/markdown`. The merged PILOT-13 task guide DTO/extension/download consumers, - their real upload/read fixtures and direct tests are reconciled without copying - its implementation. + task-read public route metadata, real upload/read fixtures and direct tests are + reconciled without copying its implementation. - Existing guide API, real PostgreSQL/MinIO intake fixtures, format, setup workspace, review-packet, CLI declaration and later PILOT-13 contributor-read tests; current README, artifact/guide specifications, roadmap and this record. @@ -124,6 +124,9 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. - The CLI public-contract formatter found the Markdown download journey was not in canonical Ruff form. The focused integration source is now formatted and remains covered by the built-CLI HTTP journey. +- Final OpenAPI review found the existing task-original GET still advertised + only the three predecessor binary formats. Its response metadata and an exact + generated-schema regression now include the already-supported Markdown original. ## Reconciliation diff --git a/backend/app/modules/tasks/router.py b/backend/app/modules/tasks/router.py index 90678cbe3..e526fafcc 100644 --- a/backend/app/modules/tasks/router.py +++ b/backend/app/modules/tasks/router.py @@ -464,6 +464,7 @@ async def verified_task_guide( "application/pdf": {"schema": {"type": "string", "format": "binary"}}, "application/vnd.openxmlformats-officedocument.wordprocessingml.document": {"schema": {"type": "string", "format": "binary"}}, "application/vnd.openxmlformats-officedocument.presentationml.presentation": {"schema": {"type": "string", "format": "binary"}}, + "text/markdown": {"schema": {"type": "string", "format": "binary"}}, }}}, ) async def get_task_guide_document(read: Annotated[VerifiedTaskGuideRead, Depends(verified_task_guide)]) -> StreamingResponse: diff --git a/backend/tests/test_guide_document_api_contract.py b/backend/tests/test_guide_document_api_contract.py index d91a9eec8..a6ba2caf5 100644 --- a/backend/tests/test_guide_document_api_contract.py +++ b/backend/tests/test_guide_document_api_contract.py @@ -167,3 +167,25 @@ def test_document_upload_openapi_binary_body_and_bounded_responses(): assert responses[code]["description"] assert responses[code]["content"]["application/json"]["schema"] == { "$ref": "#/components/schemas/ApiErrorResponse"} + + +def test_task_guide_download_openapi_lists_exact_supported_original_media_types(): + from app.core.config import Settings + from app.main import create_app + + document = create_app(Settings(environment="test")).openapi() + operation = document["paths"][ + "/api/v1/tasks/{task_id}/guide/documents/{document_id}/content" + ]["get"] + content = operation["responses"]["200"]["content"] + + assert set(content) == { + "application/pdf", + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + "application/vnd.openxmlformats-officedocument.presentationml.presentation", + "text/markdown", + } + assert all( + value["schema"] == {"type": "string", "format": "binary"} + for value in content.values() + ) From 069a1249d2b085e5c802d9f351d82a75187e24c9 Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Fri, 9 Oct 2026 09:32:14 +0100 Subject: [PATCH 09/12] test: stabilize Markdown binary format selectors --- .../changes/pilot15-markdown-guide-documents.md | 11 +++++++---- backend/tests/test_guide_formats.py | 1 + 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/.commitrail/changes/pilot15-markdown-guide-documents.md b/.commitrail/changes/pilot15-markdown-guide-documents.md index f9f12997c..2654ab20c 100644 --- a/.commitrail/changes/pilot15-markdown-guide-documents.md +++ b/.commitrail/changes/pilot15-markdown-guide-documents.md @@ -127,14 +127,17 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. - Final OpenAPI review found the existing task-original GET still advertised only the three predecessor binary formats. Its response metadata and an exact generated-schema regression now include the already-supported Markdown original. +- The binary-substitution regression originally allowed pytest to derive a ZIP + case ID from timestamp-bearing archive bytes. Stable semantic case IDs retain + both assertions while making separately collected exact-node execution reproducible. ## Reconciliation - Current-source reconciliation: Merged main - `36e8a615f01801cecd6ccf83d7411235ab1cfa8f`, where PILOT-13 is delivered as - revision `0026_task_guide_read`. The later PILOT-00 experiment and delivery - changes add no migration, so this branch retains the next linear revision, - `0027_markdown_guide_media`. + `77d6db613ad2734e80106076586052132fa466cb`, where PILOT-13 is delivered as + revision `0026_task_guide_read`. The later PILOT-00 experiment, delivery and + CI-allocation changes add no migration, so this branch retains the next linear + revision, `0027_markdown_guide_media`. - Delivered dependency reconciliation: PILOT-13's exact task-locked authority, concealment, verified streaming and safe CLI publication are unchanged; only its closed media type and extension maps admit Markdown. diff --git a/backend/tests/test_guide_formats.py b/backend/tests/test_guide_formats.py index fa58a5fd3..b4798b38c 100644 --- a/backend/tests/test_guide_formats.py +++ b/backend/tests/test_guide_formats.py @@ -267,6 +267,7 @@ def test_markdown_rejects_non_utf8_and_nul_bytes(detector, payload): @pytest.mark.parametrize( ("payload", "detected"), [(b"%PDF-1.7\n", "pdf"), (_zip({"document.txt": b"guide"}).getvalue(), "zip")], + ids=("pdf", "zip"), ) def test_markdown_declaration_does_not_relabel_known_binary_formats(detector, payload, detected): result = detector.detect(BytesIO(payload), declared_media_type="text/markdown") From eac4b3821fdc8a67804b0007625dbea071a329ab Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Fri, 9 Oct 2026 09:52:47 +0100 Subject: [PATCH 10/12] test: align Markdown CLI upload proof --- .../pilot15-markdown-guide-documents.md | 13 +++-- cli/README.md | 2 +- cli/internal/command/guide_upload.go | 2 +- .../integration/test_guide_upload_http.py | 3 +- .../integration/test_task_guide_public_api.py | 52 +++++++++++++++---- 5 files changed, 54 insertions(+), 18 deletions(-) diff --git a/.commitrail/changes/pilot15-markdown-guide-documents.md b/.commitrail/changes/pilot15-markdown-guide-documents.md index 2654ab20c..454121925 100644 --- a/.commitrail/changes/pilot15-markdown-guide-documents.md +++ b/.commitrail/changes/pilot15-markdown-guide-documents.md @@ -38,10 +38,11 @@ contributor original reads and CLI downloads with the same closed media type. - Review-packet typed metadata/model constraint plus forward-only revision `0027_markdown_guide_media` after delivered `0026_task_guide_read`, its graph/schema consumers and direct PostgreSQL refusal/acceptance proof. -- The Go guide-declaration client and current CLI/API documentation for - `text/markdown`. The merged PILOT-13 task guide DTO/extension/download consumers, - task-read public route metadata, real upload/read fixtures and direct tests are - reconciled without copying its implementation. +- The Go guide-declaration and delivered guide-upload clients/commands plus current + CLI/API documentation for `text/markdown`. The merged PILOT-13 task guide + DTO/extension/download consumers, task-read public route metadata, real + upload/read fixtures and direct tests are reconciled without copying its + implementation. - Existing guide API, real PostgreSQL/MinIO intake fixtures, format, setup workspace, review-packet, CLI declaration and later PILOT-13 contributor-read tests; current README, artifact/guide specifications, roadmap and this record. @@ -130,6 +131,10 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. - The binary-substitution regression originally allowed pytest to derive a ZIP case ID from timestamp-bearing archive bytes. Stable semantic case IDs retain both assertions while making separately collected exact-node execution reproducible. +- Reconciliation with the delivered CLI upload journey found its real-API fixture + selected the new Markdown original while still declaring and sending PDF. The + journey, transport matrix, command help and CLI guide now use the same closed + Markdown media type; backend byte inspection remains authoritative. ## Reconciliation diff --git a/cli/README.md b/cli/README.md index 0f26b293c..ee9e0e338 100644 --- a/cli/README.md +++ b/cli/README.md @@ -161,7 +161,7 @@ workstream project guide upload PROJECT_ID GUIDE_ID DOCUMENT_ID --file Guide.pdf ``` Use the guide and document IDs returned by `project guide create`. Supply the -document's declared media type explicitly: PDF, DOCX or PPTX as listed in +document's declared media type explicitly: PDF, DOCX, PPTX or UTF-8 Markdown as listed in OpenAPI. The CLI sends raw original bytes, not extracted text, JSON or multipart. It requires a nonempty regular file up to ART's 512MiB hard ceiling; Workstream can enforce smaller configured document or aggregate limits. Files are hashed diff --git a/cli/internal/command/guide_upload.go b/cli/internal/command/guide_upload.go index dfb51d8f5..798622876 100644 --- a/cli/internal/command/guide_upload.go +++ b/cli/internal/command/guide_upload.go @@ -42,7 +42,7 @@ func addGuideUpload(guide *cobra.Command, client func() (*api.Client, error), ou }, } upload.Flags().StringVar(&path, "file", "", "Required regular original file; keep its bytes unchanged for manual replay") - upload.Flags().StringVar(&mediaType, "media-type", "", "Required exact declared PDF, DOCX or PPTX media type") + upload.Flags().StringVar(&mediaType, "media-type", "", "Required exact declared PDF, DOCX, PPTX or Markdown media type") upload.Flags().StringVar(&key, "idempotency-key", "", "Required caller-owned UUID; no automatic retry") guide.AddCommand(upload) } diff --git a/cli/tests/integration/test_guide_upload_http.py b/cli/tests/integration/test_guide_upload_http.py index 080d55c6d..80dedb541 100644 --- a/cli/tests/integration/test_guide_upload_http.py +++ b/cli/tests/integration/test_guide_upload_http.py @@ -17,6 +17,7 @@ "application/pdf", "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "application/vnd.openxmlformats-officedocument.presentationml.presentation", + "text/markdown", ) @@ -112,7 +113,7 @@ def test_guide_upload_exact_binary_selectors_receipt_and_safe_text(cli, tmp_path ) assert text.stdout.count("\n") == 1 and "\x1b" not in text.stdout assert all(f'"{field}"' in text.stdout for field in receipt()) - assert len(requests) == 13 # One body per invocation, no authority preflight. + assert len(requests) == 17 # One body per invocation, no authority preflight. def test_guide_upload_bad_local_files_and_selectors_send_nothing(cli, tmp_path): diff --git a/cli/tests/integration/test_task_guide_public_api.py b/cli/tests/integration/test_task_guide_public_api.py index d2a6b56a0..8d6975e35 100644 --- a/cli/tests/integration/test_task_guide_public_api.py +++ b/cli/tests/integration/test_task_guide_public_api.py @@ -3,6 +3,7 @@ import asyncio import hashlib import json +import os from pathlib import Path import sys @@ -38,13 +39,35 @@ @pytest.fixture def task_database_env(base_task_database_env, monkeypatch, tmp_path): - # Backend owner proof uses real MinIO; the independent CLI workflow already - # supplies PostgreSQL and can exercise the same public read with local ART. - (tmp_path / "originals").mkdir(mode=0o700) - values = { - "ARTIFACT_STORE_BACKEND": "local", - "ARTIFACT_LOCAL_ROOT": str(tmp_path / "originals"), - "ARTIFACT_SCRATCH_ROOT": str(tmp_path / "scratch"), + endpoint = os.environ.get("WORKSTREAM_TEST_MINIO_ENDPOINT") + if endpoint: + bucket = os.environ.get("WORKSTREAM_TEST_MINIO_BUCKET") + prefix = os.environ.get("WORKSTREAM_TEST_MINIO_PREFIX") + if not bucket or not prefix: + pytest.fail("MinIO CLI proof requires its owned bucket and prefix") + values = { + "ARTIFACT_STORE_BACKEND": "s3_compatible", + "ARTIFACT_S3_PROVIDER_PROFILE": "minio", + "ARTIFACT_S3_REGION": "us-east-1", + "ARTIFACT_S3_BUCKET": bucket, + "ARTIFACT_S3_ENDPOINT_URL": endpoint, + "ARTIFACT_S3_PRIVATE_PREFIX": f"{prefix}/cli-guide-upload", + "ARTIFACT_S3_ADDRESSING_STYLE": "path", + "ARTIFACT_S3_CREDENTIAL_MODE": "local_static", + "ARTIFACT_S3_ACCESS_KEY_ID": "workstream-minio", + "ARTIFACT_S3_SECRET_ACCESS_KEY": "workstream-minio-secret-key", + "ARTIFACT_SCRATCH_ROOT": str(tmp_path / "scratch"), + } + else: + # The independent hosted CLI workflow owns PostgreSQL and uses local ART; + # the backend lanes separately retain provider custody evidence. + (tmp_path / "originals").mkdir(mode=0o700) + values = { + "ARTIFACT_STORE_BACKEND": "local", + "ARTIFACT_LOCAL_ROOT": str(tmp_path / "originals"), + "ARTIFACT_SCRATCH_ROOT": str(tmp_path / "scratch"), + } + values |= { "CELERY_BROKER_URL": "memory://", "CELERY_TASK_ALWAYS_EAGER": "false", } @@ -82,7 +105,7 @@ async def exercise_original_upload(cli, origin, world, tmp_path, monkeypatch): {"content": "Evaluate the evidence against the guide."} ], "documents": [ - {"label": "Instructions.pdf", "media_type": "application/pdf"} + {"label": "Instructions.md", "media_type": "text/markdown"} ], } ) @@ -109,11 +132,11 @@ async def exercise_original_upload(cli, origin, world, tmp_path, monkeypatch): ) document = guide["documents"][0]["document_id"] original = world.originals[0] - path = tmp_path / "original.pdf" + path = tmp_path / "original.md" path.write_bytes(original) key = str(new_record_id()) - async def upload(*, selectors=None, media="application/pdf"): + async def upload(*, selectors=None, media="text/markdown", upload_key=key): return await asyncio.to_thread( cli, origin, @@ -127,7 +150,7 @@ async def upload(*, selectors=None, media="application/pdf"): "--media-type", media, "--idempotency-key", - key, + upload_key, "-o", "json", ) @@ -136,6 +159,13 @@ async def upload(*, selectors=None, media="application/pdf"): selectors = tuple( value.replace("-", "") for value in (world.project["id"], guide["id"], document) ) + wrong_media = await upload( + selectors=selectors, + media="application/pdf", + upload_key=str(new_record_id()), + ) + assert wrong_media.returncode == 1 and wrong_media.stdout == "" + assert json.loads(wrong_media.stderr)["error"]["status"] == 422 stored = await upload(selectors=selectors) assert stored.returncode == 0, stored.stderr receipt = json.loads(stored.stdout) From 6256f239d2b613e790d24e58eac9cc62b9df675e Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Fri, 9 Oct 2026 10:38:40 +0100 Subject: [PATCH 11/12] ci: restore project lane headroom --- .../pilot15-markdown-guide-documents.md | 45 ++++++++++++++++++- backend/scripts/run_test_lanes.py | 4 +- backend/scripts/test_lane_catalogue.py | 15 ++++++- backend/tests/test_ci_lane_catalogue.py | 41 +++++++++++++---- backend/tests/test_ci_test_lanes.py | 8 ++-- docs/operations_backend_testing.md | 6 +++ 6 files changed, 101 insertions(+), 18 deletions(-) diff --git a/.commitrail/changes/pilot15-markdown-guide-documents.md b/.commitrail/changes/pilot15-markdown-guide-documents.md index 454121925..d25708aa8 100644 --- a/.commitrail/changes/pilot15-markdown-guide-documents.md +++ b/.commitrail/changes/pilot15-markdown-guide-documents.md @@ -46,6 +46,10 @@ contributor original reads and CLI downloads with the same closed media type. - Existing guide API, real PostgreSQL/MinIO intake fixtures, format, setup workspace, review-packet, CLI declaration and later PILOT-13 contributor-read tests; current README, artifact/guide specifications, roadmap and this record. +- `backend/scripts/run_test_lanes.py`, `backend/scripts/test_lane_catalogue.py`, + their focused CI regressions and the canonical backend-testing operations + guide: repair the measured hosted project-B timeout without changing product + tests, services, lane count or the 1200-second execution cap. ### Not allowed @@ -59,6 +63,11 @@ contributor original reads and CLI downloads with the same closed media type. - No duplicate contributor read, task authority, streaming or CLI download implementation. No compatibility variant, data rewrite, downgrade, deployment, merge or issue closure. +- Within the CI headroom repair, no product-test removal or change, skip, + deselection, lane name/count, service, timeout, runner permission, retry, + coverage, evidence format, UUID seed, generic hash algorithm or + aggregate-authority change. Only the named focused CI regression nodes may be + added. ## Design and decisions @@ -71,6 +80,24 @@ the matching additive value in the next linear migration. The setup agent stages the verified original through its current opaque handle and network-disabled workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. +### Hosted CI headroom repair + +Backend run `37908682835` completed all 766 project-B nodes but reached the +unchanged runner deadline while emitting an unbounded duration table. Retained +A/B/C phase timings were about 549/1169/948 seconds. The broad +`tests/test_projects.py` and `tests/test_guide_document_intake.py` modules now +use the same exact-node SHA-256 assignment across project A and C, excluding B. +An exact 8,812-node replay changed only 222 assignments in those modules, +preserved every node ID and execution kind, and produced project counts +768/643/743 with a 155/149 A/C subgroup. The P15 observed-overhead forecast is +about 694/1044/1029 seconds; the concurrent P14 forecast is about +1011/974/837. These are qualified forecasts subject to hosted variance. + +Ordinary and admin commands retain the 25 slowest pytest phases. This bounds +diagnostic output after completion; the measured A/C assignment supplies the +material headroom. The nine lanes, 1200-second cap, services, UUID/hash +mechanisms, evidence and fail-closed aggregate remain unchanged. + ## Acceptance criteria - [x] Public API and Go CLI declaration accept exactly `text/markdown`; HTML, @@ -88,12 +115,19 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. remains covered and unchanged; no second migration head is introduced. - [x] Current API/CLI, artifact/guide specifications and roadmap state the delivered boundary without claiming HTML, remote sites or deployed capability. +- [x] Both measured project modules partition only across A/C; the retained + hosted manifest preserves exact IDs/kinds, moves only those modules and leaves + no selected node on project B. +- [x] Current recursive collection covers 8,814 nodes exactly once across nine + lanes; the two added nodes are focused CI ownership/partition regressions. +- [x] Ordinary/admin lane commands use `--durations=25`; predecessor three-way + assignment and unbounded-duration mutations fail their exact regressions. ## Risk and review routing - Risk class: L1 - Required reviewers: architecture, reuse, security, qa, test_delta, - documentation, product_ops + documentation, product_ops, ci_integrity - Human review focus: Strict media classification without byte rewriting; unchanged ART/authority/isolation boundaries; one linear migration after PILOT-13; no duplicate contributor-read implementation; honest format limits. @@ -107,6 +141,8 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. | Setup-agent exact read | Workspace/runtime batch with Markdown version, byte-bearing grant and disabled network | Exact `.md` bytes, media type, run handle and cleanup passed | No live model inference is required for byte-access proof | | Persisted review metadata | PostgreSQL 16 predecessor/head direct-write and schema-fingerprint proofs | Predecessor rejected Markdown; 0027 admitted it past the media check; HTML remained rejected; graph and regenerated fingerprint passed | Existing rows are retained; no downgrade is supported | | Contributor REST/CLI | Real PostgreSQL/MinIO locked-read batch and built-CLI HTTP/public-API journeys | Six backend cases, 34 HTTP cases and the real built-CLI API journey passed, including concealment, successor locking, corrupt originals and `.md` publication | Deployment remains outside this change | +| Hosted timeout diagnosis | Backend run `37908682835`, lane artifact `11606836106` and aggregate artifact `11606916469` | Project B recorded 766/766 full-lifecycle nodes, zero skip/deselect, 1200.943s interruption and complete PostgreSQL/MinIO cleanup | Fresh hosted timing remains required | +| CI assignment and command repair | 8,812-node manifest replay; current recursive collection; focused catalogue/runner tests and predecessor mutations | IDs/kinds unchanged; 222 allowed assignment changes; current 8,814 nodes collected; 62 focused and 86 gate/record tests passed; both mutants failed | Forecasts do not replace fresh hosted aggregate proof | ## Review findings @@ -135,6 +171,10 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. selected the new Markdown original while still declaring and sending PDF. The journey, transport matrix, command help and CLI guide now use the same closed Markdown media type; backend byte inspection remains authoritative. +- The first hosted repair completed every project-B node but lacked deadline + headroom. An initial exclusive-A plan was rejected before commit when concurrent + P14 timings forecast it too close to the cap; exact replays of both current + manifests selected the bounded A/C subgroup instead. ## Reconciliation @@ -142,7 +182,8 @@ workspace. Contributor REST/CLI support is reconciled only from merged PILOT-13. `77d6db613ad2734e80106076586052132fa466cb`, where PILOT-13 is delivered as revision `0026_task_guide_read`. The later PILOT-00 experiment, delivery and CI-allocation changes add no migration, so this branch retains the next linear - revision, `0027_markdown_guide_media`. + revision, `0027_markdown_guide_media`. The measured CI repair supersedes only + the candidate tuple for two project modules and unbounded duration diagnostics. - Delivered dependency reconciliation: PILOT-13's exact task-locked authority, concealment, verified streaming and safe CLI publication are unchanged; only its closed media type and extension maps admit Markdown. diff --git a/backend/scripts/run_test_lanes.py b/backend/scripts/run_test_lanes.py index 79b394e1b..644416e2a 100644 --- a/backend/scripts/run_test_lanes.py +++ b/backend/scripts/run_test_lanes.py @@ -496,7 +496,7 @@ def lane_command( *_plugin_args(), "--cov=app", "--cov-report=", - "--durations=0", + "--durations=25", *nodes, ] return [ @@ -525,7 +525,7 @@ def admin_runner_command(nodes: list[str]) -> list[str]: *_plugin_args(), "--cov=app", "--cov-report=", - "--durations=0", + "--durations=25", *nodes, ] return [sys.executable, "-c", ADMIN_REDACTING_WRAPPER, *pytest_command] diff --git a/backend/scripts/test_lane_catalogue.py b/backend/scripts/test_lane_catalogue.py index 5bf65776f..ddbf659bb 100644 --- a/backend/scripts/test_lane_catalogue.py +++ b/backend/scripts/test_lane_catalogue.py @@ -10,6 +10,7 @@ "project_lifecycle_b", "project_lifecycle_c", ) +PARTITIONED_PROJECT_AC_LANES = ("project_lifecycle_a", "project_lifecycle_c") PARTITIONED_TASK_LANES = ("task_lifecycle_a", "task_lifecycle_b", "task_lifecycle_c") @@ -295,7 +296,6 @@ class TestLane: "tests/projects/guide_compilation/finalization/test_authorization_concurrency_postgresql.py", "tests/projects/guide_compilation/finalization/test_authorization_postgresql.py", "tests/projects/test_active_guide_repository.py", - "tests/test_guide_document_intake.py", "tests/projects/guide_compilation/finalization/test_concurrency_postgresql.py", "tests/projects/guide_compilation/finalization/test_contracts.py", "tests/projects/guide_compilation/finalization/test_guards_postgresql.py", @@ -391,6 +391,13 @@ class TestLane: "tests/projects/sufficiency_mutations/test_replay_repository.py", "tests/projects/sufficiency_mutations/test_public_routes.py", "tests/projects/test_retired_submission_derivation_route.py", +) + +# These broad PROJECT owners exhausted project B's execution budget. Retained +# hosted timings keep their deterministic node partition on project A and C, +# excluding B while preserving headroom across both measured current trees. +PROJECT_AC_PARTITION_MODULES = ( + "tests/test_guide_document_intake.py", "tests/test_projects.py", ) @@ -524,6 +531,7 @@ class TestLane: PARTITIONED_PROJECT_LANES, PROJECT_MODULES + CHECKER_DELIVERY_MODULES + ROUTING_AUTH_PREPARATION_MODULES, ), + (PARTITIONED_PROJECT_AC_LANES, PROJECT_AC_PARTITION_MODULES), (PARTITIONED_TASK_LANES, TASK_MODULES), ) PARTITION_LANES_BY_MODULE = { @@ -565,7 +573,10 @@ class TestLane: *( TestLane( name, - PROJECT_MODULES + CHECKER_DELIVERY_MODULES + ROUTING_AUTH_PREPARATION_MODULES, + PROJECT_MODULES + + CHECKER_DELIVERY_MODULES + + ROUTING_AUTH_PREPARATION_MODULES + + (PROJECT_AC_PARTITION_MODULES if name in PARTITIONED_PROJECT_AC_LANES else ()), ) for name in PARTITIONED_PROJECT_LANES ), diff --git a/backend/tests/test_ci_lane_catalogue.py b/backend/tests/test_ci_lane_catalogue.py index 1701603b1..3b4551e52 100644 --- a/backend/tests/test_ci_lane_catalogue.py +++ b/backend/tests/test_ci_lane_catalogue.py @@ -83,15 +83,36 @@ def test_checker_delivery_and_routing_preparation_partition_every_node_once() -> ) +def test_measured_project_modules_partition_between_project_a_and_c() -> None: + expected = { + "tests/test_guide_document_intake.py", + "tests/test_projects.py", + } + assert set(catalogue.PROJECT_AC_PARTITION_MODULES) == expected + assert not expected & set(catalogue.PROJECT_MODULES) + assert all( + catalogue.PARTITION_LANES_BY_MODULE[module] == catalogue.PARTITIONED_PROJECT_AC_LANES + for module in expected + ) + + for module in expected: + assert tuple(lane.name for lane in LANES if module in lane.modules) == ( + "project_lifecycle_a", + "project_lifecycle_c", + ) + + nodes = [f"{module}::test_owned[{index}]" for module in expected for index in range(256)] + manifest = runner.build_manifest("a" * 40, list(reversed(nodes))) + assert {row["nodeid"] for row in manifest["nodes"]} == set(nodes) + assert {row["lane"] for row in manifest["nodes"]} == set(catalogue.PARTITIONED_PROJECT_AC_LANES) + + def test_measured_hotspots_have_explicit_semantic_owners() -> None: """Keep lane balance tied to subsystem ownership and measured schema cost.""" modules_by_lane = {lane.name: set(lane.modules) for lane in LANES} - assert ( - modules_by_lane["project_lifecycle_a"] - == modules_by_lane["project_lifecycle_b"] - == modules_by_lane["project_lifecycle_c"] - == set(catalogue.CHECKER_DELIVERY_MODULES) + partitioned_project_modules = ( + set(catalogue.CHECKER_DELIVERY_MODULES) | set(catalogue.ROUTING_AUTH_PREPARATION_MODULES) | { "tests/tasks/evaluation_delivery/test_custody.py", @@ -128,7 +149,6 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: "tests/projects/guide_activation/test_rejections.py", "tests/projects/guide_activation/test_successor.py", "tests/projects/test_active_guide_repository.py", - "tests/test_guide_document_intake.py", "tests/projects/guide_compilation/test_capability_growth.py", "tests/projects/guide_compilation/test_capability_growth_postgresql.py", "tests/projects/guide_compilation/test_compilation_storage_limit.py", @@ -230,9 +250,12 @@ def test_measured_hotspots_have_explicit_semantic_owners() -> None: "tests/projects/submission_policy_mutations/test_public_routes_postgresql.py", "tests/projects/test_retired_submission_derivation_route.py", "tests/test_api_drill_repairs.py", - "tests/test_projects.py", } ) + assert modules_by_lane["project_lifecycle_b"] == partitioned_project_modules + project_ac_modules = partitioned_project_modules | set(catalogue.PROJECT_AC_PARTITION_MODULES) + assert modules_by_lane["project_lifecycle_a"] == project_ac_modules + assert modules_by_lane["project_lifecycle_c"] == project_ac_modules assert ( modules_by_lane["task_lifecycle_a"] == modules_by_lane["task_lifecycle_b"] @@ -495,7 +518,9 @@ def test_schema_nodes_share_one_lane() -> None: @pytest.mark.parametrize( - ("names", "modules"), catalogue.PARTITION_GROUPS, ids=("shared", "project", "task") + ("names", "modules"), + catalogue.PARTITION_GROUPS, + ids=("shared", "project", "project-ac", "task"), ) def test_owner_nodes_partition_deterministically(names, modules) -> None: module = modules[0] diff --git a/backend/tests/test_ci_test_lanes.py b/backend/tests/test_ci_test_lanes.py index 65cfb1e3b..3a580f453 100644 --- a/backend/tests/test_ci_test_lanes.py +++ b/backend/tests/test_ci_test_lanes.py @@ -98,12 +98,12 @@ def test_lane_command_uses_exact_nodes_and_isolation_contract(tmp_path: Path) -> assert command[-1] == nodes[0] assert "--cov=app" in command assert "--cov-report=" in command - assert "--durations=0" in command - assert "--durations=25" not in command + assert command.count("--durations=25") == 1 + assert "--durations=0" not in command admin = runner.admin_runner_command([f"{runner.ADMIN_RUNNER_MODULE}::test_one"]) - assert "--durations=0" in admin - assert "--durations=25" not in admin + assert admin.count("--durations=25") == 1 + assert "--durations=0" not in admin def test_lane_environment_uses_private_evidence_and_coverage(tmp_path: Path) -> None: diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index e4681bfe5..a460c5534 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -141,6 +141,12 @@ partition PROJECT nodes, checker execution, materialization, evaluation capacity output custody and routing AUTH preparation beside initial dispatch/delivery. All use the existing exact-node hash across the three lanes: placing the entire checker delivery group exclusively on project A exhausted its execution cap. +Two measured broad owners, `tests/test_projects.py` and +`tests/test_guide_document_intake.py`, instead partition their nodes only across +project A and C. Retained runs measured capacity across that pair and a project B +timeout when the modules used all three lanes. The lane logs retain the 25 slowest +pytest phases for diagnosis without letting unbounded diagnostic output consume +the execution deadline after all nodes finish. `task_lifecycle_a`, `task_lifecycle_b`, and `task_lifecycle_c` use the same deterministic partition mechanism for the remaining TASK and checker nodes. The single `schema_contracts` lane owns all baseline/PostgreSQL schema, reset and isolated-runner contracts. The From 61cf1326e99a1da780a7a383b345b9a92fadeeee Mon Sep 17 00:00:00 2001 From: Commitrail Probe Date: Fri, 9 Oct 2026 13:39:49 +0100 Subject: [PATCH 12/12] docs: distinguish pending Markdown support from merged capabilities --- docs/roadmap_status.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/docs/roadmap_status.md b/docs/roadmap_status.md index 1be0ce5d3..1a189d805 100644 --- a/docs/roadmap_status.md +++ b/docs/roadmap_status.md @@ -105,8 +105,7 @@ No handle, allow, receipt, routing outcome or acceptance is published. Exact assignment-invalidation authority, atomic originating publication and production delivery with enforced prefork topology are implemented. Manager task create/screen/release now use exact project authority, atomic evidence and durable -replay. ARCH-03C4 adds the three exact-authorized public task queues. PILOT-14 -remains open pending human merge; its intended outcome adds locked +replay. ARCH-03C4 adds the three exact-authorized public task queues. PILOT-14 adds locked ContributionPolicyVersion compensation to ready rows and Contributor detail for active exact-project Submitters or Reviewers, without Finance internals or claim authority. ARCH-03C5 adds exact-authorized Contributor/Manager @@ -223,7 +222,7 @@ cannot be reused as post-submission review-gate evidence. See the | Project Guide source custody | **Live foundation** | Guide creation declares documents and task examples; public document upload; immutable internal metadata snapshots; exact run-scoped setup reads and assigned-task locked-original reads; S3-backed originals and isolated agent document inspection; exact document generation through separate manager approvals and public exact-authorized CP07 activation | Public activation-to-intake drill after evaluation/remediation prerequisites; prove each enabled document reader | | Unified Project Guide compilation | **Live setup and manager proposal operations** | Committed original-document readiness dispatches one immutable attempt through Celery; complete result and crash/recovery custody; distinct pre/post proposals; deterministic sufficiency and submission-artifact-policy projections; immutable authorized setup finalization; public exact manager review, pre-submit approval and manual correction dispatch; automatic deterministic post-policy derivation, public complete policy read, separate approval and shared correction custody | Public intake cutover after evaluation/remediation prerequisites | | Contribution policy administration | **Public Finance policy workflow; binding administration internal** | Internal Finance Authority adapter-binding lifecycle; public ContributionPolicy discovery/read/create/update/publish/retire with exact Finance Authority and recoverable draft selectors; immutable operation and event history; internal exact selected-version validation; CP07 binding with live exact-project manager authority of the selected published version to the active guide generation | Public intake cutover after evaluation/remediation prerequisites | -| Task readiness and claim | **Foundation with grant-backed manager and contributor commands** | Task records, assignments and locked work context; guide-bound ContributionPolicyVersion locked before `READY` and copied to TaskAssignment; detached project/guide display; public project-scoped ready, management and operational queues with distinct current grant authority and signed bounded live pagination; public contributor/management detail with exact project and assignment authority; separate current live work-context projections with exact receipt-selected review/revision/ContributionPolicy identities and no obsolete economic fields; explicit management/operational/audit locked-context projections using one historical resolver, with separate public manager, system-Operator and Audit Authority access; immutable contributor/management requirements using one historical translator, with public exact Contributor and Manager authority; bounded public Audit Authority lifecycle evidence with atomic project/task scoping and exact transition references; public manager create/screen/release use exact covered Project Manager authority, atomic audit and replay; claim/start/contributor context use exact-project Submitter grants; assigned context lists documents only and distinct `task.guide.read` verifies locked originals before delivery; separate manager context and system-Operator start; durable claim/start retry receipts with fresh authority and exact assignment checks; hidden exact-assignment invalidation with committed cause verification, delivery fencing, exact fixed-service authority and decision-bound immutable release evidence; atomic authority-loss publication and registered prefork delivery | PILOT-14 remains open pending human merge: add both locked contribution types to ready rows and Contributor detail for exact-project Submitters or Reviewers as unpaid or exact instrument/unit/quantity awards, without Finance binding facts. Public intake cutover follows evaluation/remediation prerequisites | +| Task readiness and claim | **Foundation with grant-backed manager and contributor commands** | Task records, assignments and locked work context; guide-bound ContributionPolicyVersion locked before `READY` and copied to TaskAssignment; detached project/guide display; public project-scoped ready, management and operational queues with distinct current grant authority and signed bounded live pagination; public contributor/management detail with exact project and assignment authority; separate current live work-context projections with exact receipt-selected review/revision/ContributionPolicy identities and no obsolete economic fields; explicit management/operational/audit locked-context projections using one historical resolver, with separate public manager, system-Operator and Audit Authority access; immutable contributor/management requirements using one historical translator, with public exact Contributor and Manager authority; bounded public Audit Authority lifecycle evidence with atomic project/task scoping and exact transition references; public manager create/screen/release use exact covered Project Manager authority, atomic audit and replay; claim/start/contributor context use exact-project Submitter grants; assigned context lists documents only and distinct `task.guide.read` verifies locked originals before delivery; separate manager context and system-Operator start; durable claim/start retry receipts with fresh authority and exact assignment checks; hidden exact-assignment invalidation with committed cause verification, delivery fencing, exact fixed-service authority and decision-bound immutable release evidence; atomic authority-loss publication and registered prefork delivery; PILOT-14 exposes both locked contribution types in ready rows and Contributor detail for exact-project Submitters or Reviewers as unpaid or exact instrument/unit/quantity awards, without Finance binding facts | Public intake cutover follows evaluation/remediation prerequisites | | Contributor artifact preparation | **Hidden and proven** | One outer ZIP; bounded scratch inspection; canonical manifest; shared evaluation-content capacity and exact locked-policy checks before attempts or durable intent; platform and project prechecks; unchanged-work rejection; durable put intent; verification; capacity-charged ready admission; hidden final handoff validates the exact activated historical guide through owner ports | Complete the later public admission-only cutover | | Pre-submission intake checking | **Hidden with approved-guide lineage** | Separate versioned pre-submission catalogue, locked effective-plan compilation, platform/project checks during continuous preparation, blocking feedback before Submission creation, and one internal phase command covering execution/replay with the JSON precheck removed; ARCH-03D connects approved-guide lineage through the final durable handoff | Complete the canonical public cutover after evaluation/remediation prerequisites; passing intake must never substitute for post-submit evaluation | | Immutable Submission creation | **Hidden foundation; public packet creation retired** | Contributor preparation authority; durable pre-submit reservation and exact completed-evidence recovery without rerunning checks; atomic admission consumption; TASK-owned admission-backed creation with exact assignment ContributionPolicyVersion, locked policy lineage and service/database checked-packet custody; fixed-service artifact binding; replay/concurrency/rollback proof | Finish downstream evaluation and the canonical public integration. The retained submission-list GET is not a usable creation POST | @@ -338,8 +337,7 @@ cannot be reused as post-submission review-gate evidence. See the [CLI contributor discovery](../.commitrail/initiatives/WS-CLI-001/WS-CLI-001-05.md) adds ready-task pages and contributor instructions through public reads, preserving exact-project contributor scope and assignment visibility without - claiming work. PILOT-14 remains open pending human merge; its intended outcome - displays the task-locked ContributionPolicyVersion UUID and both unpaid or + claiming work. PILOT-14 displays the task-locked ContributionPolicyVersion UUID and both unpaid or exact award terms in ready rows and task detail to a Submitter or Reviewer, while claim/start remain Submitter-only. [CLI contributor claim/start](../.commitrail/initiatives/WS-CLI-001/WS-CLI-001-06.md) @@ -447,8 +445,8 @@ cannot be reused as post-submission review-gate evidence. See the response supplies document IDs for the public binary upload route; no separate source-snapshot creation call remains. A starting idea is sufficient and optional example fields do not repeat requirements from the guide. Each snapshot/run - binds the exact version's examples. Upload admission accepts PDF, DOCX, PPTX - and byte-preserved UTF-8 Markdown (`.md`) and checks bounded format, digest and size. + binds the exact version's examples. Upload admission accepts PDF, DOCX and PPTX + and checks bounded format, digest and size. Committed originals do not bypass the separate verification required for submission ZIPs. Embedded document bodies and extractors are removed. - Committed original-document readiness automatically runs one unified compilation @@ -553,6 +551,12 @@ Only open pull requests describe transient work. Use the repository's [open pull-request view](https://github.com/Flow-Research/workstream/pulls) to see whether any item below is already under review. +[PILOT-15 / PR #514](https://github.com/Flow-Research/workstream/pull/514) +proposes byte-preserved UTF-8 Markdown (`.md`) guide admission, setup-agent reads +and task-locked contributor REST/CLI reads. This remains open work until its +code, migration, tests and review evidence are merged; current `main` supports +PDF, DOCX and PPTX originals. + Delivered capabilities are summarized in the [scoreboard](#end-to-end-lifecycle-scoreboard); their evidence is linked under [completed work](#what-has-been-completed). The [dependency and ownership plan](../.commitrail/initiatives/WS-ARCH-001/planning/PLAN.md#current-dependency-contract) @@ -896,8 +900,7 @@ remaining trace sequence is: Shared dispatch authority is delivered by AUTH-OUTBOX-02. Live assignment invalidation has atomic producer fan-out and registered delivery in ARCH-03C2. Manager readiness commands are public with exact authority in ARCH-03C3; - ARCH-03C4 exposes ready, management and Operator-status queues. PILOT-14 - remains open pending human merge; its intended outcome adds contributor-safe + ARCH-03C4 exposes ready, management and Operator-status queues. PILOT-14 adds contributor-safe locked compensation to ready/detail without restoring removed task payment fields or exposing Finance bindings. ARCH-03C5 exposes task detail and requirements; ARCH-03C6 exposes distinct