Harden Stage 0 canonical manifest acceptance

.github/workflows/stage0-manifest.yml

@@ -0,0 +1,33 @@
+name: Stage 0 Manifest Acceptance
+
+on:
+  workflow_dispatch:
+  pull_request:
+    paths:
+      - 'src/stage0_manifest.py'
+      - 'tests/test_stage0_manifest.py'
+      - '.github/workflows/stage0-manifest.yml'
+  push:
+    branches:
+      - stage0-manifest-hardening
+
+permissions:
+  contents: read
+
+jobs:
+  stage0-acceptance:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install test dependencies
+        run: python -m pip install pytest
+
+      - name: Execute Stage 0 adversarial acceptance suite
+        run: python -m pytest -q tests/test_stage0_manifest.py

.github/workflows/stage1-evidence.yml

@@ -0,0 +1,33 @@
+name: Stage 1 Evidence Boundary
+
+on:
+  pull_request:
+    paths:
+      - 'src/stage0_manifest.py'
+      - 'src/stage1_evidence.py'
+      - 'src/stage1_cli.py'
+      - 'tests/test_stage0_manifest.py'
+      - 'tests/test_stage1_evidence.py'
+      - 'requirements-dev.txt'
+      - '.github/workflows/stage1-evidence.yml'
+  push:
+    branches:
+      - stage1-evidence-boundary
+      - main
+
+permissions:
+  contents: read
+
+jobs:
+  stage1:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: pip
+      - name: Install dependencies
+        run: python -m pip install -r requirements-dev.txt
+      - name: Run Stage 0 and Stage 1 regression suites
+        run: python -m pytest -q tests/test_stage0_manifest.py tests/test_stage1_evidence.py

docs/STAGE1_EVIDENCE_CONTRACT.md

@@ -0,0 +1,29 @@
+# Stage 1 Evidence Construction Contract
+
+Stage 1 is a deterministic evidence-construction boundary. It extracts and describes source evidence but does not determine whether that evidence constitutes a vulnerability.
+
+## Boundary
+
+Stage 0 manifest -> verify canonical payload and `manifest_sha256` -> verify commit/object bindings -> deterministic router -> AST extraction for `PRIMARY` and `DELETED` -> bounded evidence construction -> canonical Evidence Manifest -> `stage1_sha256`.
+
+`SECONDARY`, `DEPENDENCY`, and `NON_CODE` records do not enter AST evidence construction.
+
+## Authority
+
+The exact Git object referenced by the verified Stage 0 record is authoritative. The working tree is an execution environment only. Evidence content is materialized from the immutable Git object after binding verification.
+
+## Evidence identity
+
+Evidence IDs are deterministic and assigned after canonical ordering by path, byte range, node type, and evidence type. Each evidence block binds to commit, Git object ID, SHA-256 content hash, and an exact byte range.
+
+## Bounds
+
+Evidence construction has explicit limits for block count, block size, referenced nodes, and total payload bytes. Exceeding a limit produces an explicit incomplete state; content is never silently truncated and treated as complete evidence.
+
+## Security boundary
+
+Structural labels such as `CALL`, `CONDITION`, `RETURN`, and `AUTHORIZATION_CHECK` describe extracted evidence. They are not vulnerability conclusions. Stage 2 is the first layer permitted to reason about claims supported by the evidence.
+
+## Reproducibility
+
+Identical Stage 0 input, Git objects, parser metadata, and Stage 1 ruleset must produce byte-identical evidence payloads and `stage1_sha256`. Parser or ruleset changes are expected to change the committed artifact.

docs/STAGE1_GATE.md

@@ -0,0 +1 @@
+Stage 1 gate: CI must pass before Stage 2. No LLM integration is part of this boundary.

docs/STAGE1_IMPLEMENTATION.md

@@ -0,0 +1 @@
+Implementation status is CI-gated.

docs/STAGE1_NEXT.md

@@ -0,0 +1 @@
+Stage 1 implementation is prepared on the Stage 0 hardening line. The acceptance gate remains CI-bound; no Stage 2 integration is authorized until the deterministic evidence boundary passes.

docs/STAGE1_READY.md

@@ -0,0 +1 @@
+Stage 1 deterministic evidence boundary implementation line.

docs/STAGE1_STATUS.md

@@ -0,0 +1,7 @@
+# Stage 1 Status
+
+Stage 1 remains gated on the hardened Stage 0 contract. This branch contains the deterministic evidence-construction implementation and regression suite, but Stage 1 is not accepted until CI verifies the complete boundary.
+
+Acceptance state: PENDING CI
+
+Stage 2 LLM integration: BLOCKED

requirements-dev.txt

@@ -1,3 +1,5 @@
 pytest>=8.4,<9.0
 httpx>=0.28,<1.0
+tree-sitter>=0.25,<0.26
+tree-sitter-rust>=0.24,<0.25
 -r services/api/requirements.txt

src/stage0_manifest.py

@@ -0,0 +1,179 @@
+"""Stage 0 canonical VCS change manifest primitives."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from dataclasses import dataclass
+from enum import Enum
+from pathlib import Path, PurePosixPath
+from typing import Iterable, Mapping
+
+SCHEMA_VERSION = "1"
+RULESET_VERSION = "stage0-v1"
+
+
+class Disposition(str, Enum):
+    PRIMARY = "PRIMARY"
+    SECONDARY = "SECONDARY"
+    DEPENDENCY = "DEPENDENCY"
+    GENERATED = "GENERATED"
+    OVERSIZED = "OVERSIZED"
+    NON_CODE = "NON_CODE"
+    DELETED = "DELETED"
+    REJECTED = "REJECTED"
+
+
+class Reason(str, Enum):
+    CODE_CHANGE = "code_change"
+    MISLEADING_EXTENSION = "misleading_extension"
+    DEPENDENCY = "dependency"
+    GENERATED_ARTIFACT = "generated_artifact"
+    OVERSIZED_ARTIFACT = "oversized_artifact"
+    NON_CODE = "non_code"
+    DELETED = "deleted"
+    PATH_TRAVERSAL_ATTEMPT = "path_traversal_attempt"
+    VCS_OBJECT_MISMATCH = "vcs_object_mismatch"
+    VCS_OBJECT_MISSING = "vcs_object_missing"
+    INVALID_PATH = "invalid_path"
+    BINARY = "binary"
+
+
+@dataclass(frozen=True)
+class ChangeRecord:
+    path: str
+    change_type: str
+    disposition: Disposition
+    reason: Reason
+    size_bytes: int
+    is_binary: bool
+    blob_object: str | None
+    content_sha256: str | None
+    rename_group_id: str | None = None
+
+    def canonical(self) -> dict[str, object]:
+        return {
+            "path": self.path,
+            "change_type": self.change_type,
+            "disposition": self.disposition.value,
+            "reason": self.reason.value,
+            "size_bytes": self.size_bytes,
+            "is_binary": self.is_binary,
+            "blob_object": self.blob_object,
+            "content_sha256": self.content_sha256,
+            "rename_group_id": self.rename_group_id,
+        }
+
+
+@dataclass(frozen=True)
+class Manifest:
+    schema_version: str
+    ruleset_version: str
+    repository: str
+    base_commit: str
+    head_commit: str
+    pr_number: int
+    files: tuple[ChangeRecord, ...]
+    manifest_sha256: str
+
+    def payload(self) -> dict[str, object]:
+        return {
+            "schema_version": self.schema_version,
+            "ruleset_version": self.ruleset_version,
+            "repository": self.repository,
+            "base_commit": self.base_commit,
+            "head_commit": self.head_commit,
+            "pr_number": self.pr_number,
+            "files": [record.canonical() for record in self.files],
+        }
+
+    def canonical_json(self) -> bytes:
+        return canonical_json(self.payload())
+
+
+def canonical_json(payload: Mapping[str, object]) -> bytes:
+    return json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode("utf-8")
+
+
+def sha256_hex(data: bytes) -> str:
+    return hashlib.sha256(data).hexdigest()
+
+
+def rename_group_id(base_commit: str, head_commit: str, source_path: str, destination_path: str) -> str:
+    payload = {
+        "base_commit": base_commit,
+        "head_commit": head_commit,
+        "source_path": source_path,
+        "destination_path": destination_path,
+    }
+    return sha256_hex(canonical_json(payload))
+
+
+def freeze_manifest(*, repository: str, base_commit: str, head_commit: str, pr_number: int, files: Iterable[ChangeRecord]) -> Manifest:
+    ordered = tuple(sorted(files, key=lambda record: (record.path, record.change_type, record.rename_group_id or "")))
+    payload = {
+        "schema_version": SCHEMA_VERSION,
+        "ruleset_version": RULESET_VERSION,
+        "repository": repository,
+        "base_commit": base_commit,
+        "head_commit": head_commit,
+        "pr_number": pr_number,
+        "files": [record.canonical() for record in ordered],
+    }
+    digest = sha256_hex(canonical_json(payload))
+    return Manifest(SCHEMA_VERSION, RULESET_VERSION, repository, base_commit, head_commit, pr_number, ordered, digest)
+
+
+def safe_join(resolved_root: Path, candidate: str) -> Path:
+    root = resolved_root.resolve(strict=True)
+    candidate_path = (root / candidate).resolve(strict=False)
+    try:
+        candidate_path.relative_to(root)
+    except ValueError as exc:
+        raise ValueError("path_traversal_attempt") from exc
+    return candidate_path
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as stream:
+        for chunk in iter(lambda: stream.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def is_binary_bytes(content: bytes) -> bool:
+    try:
+        content.decode("utf-8")
+    except UnicodeDecodeError:
+        return True
+    return b"\x00" in content
+
+
+def classify_path(path: str, content: bytes, *, generated: bool = False, dependency: bool = False, oversized: bool = False) -> tuple[Disposition, Reason]:
+    if oversized:
+        return Disposition.OVERSIZED, Reason.OVERSIZED_ARTIFACT
+    if generated:
+        return Disposition.GENERATED, Reason.GENERATED_ARTIFACT
+    if dependency:
+        return Disposition.DEPENDENCY, Reason.DEPENDENCY
+    if is_binary_bytes(content):
+        return Disposition.NON_CODE, Reason.BINARY
+    suffix = PurePosixPath(path).suffix.lower()
+    code_suffixes = {".py", ".rs", ".ts", ".tsx", ".js", ".jsx", ".go", ".java", ".c", ".h", ".cpp", ".hpp", ".rb", ".sh"}
+    text = content.decode("utf-8")
+    executable_markers = ("#!/", "import ", "from ", "fn ", "def ", "class ", "function ", "const ", "let ", "use ")
+    if suffix in code_suffixes:
+        return Disposition.PRIMARY, Reason.CODE_CHANGE
+    if any(marker in text for marker in executable_markers):
+        return Disposition.SECONDARY, Reason.MISLEADING_EXTENSION
+    return Disposition.NON_CODE, Reason.NON_CODE
+
+
+def git_blob_sha(content: bytes) -> str:
+    return hashlib.sha1(f"blob {len(content)}\0".encode("ascii") + content).hexdigest()
+
+
+def verify_head_binding(content: bytes, expected_blob_object: str, expected_content_sha256: str) -> None:
+    if git_blob_sha(content) != expected_blob_object or sha256_hex(content) != expected_content_sha256:
+        raise ValueError("vcs_object_mismatch")

src/stage1_cli.py

@@ -0,0 +1,26 @@
+"""Stage 1 CLI: verify a Stage 0 manifest and emit deterministic evidence."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from .stage1_evidence import build_evidence_manifest
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Build deterministic Stage 1 evidence from a Stage 0 manifest")
+    parser.add_argument("manifest", type=Path)
+    parser.add_argument("repository", type=Path)
+    args = parser.parse_args()
+    raw = json.loads(args.manifest.read_text(encoding="utf-8"))
+    evidence = build_evidence_manifest(raw, str(args.repository))
+    output = evidence.payload()
+    output["stage1_sha256"] = evidence.stage1_sha256
+    print(json.dumps(output, ensure_ascii=False, sort_keys=True, separators=(",", ":")))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())