From 66498c6f16bf8c6c4eb805fdc3dec32d6486b2da Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 05:43:17 -0800 Subject: [PATCH 01/68] refactor: Refine ADR-0004 documentation based on feedback This commit addresses 8 specific issues raised during review of the initial ADR-0004 documentation. The changes improve clarity, consistency, and technical precision across the specification. Fixes include: - **FEATURES.md**: - Rewrote acceptance criteria for F9-US-DEV to be specific and verifiable. - Removed the superseded F6 feature to avoid confusion with F9. - **SPEC.md**: - Corrected Mermaid diagram type casing ( -> ) to align with JSON standards. - Clarified that private blob decryption during pointer resolution is conditional, not assumed. - **TECH-SPEC.md**: - Removed redundant participant aliases in the projection sequence diagram. - Replaced the unconventional endpoint with a more RESTful URL. - Improved diagram precision by changing 'node' to 'field path'. - Specified JWS with a detached payload as the required authentication mechanism for pointer resolution. --- docs/FEATURES.md | 62 +++-- docs/SPEC.md | 60 +++-- docs/TECH-SPEC.md | 53 ++++- docs/USE-CASES.md | 10 + docs/decisions/ADR-0004/DECISION.md | 225 ++++++++++++++++++ docs/decisions/README.md | 1 + schemas/v1/privacy/opaque_pointer.schema.json | 46 ++++ 7 files changed, 407 insertions(+), 50 deletions(-) create mode 100644 docs/decisions/ADR-0004/DECISION.md create mode 100644 schemas/v1/privacy/opaque_pointer.schema.json diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 9b655fc2..68068e7e 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -141,29 +141,6 @@ Each feature includes user stories per relevant stakeholders (format requested), --- -## F6 — Opaque Pointers & CAS - -### F6-US-DML - -| | | -|--|--| -| **As a...** | Data/ML Engineer | -| **I want..** | encrypted artifacts with verifiable pointers | -| **So that...** | I can ship models across untrusted storage | - -#### Acceptance Criteria - -- [ ] Pointer includes plaintext hash, ciphertext hash, cipher meta -- [ ] Rekey operation available - -#### Test Plan - -- [ ] Golden: decrypt with correct key → match plaintext hash -- [ ] Edge: wrong bytes → hash mismatch -- [ ] Failure: rekey without authorization → deny - ---- - ## F7 — Epochs & Compaction ### F7-US-PENG @@ -208,3 +185,42 @@ Each feature includes user stories per relevant stakeholders (format requested), - [ ] Golden: metrics show non-zero counters post workload - [ ] Edge: cache stale → doctor recommends rebuild - [ ] Failure: FF-only violation → doctor flags critical +--- + +## F9 — Hybrid Privacy Model + +See also: [ADR-0004](./decisions/ADR-0004/DECISION.md). + +### F9-US-DEV + +| | | +|--|--| +| **As a...** | App Developer | +| **I want..** | to store sensitive data (PII, secrets) in a private store | +| **So that...** | my public, verifiable state does not contain confidential information | + +#### Acceptance Criteria + +- [ ] Given a `policy.yaml` file with a rule to `pointerize` the path `sensitive.field`, when the state is folded, the resulting public state tree MUST replace the value of `sensitive.field` with a canonical Opaque Pointer. +- [ ] Given a `policy.yaml` file with a rule to `pointerize` a field, the public state MUST contain a canonical Opaque Pointer at the specified path, with its `digest`, `location`, and `capability` fields correctly populated. +- [ ] When the Client SDK attempts to resolve an Opaque Pointer using the specified `location` and `capability`, and the client possesses the necessary authorization, the SDK MUST successfully retrieve and decrypt the original private data. + +### F9-US-SEC + +| | | +|--|--| +| **As a...** | Security/Compliance | +| **I want..** | to audit the separation of public and private data | +| **So that...** | I can verify that sensitive data is properly isolated and access is controlled | + +#### Acceptance Criteria + +- [ ] Opaque Pointer resolution fails without a valid capability. +- [ ] Private blob digest matches the digest in the public pointer. +- [ ] Commit trailers accurately report the number of redactions/pointers. + +#### Test Plan + +- [ ] Golden: project a unified state, resolve pointer, and verify content matches original. +- [ ] Edge: attempt to resolve a pointer with an invalid capability URI → DENY. +- [ ] Failure: tamper with a private blob → digest mismatch on resolution. diff --git a/docs/SPEC.md b/docs/SPEC.md index 041cf0bd..b9cc5310 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -128,6 +128,7 @@ graph TD A1 --> B6(audit) A1 --> B7(cache) A1 --> B8(epoch) + A1 --> B9(private) C(notes) --> C1(gatos) end subgraph Workspace @@ -148,6 +149,8 @@ The normative layout is as follows: │ └── gatos/ │ ├── journal/ │ ├── state/ +│ ├── private/ +│ │ └── / │ ├── mbus/ │ ├── mbus-ack/ │ ├── jobs/ @@ -282,28 +285,57 @@ On **DENY**, the gate **MUST** append an audit decision to `refs/gatos/audit/pol --- -## 7. Blob Pointers & Opaque Storage +## 7. Privacy and Opaque Pointers -Large or sensitive data is stored out-of-band in a content-addressed store and referenced via pointers. +See also: [ADR‑0004](./decisions/ADR-0004/DECISION.md). + +GATOS supports a hybrid privacy model where state can be separated into a verifiable public projection and a confidential private overlay. This is achieved by applying a deterministic **Projection Functor** during the state fold process, which replaces sensitive or large data with **Opaque Pointers**. + +### 7.1 Projection Model + +The State Engine (`gatos-echo`) can be configured with privacy rules. When folding history, it first computes a `UnifiedState` containing all data. It then applies the privacy rules to produce a `PublicState` and a set of `PrivateBlobs`. + +- **`PublicState`**: Contains only public data and Opaque Pointers. This is committed to the public `refs/gatos/state/public/...` namespace and is globally verifiable. +- **`PrivateBlobs`**: The raw data that was redacted or pointerized. This data is stored in a separate, private store (e.g., a local directory, a private object store) and is addressed by its content hash. + +Any commit that is the result of a privacy projection **MUST** include trailers indicating the number of redactions and pointers created. + +```text +Privacy-Redactions: 5 +Privacy-Pointers: 2 +``` + +### 7.2 Opaque Pointers + +An Opaque Pointer is a canonical JSON object that acts as a verifiable, addressable link to a private blob. It replaces the sensitive data in the `PublicState`. ```mermaid classDiagram - class BlobPointer { - +String kind: "blobptr" - +String algo - +String hash - +Number size - } class OpaquePointer { - +String kind: "opaque" - +String algo - +String hash - +String ciphertext_hash - +Object cipher_meta + +string kind: "opaque_pointer" + +string algo: "blake3" + +string digest: "blake3:" + +number size + +string location + +string capability } ``` -Pointers **MUST** refer to bytes in `gatos/objects//`. For opaque objects, no plaintext **MAY** be stored in Git. +- `digest`: The **REQUIRED** `blake3` hash of the raw private data. This ensures the integrity of the private blob. +- `location`: A **REQUIRED** URI indicating where the blob can be fetched (e.g., `gatos-node://ed25519:`, `s3://...`). +- `capability`: A **REQUIRED** URI defining the auth/authz and decryption mechanism needed to access the blob (e.g., `gatos-key://...`, `kms://...`). + +The pointer itself is canonicalized and its `content_id` can be computed for verification purposes. + +### 7.3 Pointer Resolution + +A client resolving an Opaque Pointer **MUST** perform the following steps: +1. Fetch the private blob from the `location` URI, authenticating if required by the endpoint protocol. +2. Acquire the necessary authorization and/or decryption keys by interacting with the `capability` URI's system. +3. If the blob is encrypted, decrypt it. +4. Verify that the `blake3` hash of the resulting plaintext exactly matches the `digest` in the pointer. The resolution **MUST** fail if the hashes do not match. + +This process guarantees that even though the data is stored privately, its integrity is verifiable against the public ledger. --- diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index 4f991b54..0050740c 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -101,10 +101,10 @@ graph TD | `gatos-ledger-git` | `std`-dependent storage backend using `libgit2`. | | `gatos-ledger` | Composes ledger components via feature flags. | | `gatos-mind` | Asynchronous, commit-backed message bus (pub/sub). | -| `gatos-echo` | Deterministic state engine for processing events ("folds"). | -| `gatos-policy` | Deterministic policy engine for executing compiled rules and managing the Consensus Governance lifecycle. | +| `gatos-echo` | Deterministic state engine for processing events ("folds"). Privacy projection logic. | +| `gatos-policy` | Deterministic policy engine for executing compiled rules, managing Consensus Governance, and privacy rule evaluation. | | `gatos-kv` | Git-backed key-value state cache. | -| `gatosd` | Main binary for the CLI and the JSONL RPC daemon. | +| `gatosd` | Main binary for the CLI, JSONL RPC daemon, and Opaque Pointer resolution endpoint. | | `gatos-compute` | Worker that discovers and executes jobs from the Job Plane. | | `gatos-wasm-bindings`| WASM bindings for browser and Node.js environments. | | `gatos-ffi-bindings` | C-compatible FFI for integration with other languages. | @@ -160,22 +160,48 @@ sequenceDiagram --- -## 6. Opaque Pointers +## 6. Privacy Projection and Resolution -The `rekey` command allows updating the encryption key for an opaque blob. +See also: [ADR‑0004](./decisions/ADR-0004/DECISION.md). + +The implementation of the hybrid privacy model involves a coordinated effort between the state, policy, and daemon components. + +### 6.1 Projection Implementation + +The projection from a `UnifiedState` to a `PublicState` is handled by `gatos-echo` with rules supplied by `gatos-policy`. ```mermaid sequenceDiagram - participant User - participant GATOS - - User->>GATOS: gatos blob rekey --to - GATOS->>GATOS: Create new Opaque Pointer - GATOS->>GATOS: Encrypt data with new pubkey - GATOS->>GATOS: Store new ciphertext in CAS - GATOS->>GATOS: Atomically update references + participant gatos-echo + participant gatos-policy + participant gatos-ledger + participant PrivateStore + + Echo->>Echo: 1. Fold event history to produce UnifiedState + Echo->>Policy: 2. Request privacy rules for the current context + Policy-->>Echo: 3. Return `select` and `action` rules +loop for each field path in the UnifiedState tree + gatos-echo->>gatos-echo: 4. Match field path against rules + alt rule matches (e.g., "pointerize") + Echo->>Echo: 5. Generate Opaque Pointer envelope + Echo->>PrivateStore: 6. Store original node value as private blob, keyed by its blake3 digest + Echo->>Echo: 7. Replace node in state tree with pointer + end + end + Echo->>Ledger: 8. Commit the final PublicState tree ``` +The `PrivateStore` is a pluggable trait, allowing for backends like a local filesystem, S3, or another GATOS node. + +### 6.2 Resolution Implementation + +The `gatosd` daemon exposes a secure endpoint for resolving Opaque Pointers. + +- **Endpoint**: `gatosd` will listen for authenticated requests, for example at `/gatos/private/blobs/{digest}`. +- **Authentication**: The client SDK **MUST** send a `Authorization` header containing a JSON Web Signature (JWS) with a detached payload. The JWS payload **MUST** be the BLAKE3 hash of the request body. `gatosd` verifies the signature against the actor's public key. +- **Authorization**: Upon receiving a valid request, `gatosd` queries `gatos-policy` to determine if the requesting actor has the capability to access the blob identified by `{digest}`. +- **Response**: If authorized, `gatosd` fetches the (likely encrypted) blob from its configured `PrivateStore` and returns it to the client. The client is then responsible for decryption via the `capability` URI. + --- ## 7. JSONL Protocol @@ -236,6 +262,7 @@ graph TD C --> C1(Golden Vectors); C --> C2(Torture Tests); C --> C3(Reconcile Harness); + C --> C4(Projection Determinism); ``` --- diff --git a/docs/USE-CASES.md b/docs/USE-CASES.md index f3c52cb7..26e7100c 100644 --- a/docs/USE-CASES.md +++ b/docs/USE-CASES.md @@ -83,3 +83,13 @@ This document illustrates practical scenarios where GATOS provides unique value. |**Goal** | Signed toggles with audit and rollbacks. | | **How** | KV‑style events + index refs; push‑gate for enforcement. | | **Why GATOS** | Auditable configuration without a new database. | + +--- + +## 9) Verifiable, Compliant PII Management + +| | | +|---|---| +|**Goal** | Manage customer data (PII) in a way that is both auditable and privacy-preserving. | +| **How** | A privacy policy projects the unified state into a public state with PII replaced by Opaque Pointers. The private data lives in an actor-anchored, encrypted blob store. | +| **Why GATOS** | Provides a verifiable public audit trail ("a user's data was accessed") without ever exposing the private data ("the user's address is...") to the public ledger. Access is gated by cryptographic capabilities. | \ No newline at end of file diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md new file mode 100644 index 00000000..501016e7 --- /dev/null +++ b/docs/decisions/ADR-0004/DECISION.md @@ -0,0 +1,225 @@ +--- +Status: Accepted +Date: 2025-11-10 +ADR: ADR-0004 +Authors: [flyingrobots, gemini-agent] +Requires: [ADR-0001] +Related: [ADR-0002, ADR-0003] +Tags: [Privacy, Projection, Opaque Pointers, Morphology Calculus] +Schemas: + - schemas/v1/privacy/opaque_pointer.schema.json +--- + +# ADR‑0004: Hybrid Privacy Model (Public Projection + Private Overlay) + +## Scope + +This ADR defines a **hybrid privacy model** for the GATOS operating surface. It formalizes the separation of state into a public, verifiable component and a private, actor-anchored overlay. This is achieved by introducing a **Projection Functor** that transforms a unified state into a public projection, leaving sensitive data in a private store referenced by **Opaque Pointers**. + +## Rationale + +GATOS's core value proposition is its verifiable, deterministic public ledger. However, many real-world applications require storing sensitive or large data (PII, secrets, large binaries) without committing it to the public history. The previous ad-hoc approach of using local, out-of-repo storage lacks the formal guarantees required by the GATOS Morphology Calculus. + +This ADR makes the hybrid model **normative, deterministic, and provable**. It ensures that public state remains globally verifiable while private data is securely addressable, auditable, and tied to the GATOS identity and policy model. + +## Mathematical Foundation (Morphology Calculus) + +This model is a direct application of the GATOS Morphology Calculus. + +1. **Shape Categories**: We define three categories of shapes: + * `Sh_Unified`: The category of shapes containing both public and private data. + * `Sh_Public`: The category of shapes containing only public data and opaque pointers. + * `Sh_Private`: The category of shapes containing only the private data blobs. + +2. **Projection as a Functor**: The privacy model is implemented as a functor, `Proj`, which maps shapes and morphisms from the unified category to the public category. + `Proj: Sh_Unified -> Sh_Public` + + This functor applies the privacy policy rules (`redact`, `pointerize`) to transform a unified shape into its public projection. The private data is extracted into `Sh_Private` during this process. + + ```mermaid + graph TD + subgraph Sh_Unified + U1("Unified Shape 1") + U2("Unified Shape 2") + U1 -- "Commit c" --> U2 + end + + subgraph Sh_Public + P1("Public Shape 1") + P2("Public Shape 2") + P1 -- "Proj(c)" --> P2 + end + + subgraph Sh_Private + B1("Private Blobs 1") + B2("Private Blobs 2") + end + + U1 -- "Proj" --> P1 + U2 -- "Proj" --> P2 + + U1 -- "Extract" --> B1 + U2 -- "Extract" --> B2 + + style P1 fill:#cde,stroke:#333 + style P2 fill:#cde,stroke:#333 + ``` + +This ensures that the transformation is structure-preserving and that the public history remains a valid, deterministic projection of the complete history. + +## Decision + +### 1. Actor-Anchored Private Namespace (Normative) + +Private data overlays are fundamentally tied to an actor's identity, not an ephemeral session. This anchors private data within the GATOS trust graph. + +- **Actor ID:** The canonical identifier for an actor, e.g., `ed25519:`. +- **Private Refs:** Private data is stored under refs namespaced by the actor ID. + ``` + refs/gatos/private/// + ``` +- **Public Refs:** The corresponding public projection lives in the main state namespace. + ``` + refs/gatos/state/public// + ``` + +### 2. Opaque Pointers (Normative) + +When private data is elided from the `PublicState`, a canonical JSON **Opaque Pointer** envelope is inserted in its place. + +```mermaid +classDiagram + class OpaquePointer { + +String kind: "opaque_pointer" + +String algo: "blake3" + +String digest: "blake3:" + +Number size + +String location + +String capability + } +``` + +- **`digest`**: The content-address of the private blob (`blake3(private_bytes)`). This is the immutable link between the public and private worlds. +- **`location`**: A URI indicating where to resolve the blob. Supported schemes include: + - `gatos-node://ed25519:`: Resolve via the GATOS trust graph. + - `https://...`, `s3://...`, `ipfs://...`: Standard distributed storage. + - `file:///...`: For local development and testing. +- **`capability`**: A URI defining the authorization and decryption mechanism required to access the blob. + - `gatos-key://v1/aes-256-gcm/`: A symmetric key managed by a GATOS-aware key service. + - `kms://...`, `age://...`, `sops://...`: Integration with standard secret management tools. + +The canonical `content_id` of the pointer itself is `blake3(canonical_json_bytes)`. + +**Schema:** `schemas/v1/privacy/opaque_pointer.schema.json` + +### 3. The Projection Function (Normative) + +The State Engine (`gatos-echo`) is responsible for executing the projection. + +1. It computes a **UnifiedState** by folding the complete event history. +2. It consults the **Privacy Policy** (`.gatos/policy.yaml`). +3. It traverses the `UnifiedState` tree, applying `redact` or `pointerize` rules. + - `redact`: The field is removed from the public state. + - `pointerize`: The field's value is stored as a private blob, and an Opaque Pointer is substituted in the public state. +4. The resulting `PublicState` is committed to the public refs, and the `Private Blobs` are persisted to their specified `location`. + +```mermaid +sequenceDiagram + participant E as State Engine (gatos-echo) + participant Pol as Policy Engine + participant L as Ledger (Git) + participant PS as Private Store + + E->>E: 1. Fold history into UnifiedState + E->>Pol: 2. Fetch privacy rules + Pol-->>E: 3. Return rules (redact/pointerize) + E->>E: 4. Apply rules to create PublicState + PrivateBlobs + E->>L: 5. Commit PublicState to public refs + E->>PS: 6. Store PrivateBlobs by digest +``` + +### 4. Pointer Resolution Protocol (Normative) + +A client resolving an Opaque Pointer **MUST** follow this protocol: + +1. **Parse Pointer**: Extract `digest`, `location`, and `capability`. +2. **Fetch Blob**: + - If `gatos-node://`, resolve the actor's endpoint from the trust graph. + - The client **MUST** send an authenticated request to the node (e.g., with a JWT or a signed challenge). + - The node's endpoint (e.g., `GET /.well-known/gatos/private/{digest}`) **MUST** verify the client's authorization against its policy before returning the blob. +3. **Acquire Capability**: + - Parse the `capability` URI. + - Interact with the specified system (KMS, key server) to get the decryption key. This step will have its own auth/authz protocol. +4. **Decrypt and Verify**: + - Decrypt the fetched blob using the key. + - Compute `blake3(decrypted_bytes)`. + - The operation **MUST FAIL** if the computed hash does not exactly match the `digest` in the pointer. + +```mermaid +sequenceDiagram + participant C as Client + participant PN as Private GATOS Node + participant KMS as Key Management Service + + C->>C: 1. Read OpaquePointer + C->>PN: 2. GET /private/{digest} (Authenticated) + PN->>PN: 3. Check policy (is C allowed?) + alt Authorized + PN-->>C: 4. Return encrypted blob + C->>KMS: 5. Request key for {capability} + KMS-->>C: 6. Return decryption key + C->>C: 7. Decrypt blob + C->>C: 8. Verify blake3(decrypted) == digest + else Unauthorized + PN-->>C: 4. Return 403 Forbidden + end +``` + +### 5. Policy Hooks (Normative) + +The privacy policy is defined in `.gatos/policy.yaml` and extends the policy engine's domain. + +```yaml +privacy: + rules: + - select: "path.to.sensitive.data" + action: "pointerize" + capability: "gatos-key://v1/aes-256-gcm/ops-key-01" + location: "gatos-node://ed25519:" + - select: "path.to.transient.data" + action: "redact" +``` + +The `select` syntax will use a simple path-matching language (e.g., glob patterns) defined by the policy engine. + +### 6. Auditability and Trailers (Normative) + +To make privacy operations transparent and auditable, any commit that creates a `PublicState` from a projection **MUST** include the following trailers: + +``` +Privacy-Redactions: 3 +Privacy-Pointers: 12 +``` + +This provides a simple, top-level indicator that a projection has occurred, prompting auditors to look deeper if necessary. + +## Consequences + +### Pros + +- **Provable Privacy**: The model is grounded in the Morphology Calculus, making it verifiable. +- **Decoupled Storage**: Private data can live in any storage system (S3, IPFS, local disk) without affecting the public ledger's logic. +- **Integrated Auth/Authz**: By tying pointers to actor identities and capabilities, access to private data is governed by the existing GATOS trust and policy model. +- **Preserves Verifiability**: The `PublicState` remains globally verifiable, as pointers are just content-addressed links. + +### Cons + +- **Increased Complexity**: Resolution requires network requests and interaction with key management systems, adding latency and potential points of failure. +- **Operational Overhead**: Operators must manage the private blob stores and ensure their availability and security. + +## Feature Payoff + +- **Secure PII/Secret Storage**: Store sensitive data off-chain while retaining an auditable link to it. +- **Large Artifact Management**: Handle large binaries (ML models, videos) without bloating the Git repository. +- **Compliant Data Sharing**: Share a public, redacted dataset with third parties while retaining private access to the full, unified view. +- **Federated Learning**: Different actors can hold private models locally, referenced by pointers in a public "training plan" shape. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 94b0994e..b61db3c4 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -20,3 +20,4 @@ Each ADR will have a status, typically one of the following: | [ADR-0001](./ADR-0001/DECISION.md) | Split gatos-ledger into no_std Core and std Backends | Accepted | 2025-11-08 | | [ADR-0002](./ADR-0002/DECISION.md) | Distributed Compute via a Job Plane | Accepted | 2025-11-08 | | [ADR-0003](./ADR-0003/DECISION.md) | Consensus Governance for Gated Actions | Accepted | 2025-11-08 | +| [ADR-0004](./ADR-0004/DECISION.md) | Hybrid Privacy Model (Public Projection + Private Overlay) | Accepted | 2025-11-10 | diff --git a/schemas/v1/privacy/opaque_pointer.schema.json b/schemas/v1/privacy/opaque_pointer.schema.json new file mode 100644 index 00000000..78a4d61b --- /dev/null +++ b/schemas/v1/privacy/opaque_pointer.schema.json @@ -0,0 +1,46 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "GATOS Opaque Pointer", + "description": "A canonical pointer to a private data blob, used to replace sensitive or large data in a public state projection.", + "type": "object", + "properties": { + "kind": { + "description": "The object kind, MUST be 'opaque_pointer'.", + "type": "string", + "const": "opaque_pointer" + }, + "algo": { + "description": "The hashing algorithm used for the digest, MUST be 'blake3'.", + "type": "string", + "const": "blake3" + }, + "digest": { + "description": "The content-address of the private blob, prefixed with the algorithm.", + "type": "string", + "pattern": "^blake3:[a-f0-9]{64}$" + }, + "size": { + "description": "Optional: The size of the private blob in bytes.", + "type": "integer", + "minimum": 0 + }, + "location": { + "description": "A URI indicating where the private blob can be resolved.", + "type": "string", + "format": "uri" + }, + "capability": { + "description": "A URI defining the authorization and/or decryption mechanism for the blob.", + "type": "string", + "format": "uri" + } + }, + "required": [ + "kind", + "algo", + "digest", + "location", + "capability" + ], + "additionalProperties": false +} From 46ec339ffa46af25bc00c04741a68be0759d9569 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 05:55:53 -0800 Subject: [PATCH 02/68] docs: Clarify commit trailer format in F9-US-SEC acceptance criteria Updated the acceptance criteria for F9-US-SEC in FEATURES.md to explicitly mention the and commit trailers, improving clarity for developers. --- docs/FEATURES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 68068e7e..28cf4d7b 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -217,7 +217,7 @@ See also: [ADR-0004](./decisions/ADR-0004/DECISION.md). - [ ] Opaque Pointer resolution fails without a valid capability. - [ ] Private blob digest matches the digest in the public pointer. -- [ ] Commit trailers accurately report the number of redactions/pointers. +- [ ] Commit trailers (`Privacy-Redactions`, `Privacy-Pointers`) accurately report the number of redactions/pointers. #### Test Plan From 676b6fa02a194e27f162425510502bc999335d22 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 05:58:20 -0800 Subject: [PATCH 03/68] docs: Clarify actor-id format in SPEC.md --- docs/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index b9cc5310..f0c5b079 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -150,7 +150,7 @@ The normative layout is as follows: │ ├── journal/ │ ├── state/ │ ├── private/ -│ │ └── / +│ │ └── / # e.g., the actor's ed25519 public key │ ├── mbus/ │ ├── mbus-ack/ │ ├── jobs/ From 7c35665b2d7e06b74e34e2176eadf7639aa6da87 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:00:34 -0800 Subject: [PATCH 04/68] docs: Clarify size unit for OpaquePointer in SPEC.md --- docs/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index f0c5b079..4379fd12 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -321,7 +321,7 @@ classDiagram } ``` -- `digest`: The **REQUIRED** `blake3` hash of the raw private data. This ensures the integrity of the private blob. +- `size`: The size of the private blob in bytes. - `location`: A **REQUIRED** URI indicating where the blob can be fetched (e.g., `gatos-node://ed25519:`, `s3://...`). - `capability`: A **REQUIRED** URI defining the auth/authz and decryption mechanism needed to access the blob (e.g., `gatos-key://...`, `kms://...`). From fce8bb1e11d666d479bd21df90b97dc261592461 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:01:30 -0800 Subject: [PATCH 05/68] docs: Clarify error handling for digest mismatch in SPEC.md Pointer Resolution --- docs/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index 4379fd12..cc49e685 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -333,7 +333,7 @@ A client resolving an Opaque Pointer **MUST** perform the following steps: 1. Fetch the private blob from the `location` URI, authenticating if required by the endpoint protocol. 2. Acquire the necessary authorization and/or decryption keys by interacting with the `capability` URI's system. 3. If the blob is encrypted, decrypt it. -4. Verify that the `blake3` hash of the resulting plaintext exactly matches the `digest` in the pointer. The resolution **MUST** fail if the hashes do not match. +4. Verify that the `blake3` hash of the resulting plaintext exactly matches the `digest` in the pointer. If the hashes do not match, the resolution **MUST** fail with a `DigestMismatch` error, and the client **SHOULD** log a security warning, as this may indicate data tampering. This process guarantees that even though the data is stored privately, its integrity is verifiable against the public ledger. From 3d6bcbdac9b48caae4d4a05eaa6d6717f85481bc Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:02:25 -0800 Subject: [PATCH 06/68] docs: Clarify PrivateStore participant as interface in TECH-SPEC.md diagram --- docs/TECH-SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index 0050740c..716ffef0 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -175,7 +175,7 @@ sequenceDiagram participant gatos-echo participant gatos-policy participant gatos-ledger - participant PrivateStore + participant "PrivateStore (Interface)" as "Storage Backend" Echo->>Echo: 1. Fold event history to produce UnifiedState Echo->>Policy: 2. Request privacy rules for the current context From 935ee36ac4c64181c1837c6287577d878ab628df Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:02:55 -0800 Subject: [PATCH 07/68] docs: Add details for Projection Determinism test suite in TECH-SPEC.md --- docs/TECH-SPEC.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index 716ffef0..b99236af 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -265,6 +265,8 @@ graph TD C --> C4(Projection Determinism); ``` +- **Projection Determinism**: Verifies that applying the same privacy policy to the same `UnifiedState` on different platforms (Linux, macOS, Windows) produces a byte-for-byte identical `PublicState` and the same set of private blobs. + --- ## 10. Security From f643aead23916f5c1dabc6423d8c288e3b3be6a2 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:03:23 -0800 Subject: [PATCH 08/68] docs: Update F9-US-DEV user story to BDD format in FEATURES.md --- docs/FEATURES.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 28cf4d7b..96ef4945 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -193,11 +193,9 @@ See also: [ADR-0004](./decisions/ADR-0004/DECISION.md). ### F9-US-DEV -| | | -|--|--| -| **As a...** | App Developer | -| **I want..** | to store sensitive data (PII, secrets) in a private store | -| **So that...** | my public, verifiable state does not contain confidential information | +**Given** I am an App Developer +**When** I define a piece of state as sensitive in my policy +**Then** that state should be stored in a private store and replaced with an Opaque Pointer in the public state. #### Acceptance Criteria From b39373cc519fca2cf003be51b44aefc687d6239a Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:03:38 -0800 Subject: [PATCH 09/68] docs: Mark gatos-compute as planned in SPEC.md system diagram --- docs/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index cc49e685..1f86ce4d 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -71,7 +71,7 @@ graph TD end subgraph "Job Plane" - Compute("gatos-compute"); + Compute("gatos-compute (planned)"); end subgraph "Ledger Plane" From 412b9b91083f4d3d82d28979876630e61b47ea36 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:03:57 -0800 Subject: [PATCH 10/68] docs: Change PoC envelope storage requirement from SHOULD to MUST in SPEC.md --- docs/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index 1f86ce4d..0bd47dfa 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -675,7 +675,7 @@ Proposal → Approvals (N‑of‑M) → Grant. Quorum groups (e.g., `@leads`) MU - A sorted list (by `Signer`) of all valid approvals used to reach quorum (by value or `Approval-Id`). - The governance rule id (`Policy-Rule`) and effective quorum parameters. -PoC envelope SHOULD be stored canonically under `refs/gatos/audit/proofs/governance/`; the Grant’s `Proof-Of-Consensus` trailer MUST equal `blake3(envelope_bytes)`. +PoC envelope MUST be stored canonically under `refs/gatos/audit/proofs/governance/`; the Grant’s `Proof-Of-Consensus` trailer MUST equal `blake3(envelope_bytes)`. ### 20.4 Lifecycle States From c63785ee06c7dd6f46f1c697b3e49d383f62cdd7 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:04:31 -0800 Subject: [PATCH 11/68] docs: Clarify purpose of gatos-kv crate in TECH-SPEC.md --- docs/TECH-SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index b99236af..31a359bc 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -103,7 +103,7 @@ graph TD | `gatos-mind` | Asynchronous, commit-backed message bus (pub/sub). | | `gatos-echo` | Deterministic state engine for processing events ("folds"). Privacy projection logic. | | `gatos-policy` | Deterministic policy engine for executing compiled rules, managing Consensus Governance, and privacy rule evaluation. | -| `gatos-kv` | Git-backed key-value state cache. | +| `gatos-kv` | Git-backed key-value state cache, used for materializing and indexing queryable views of folded state. | | `gatosd` | Main binary for the CLI, JSONL RPC daemon, and Opaque Pointer resolution endpoint. | | `gatos-compute` | Worker that discovers and executes jobs from the Job Plane. | | `gatos-wasm-bindings`| WASM bindings for browser and Node.js environments. | From 1ca5ec6b2a0c70da087e324be116558407f03531 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:04:46 -0800 Subject: [PATCH 12/68] docs: Mark chart data as illustrative in TECH-SPEC.md Performance Guidance --- docs/TECH-SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index 31a359bc..58947d41 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -310,7 +310,7 @@ Tuning batch size is a trade-off between latency and commit churn. ```mermaid xychart-beta - title "Batch Size Trade-off" + title "Batch Size Trade-off (Illustrative)" x-axis "Batch Size" y-axis "Metric" line "Latency" [50, 40, 35, 32, 30] From b5235c41de09ab328fab0446f5dffeb5c6c728b6 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:10:23 -0800 Subject: [PATCH 13/68] docs: Refine F9-US-DEV acceptance criteria to be more granular --- docs/FEATURES.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 96ef4945..7bb9c3b6 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -199,9 +199,11 @@ See also: [ADR-0004](./decisions/ADR-0004/DECISION.md). #### Acceptance Criteria -- [ ] Given a `policy.yaml` file with a rule to `pointerize` the path `sensitive.field`, when the state is folded, the resulting public state tree MUST replace the value of `sensitive.field` with a canonical Opaque Pointer. -- [ ] Given a `policy.yaml` file with a rule to `pointerize` a field, the public state MUST contain a canonical Opaque Pointer at the specified path, with its `digest`, `location`, and `capability` fields correctly populated. -- [ ] When the Client SDK attempts to resolve an Opaque Pointer using the specified `location` and `capability`, and the client possesses the necessary authorization, the SDK MUST successfully retrieve and decrypt the original private data. +- [ ] Given a `policy.yaml` with a rule to `pointerize` the path `sensitive.field`, when the state is folded, the public state tree MUST NOT contain the original value of `sensitive.field`. +- [ ] Given the same scenario, the public state tree MUST contain a canonical Opaque Pointer object at the `sensitive.field` path. +- [ ] The generated Opaque Pointer's `digest` field MUST match the BLAKE3 hash of the original, private value. +- [ ] The generated Opaque Pointer's `location` and `capability` fields MUST match the values specified in the `policy.yaml` rule. +- [ ] When the Client SDK resolves the pointer with correct authorization, the returned data MUST be byte-for-byte identical to the original `sensitive.field` value. ### F9-US-SEC From 8805e1b3b8776a2718950b8dc80b4c645d6319a4 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:10:44 -0800 Subject: [PATCH 14/68] docs: Simplify PrivateStore participant name in TECH-SPEC.md diagram --- docs/TECH-SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index 58947d41..8f927467 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -175,7 +175,7 @@ sequenceDiagram participant gatos-echo participant gatos-policy participant gatos-ledger - participant "PrivateStore (Interface)" as "Storage Backend" + participant "StorageBackend (Interface)" Echo->>Echo: 1. Fold event history to produce UnifiedState Echo->>Policy: 2. Request privacy rules for the current context From cc8d49e0e6f414dd898d728db9b8bbdf74daf73d Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:11:57 -0800 Subject: [PATCH 15/68] docs: Remove redundant acceptance criteria for F9-US-DEV --- docs/FEATURES.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7bb9c3b6..d5dd607d 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -197,14 +197,6 @@ See also: [ADR-0004](./decisions/ADR-0004/DECISION.md). **When** I define a piece of state as sensitive in my policy **Then** that state should be stored in a private store and replaced with an Opaque Pointer in the public state. -#### Acceptance Criteria - -- [ ] Given a `policy.yaml` with a rule to `pointerize` the path `sensitive.field`, when the state is folded, the public state tree MUST NOT contain the original value of `sensitive.field`. -- [ ] Given the same scenario, the public state tree MUST contain a canonical Opaque Pointer object at the `sensitive.field` path. -- [ ] The generated Opaque Pointer's `digest` field MUST match the BLAKE3 hash of the original, private value. -- [ ] The generated Opaque Pointer's `location` and `capability` fields MUST match the values specified in the `policy.yaml` rule. -- [ ] When the Client SDK resolves the pointer with correct authorization, the returned data MUST be byte-for-byte identical to the original `sensitive.field` value. - ### F9-US-SEC | | | From f1e6f8b22a19902e0c6db7001ffa0e22a27a9d76 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:12:14 -0800 Subject: [PATCH 16/68] docs: Add missing field description for digest in OpaquePointer --- docs/SPEC.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/SPEC.md b/docs/SPEC.md index 0bd47dfa..b9ff073f 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -321,6 +321,7 @@ classDiagram } ``` +- `digest`: The **REQUIRED** `blake3` hash of the raw private data. This ensures the integrity of the private blob. - `size`: The size of the private blob in bytes. - `location`: A **REQUIRED** URI indicating where the blob can be fetched (e.g., `gatos-node://ed25519:`, `s3://...`). - `capability`: A **REQUIRED** URI defining the auth/authz and decryption mechanism needed to access the blob (e.g., `gatos-key://...`, `kms://...`). From 8cc609110e656d5c5ffe32e7909aeaf665edf755 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:12:30 -0800 Subject: [PATCH 17/68] docs: Clarify interaction of expiration dates in Consensus Governance --- docs/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index b9ff073f..8ced39ae 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -657,7 +657,7 @@ Proposal → Approvals (N‑of‑M) → Grant. Quorum groups (e.g., `@leads`) MU Proposal-Id: blake3: Approval-Id: blake3: Signer: ed25519: - Expires-At: # OPTIONAL + Expires-At: # OPTIONAL. If present, the approval is only valid until this time. It cannot extend the proposal's expiration. ``` - Grant (at `refs/gatos/grants/…`): From f8de6f234eaa39151251a595ae18390aec388219 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:12:45 -0800 Subject: [PATCH 18/68] docs: Specify HTTP GET method for pointer resolution endpoint --- docs/TECH-SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index 8f927467..74c0da5b 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -197,7 +197,7 @@ The `PrivateStore` is a pluggable trait, allowing for backends like a local file The `gatosd` daemon exposes a secure endpoint for resolving Opaque Pointers. -- **Endpoint**: `gatosd` will listen for authenticated requests, for example at `/gatos/private/blobs/{digest}`. +- **Endpoint**: `gatosd` will listen for authenticated `GET` requests at `/gatos/private/blobs/{digest}`. - **Authentication**: The client SDK **MUST** send a `Authorization` header containing a JSON Web Signature (JWS) with a detached payload. The JWS payload **MUST** be the BLAKE3 hash of the request body. `gatosd` verifies the signature against the actor's public key. - **Authorization**: Upon receiving a valid request, `gatosd` queries `gatos-policy` to determine if the requesting actor has the capability to access the blob identified by `{digest}`. - **Response**: If authorized, `gatosd` fetches the (likely encrypted) blob from its configured `PrivateStore` and returns it to the client. The client is then responsible for decryption via the `capability` URI. From e00b5780f0245b1fa1e03fc94336867274ce5cb3 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:13:05 -0800 Subject: [PATCH 19/68] docs: Correct type casing in ADR-0004 OpaquePointer diagram --- docs/decisions/ADR-0004/DECISION.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index 501016e7..de7a356d 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -90,12 +90,12 @@ When private data is elided from the `PublicState`, a canonical JSON **Opaque Po ```mermaid classDiagram class OpaquePointer { - +String kind: "opaque_pointer" - +String algo: "blake3" - +String digest: "blake3:" - +Number size - +String location - +String capability + +string kind: "opaque_pointer" + +string algo: "blake3" + +string digest: "blake3:" + +number size + +string location + +string capability } ``` From e48b7f9ab372a4ed401b3e301d47bcc2a039683b Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:17:48 -0800 Subject: [PATCH 20/68] docs: Further refine F9-US-DEV acceptance criteria with BDD-style granularity --- docs/FEATURES.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d5dd607d..a12ca4ae 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -193,9 +193,13 @@ See also: [ADR-0004](./decisions/ADR-0004/DECISION.md). ### F9-US-DEV -**Given** I am an App Developer -**When** I define a piece of state as sensitive in my policy -**Then** that state should be stored in a private store and replaced with an Opaque Pointer in the public state. +#### Acceptance Criteria + +- [ ] **Given** a `policy.yaml` with a rule to `pointerize` the path `sensitive.field`, **when** the state is folded, **then** the resulting public state tree MUST NOT contain the original value of `sensitive.field`. +- [ ] **Given** the same scenario, **when** the state is folded, **then** the public state tree MUST contain a canonical Opaque Pointer object at the `sensitive.field` path. +- [ ] **Given** a `pointerized` field, **when** the Opaque Pointer is generated, **then** its `digest` field MUST match the BLAKE3 hash of the original, private value. +- [ ] **Given** a `pointerized` field, **when** the Opaque Pointer is generated, **then** its `location` and `capability` fields MUST match the values specified in the `policy.yaml` rule. +- [ ] **Given** a valid Opaque Pointer, **when** the Client SDK resolves it with correct authorization, **then** the returned data MUST be byte-for-byte identical to the original private data. ### F9-US-SEC From b8b950458f7483b31621b3bfe01403670dec58c0 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:18:14 -0800 Subject: [PATCH 21/68] docs: Remove redundant acceptance criteria for F9-US-DEV in FEATURES.md --- docs/FEATURES.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index a12ca4ae..5706e3ee 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -193,14 +193,6 @@ See also: [ADR-0004](./decisions/ADR-0004/DECISION.md). ### F9-US-DEV -#### Acceptance Criteria - -- [ ] **Given** a `policy.yaml` with a rule to `pointerize` the path `sensitive.field`, **when** the state is folded, **then** the resulting public state tree MUST NOT contain the original value of `sensitive.field`. -- [ ] **Given** the same scenario, **when** the state is folded, **then** the public state tree MUST contain a canonical Opaque Pointer object at the `sensitive.field` path. -- [ ] **Given** a `pointerized` field, **when** the Opaque Pointer is generated, **then** its `digest` field MUST match the BLAKE3 hash of the original, private value. -- [ ] **Given** a `pointerized` field, **when** the Opaque Pointer is generated, **then** its `location` and `capability` fields MUST match the values specified in the `policy.yaml` rule. -- [ ] **Given** a valid Opaque Pointer, **when** the Client SDK resolves it with correct authorization, **then** the returned data MUST be byte-for-byte identical to the original private data. - ### F9-US-SEC | | | From 5dca809b723d9b173127684afa7a490d1963ec18 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:18:44 -0800 Subject: [PATCH 22/68] docs: Clarify sorting order for approvals in SPEC.md PoC section --- docs/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index 8ced39ae..3eb8523f 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -673,7 +673,7 @@ Proposal → Approvals (N‑of‑M) → Grant. Quorum groups (e.g., `@leads`) MU `Proof-Of-Consensus` is the BLAKE3 of a canonical JSON envelope containing: - The canonical proposal envelope (by value or `Proposal-Id`). -- A sorted list (by `Signer`) of all valid approvals used to reach quorum (by value or `Approval-Id`). +- A lexicographically sorted list (by Signer's public key) of all valid approvals used to reach quorum (each by value or `Approval-Id`). - The governance rule id (`Policy-Rule`) and effective quorum parameters. PoC envelope MUST be stored canonically under `refs/gatos/audit/proofs/governance/`; the Grant’s `Proof-Of-Consensus` trailer MUST equal `blake3(envelope_bytes)`. From a7b1a215c9381ac558e3e1f01de175bc45216b9c Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 06:19:49 -0800 Subject: [PATCH 23/68] docs: Correct inconsistent endpoint URL in ADR-0004 --- docs/decisions/ADR-0004/DECISION.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index de7a356d..3fd6786a 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -146,7 +146,7 @@ A client resolving an Opaque Pointer **MUST** follow this protocol: 2. **Fetch Blob**: - If `gatos-node://`, resolve the actor's endpoint from the trust graph. - The client **MUST** send an authenticated request to the node (e.g., with a JWT or a signed challenge). - - The node's endpoint (e.g., `GET /.well-known/gatos/private/{digest}`) **MUST** verify the client's authorization against its policy before returning the blob. + - The node's endpoint (e.g., `GET /gatos/private/blobs/{digest}`) **MUST** verify the client's authorization against its policy before returning the blob. 3. **Acquire Capability**: - Parse the `capability` URI. - Interact with the specified system (KMS, key server) to get the decryption key. This step will have its own auth/authz protocol. From d3636881691c1d29143dfbc627ef3be91bf22739 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 08:04:23 -0800 Subject: [PATCH 24/68] docs(ADR-0004,SPEC,TECH-SPEC): switch resolver to POST + JWT; add optional HTTP Message Signatures profile; tighten determinism (RFC 8785 JCS + key ordering); clarify namespaces and error taxonomy; add pointer rotation trailer\nschema(privacy): add ciphertext_digest + extensions; bump to draft 2020-12\nschema(policy): add optional privacy.classes + rules\nexamples: add privacy_min and updated opaque_pointer example\nscripts: validate privacy schema/examples in CI\nmake: fix tab indentation for lint/fix targets\nchore: add gatos-privacy crate with OpaquePointer type and notes --- Cargo.lock | 12 +++ Cargo.toml | 1 + Makefile | 10 +- crates/gatos-privacy/Cargo.toml | 13 +++ crates/gatos-privacy/src/lib.rs | 42 ++++++++ docs/SPEC.md | 50 +++++++--- docs/TECH-SPEC.md | 27 ++++- docs/decisions/ADR-0003/DECISION.md | 2 +- docs/decisions/ADR-0004/DECISION.md | 99 +++++++++++++++---- examples/v1/policy/privacy_min.json | 21 ++++ examples/v1/privacy/opaque_pointer_min.json | 9 ++ .../v1/policy/governance_policy.schema.json | 34 +++++++ schemas/v1/privacy/opaque_pointer.schema.json | 50 +++------- scripts/validate_schemas.sh | 4 + 14 files changed, 293 insertions(+), 81 deletions(-) create mode 100644 crates/gatos-privacy/Cargo.toml create mode 100644 crates/gatos-privacy/src/lib.rs create mode 100644 examples/v1/policy/privacy_min.json create mode 100644 examples/v1/privacy/opaque_pointer_min.json diff --git a/Cargo.lock b/Cargo.lock index 6c947707..b310fe4f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -328,6 +328,18 @@ version = "0.1.0" name = "gatos-policy" version = "0.1.0" +[[package]] +name = "gatos-privacy" +version = "0.1.0" +dependencies = [ + "anyhow", + "blake3", + "gatos-ledger-core", + "hex", + "serde", + "serde_json", +] + [[package]] name = "gatos-wasm-bindings" version = "0.1.0" diff --git a/Cargo.toml b/Cargo.toml index 67934114..9279a17a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -7,6 +7,7 @@ members = [ "crates/gatos-mind", "crates/gatos-echo", "crates/gatos-policy", + "crates/gatos-privacy", "crates/gatos-kv", "crates/gatosd", "bindings/wasm", diff --git a/Makefile b/Makefile index 5259946a..39b8f699 100644 --- a/Makefile +++ b/Makefile @@ -12,14 +12,14 @@ diagrams: @bash -lc 'scripts/mermaid/generate_all.sh' lint-md: - @bash -lc 'if command -v node >/dev/null 2>&1; then \ + @bash -lc 'if command -v node >/dev/null 2>&1; then \ npx -y markdownlint-cli "**/*.md" --config .markdownlint.json; \ elif command -v docker >/dev/null 2>&1; then \ docker run --rm -v "$$PWD:/work" -w /work node:20 bash -lc "npx -y markdownlint-cli \"**/*.md\" --config .markdownlint.json"; \ else echo "Need Node.js or Docker" >&2; exit 1; fi' fix-md: - @bash -lc 'if command -v node >/dev/null 2>&1; then \ + @bash -lc 'if command -v node >/dev/null 2>&1; then \ npx -y markdownlint-cli "**/*.md" --fix --config .markdownlint.json; \ elif command -v docker >/dev/null 2>&1; then \ docker run --rm -v "$$PWD:/work" -w /work node:20 bash -lc "npx -y markdownlint-cli \"**/*.md\" --fix --config .markdownlint.json"; \ @@ -44,7 +44,8 @@ schema-compile: npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/grant.schema.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/revocation.schema.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proof_of_consensus_envelope.schema.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json' + npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json && \ + npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -r schemas/v1/common/ids.schema.json' schema-validate: @bash -lc 'set -euo pipefail; \ @@ -57,7 +58,8 @@ schema-validate: npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/grant.schema.json -d examples/v1/governance/grant_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/revocation.schema.json -d examples/v1/governance/revocation_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proof_of_consensus_envelope.schema.json -d examples/v1/governance/poc_envelope_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.json' + npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.json && \ + npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -d examples/v1/privacy/opaque_pointer_min.json -r schemas/v1/common/ids.schema.json' schema-negative: @bash -lc 'set -euo pipefail; \ diff --git a/crates/gatos-privacy/Cargo.toml b/crates/gatos-privacy/Cargo.toml new file mode 100644 index 00000000..631ea04c --- /dev/null +++ b/crates/gatos-privacy/Cargo.toml @@ -0,0 +1,13 @@ +[package] +name = "gatos-privacy" +version = "0.1.0" +edition = "2021" + +[dependencies] +gatos-ledger-core = { path = "../gatos-ledger-core" } +serde = { workspace = true, features = ["derive"] } +serde_json = { workspace = true } +blake3 = { workspace = true } +hex = { workspace = true } +anyhow = { workspace = true } + diff --git a/crates/gatos-privacy/src/lib.rs b/crates/gatos-privacy/src/lib.rs new file mode 100644 index 00000000..28e54abb --- /dev/null +++ b/crates/gatos-privacy/src/lib.rs @@ -0,0 +1,42 @@ +//! gatos-privacy — Opaque Pointer types and helpers +//! +//! This crate defines the JSON-facing pointer envelope used by the +//! hybrid privacy model (ADR-0004). The struct mirrors the v1 schema +//! in `schemas/v1/privacy/opaque_pointer.schema.json`. +//! +//! Canonicalization: when computing content IDs or digests, callers +//! MUST serialize JSON using RFC 8785 JCS. This crate intentionally +//! does not take a dependency on a specific JCS implementation to +//! keep the workspace lean; higher layers may provide one. + +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct OpaquePointer { + pub kind: Kind, + pub algo: Algo, + pub digest: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub ciphertext_digest: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub size: Option, + pub location: String, + pub capability: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub extensions: Option, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Kind { + OpaquePointer, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum Algo { + Blake3, +} + diff --git a/docs/SPEC.md b/docs/SPEC.md index 3eb8523f..c1f19549 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -314,27 +314,49 @@ classDiagram class OpaquePointer { +string kind: "opaque_pointer" +string algo: "blake3" - +string digest: "blake3:" - +number size + +string digest: "blake3:" // plaintext digest + +string ciphertext_digest: "blake3:" // optional + +int size // bytes; SHOULD be present +string location - +string capability + +string capability // MUST NOT embed secrets + +object extensions // forward-compatible } ``` -- `digest`: The **REQUIRED** `blake3` hash of the raw private data. This ensures the integrity of the private blob. -- `size`: The size of the private blob in bytes. -- `location`: A **REQUIRED** URI indicating where the blob can be fetched (e.g., `gatos-node://ed25519:`, `s3://...`). -- `capability`: A **REQUIRED** URI defining the auth/authz and decryption mechanism needed to access the blob (e.g., `gatos-key://...`, `kms://...`). +- `digest`: The **REQUIRED** `blake3` hash of the plaintext. For low‑entropy privacy classes, the public pointer MUST NOT expose this value. +- `ciphertext_digest`: The `blake3` hash of the stored ciphertext. For low‑entropy privacy classes, this field MUST be present in the public pointer. +- `size`: The size of the private blob in bytes (RECOMMENDED). +- `location`: A **REQUIRED** stable URI indicating where the blob can be fetched (e.g., `gatos-node://ed25519:`, `s3://bucket/key`). Do not embed pre‑signed tokens. +- `capability`: A **REQUIRED** reference to the authn/z + decryption mechanism (e.g., `gatos-key://...`, `kms://...`). It MUST NOT embed secrets; resolution occurs at the policy layer. -The pointer itself is canonicalized and its `content_id` can be computed for verification purposes. +The pointer itself is canonicalized via RFC 8785 JCS and its `content_id` is `blake3(JCS(pointer_json))`. ### 7.3 Pointer Resolution -A client resolving an Opaque Pointer **MUST** perform the following steps: -1. Fetch the private blob from the `location` URI, authenticating if required by the endpoint protocol. -2. Acquire the necessary authorization and/or decryption keys by interacting with the `capability` URI's system. -3. If the blob is encrypted, decrypt it. -4. Verify that the `blake3` hash of the resulting plaintext exactly matches the `digest` in the pointer. If the hashes do not match, the resolution **MUST** fail with a `DigestMismatch` error, and the client **SHOULD** log a security warning, as this may indicate data tampering. +Endpoint and AuthN: +- Clients MUST resolve via `POST /gatos/private/blobs/resolve` with body `{ "digest": "blake3:", "want": "plaintext"|"ciphertext" }` and `Authorization: Bearer `. +- Tokens MUST include standard claims (`sub`, `aud`, `method`, `path`, `exp`, `nbf`); skew tolerance ±300s. 401 for authn failures; 403 for policy denials. + +Verification Steps: +1. Fetch the ciphertext blob from `location` via the node’s resolver endpoint. +2. Acquire the necessary keys via the `capability` reference (policy-driven; no secrets in the pointer). +3. Decrypt. Compute `blake3(ciphertext)` and compare with `ciphertext_digest` when present; compute `blake3(plaintext)` and compare with `digest` when exposed. Any mismatch MUST yield `DigestMismatch`. +4. Servers SHOULD return `X-BLAKE3-Digest` and `Digest: sha-256=…` headers for response integrity. + +Error Taxonomy: +- `Unauthorized` (401), `Forbidden` (403), `NotFound` (404), `DigestMismatch` (409), `CapabilityUnavailable` (503), `PolicyDenied` (403). + +Optional HTTP Message Signatures profile (RFC 9421): +- As an alternative to JWT, clients MAY sign `@method`, `@target-uri`, `date`, `host`, `content-digest` and send `Signature-Input`/`Signature` headers. Servers SHOULD still emit `Digest` and `X-BLAKE3-Digest` response headers. + +Pointer Rotation (Rekey): +1) fetch ciphertext; 2) decrypt; 3) re‑encrypt per new capability; 4) store new ciphertext; 5) emit rotation event updating pointer fields (capability/location). `digest` (plaintext) MUST remain stable. Add trailer `Privacy-Pointer-Rotations: `. + +Namespacing: +- `refs/gatos/private//…` holds private overlay indices/metadata only; workspace mirror is `gatos/private//…`. Blobs live in external stores keyed by digest. + +Canonicalization: +- All JSON labeled as canonical MUST use RFC 8785 JCS; non‑JSON maps MUST be ordered lexicographically by lowercase UTF‑8 keys. This process guarantees that even though the data is stored privately, its integrity is verifiable against the public ledger. @@ -673,7 +695,7 @@ Proposal → Approvals (N‑of‑M) → Grant. Quorum groups (e.g., `@leads`) MU `Proof-Of-Consensus` is the BLAKE3 of a canonical JSON envelope containing: - The canonical proposal envelope (by value or `Proposal-Id`). -- A lexicographically sorted list (by Signer's public key) of all valid approvals used to reach quorum (each by value or `Approval-Id`). +- A lexicographically sorted list of approvals ordered by the lowercase ASCII of each approval's `Signer` value (the `ed25519:` string). Each approval is included by value or via `Approval-Id`. - The governance rule id (`Policy-Rule`) and effective quorum parameters. PoC envelope MUST be stored canonically under `refs/gatos/audit/proofs/governance/`; the Grant’s `Proof-Of-Consensus` trailer MUST equal `blake3(envelope_bytes)`. diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index 74c0da5b..dfe0599e 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -197,10 +197,29 @@ The `PrivateStore` is a pluggable trait, allowing for backends like a local file The `gatosd` daemon exposes a secure endpoint for resolving Opaque Pointers. -- **Endpoint**: `gatosd` will listen for authenticated `GET` requests at `/gatos/private/blobs/{digest}`. -- **Authentication**: The client SDK **MUST** send a `Authorization` header containing a JSON Web Signature (JWS) with a detached payload. The JWS payload **MUST** be the BLAKE3 hash of the request body. `gatosd` verifies the signature against the actor's public key. -- **Authorization**: Upon receiving a valid request, `gatosd` queries `gatos-policy` to determine if the requesting actor has the capability to access the blob identified by `{digest}`. -- **Response**: If authorized, `gatosd` fetches the (likely encrypted) blob from its configured `PrivateStore` and returns it to the client. The client is then responsible for decryption via the `capability` URI. +- Endpoint: `POST /gatos/private/blobs/resolve` +- Content-Type: `application/json` +- Request body (JCS canonical JSON): + ```json + { "digest": "blake3:", "want": "plaintext" } + ``` + - `want` OPTIONAL: `"plaintext" | "ciphertext"` (default `"plaintext"`). +- Authentication: `Authorization: Bearer ` + - Claims (example): `iss`, `sub` (ed25519:), `aud` ("gatos-node:"), `exp`, `nbf`, `jti`, `method` ("POST"), `path` ("/gatos/private/blobs/resolve"), `digest` (MUST match body.digest). + - Clock skew tolerance: ±300 seconds. +- Authorization: Node evaluates policy for `` on ``. +- Response (200 OK): + - Headers: `Digest: sha-256=`, `X-BLAKE3-Digest: blake3:` + - Body: requested bytes (ciphertext or plaintext). + +Errors: 401 Unauthorized, 403 Forbidden, 404 Not Found, 409 DigestMismatch, 503 CapabilityUnavailable. + +Optional profile (HTTP Message Signatures, RFC 9421): +- Clients MAY authenticate by signing components: `@method`, `@target-uri`, `date`, `host`, `content-digest` (SHA-256 over request body) and sending `Signature-Input: sig1=...` and `Signature: sig1=::`. +- Servers STILL apply policy and SHOULD return `Digest` and `X-BLAKE3-Digest` headers. + +Pointer Rotation (Rekey): +- Implement a rotation that: (1) fetches; (2) decrypts; (3) re‑encrypts; (4) stores; (5) emits an audit event updating pointer fields while keeping plaintext `digest` stable. Add trailer `Privacy-Pointer-Rotations: ` when a projection commit includes rotations. --- diff --git a/docs/decisions/ADR-0003/DECISION.md b/docs/decisions/ADR-0003/DECISION.md index 45f29beb..d33c69a5 100644 --- a/docs/decisions/ADR-0003/DECISION.md +++ b/docs/decisions/ADR-0003/DECISION.md @@ -86,7 +86,7 @@ Define a system for gating specific GATOS actions (e.g., locking a file, publish 7. Proof‑Of‑Consensus (normative) - The `Proof-Of-Consensus` digest MUST be the BLAKE3 of a canonical envelope that includes (see schema: [`schemas/v1/governance/proof_of_consensus_envelope.schema.json`](../../../schemas/v1/governance/proof_of_consensus_envelope.schema.json)): - The canonical proposal envelope (by value or by `Proposal-Id`). - - A sorted list (by `Signer`) of all valid approvals used to reach quorum (each by value or `Approval-Id`). + - A lexicographically sorted list of approvals by the lowercase ASCII of each approval's `Signer` value (the `ed25519:` string). Each approval is included by value or via `Approval-Id`. - The governance rule id (`Policy-Rule`) and effective quorum parameters. - Implementations MUST use canonical JSON (UTF‑8, sorted keys, no insignificant whitespace) to build this envelope before hashing. All hex encodings MUST be lowercase. Ordering by signer is an application‑level MUST; JSON Schema cannot enforce sort order. - Storage: The canonical PoC envelope JSON SHOULD be persisted as a blob referenced under `refs/gatos/audit/proofs/governance/`; the `Proof-Of-Consensus` trailer MUST equal `blake3(envelope_bytes)`. diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index 3fd6786a..13804d5e 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -92,23 +92,25 @@ classDiagram class OpaquePointer { +string kind: "opaque_pointer" +string algo: "blake3" - +string digest: "blake3:" - +number size + +string digest: "blake3:" // plaintext digest + +string ciphertext_digest "blake3:" // MAY be present + +int size // SHOULD be present (bytes) +string location - +string capability + +string capability // MUST NOT embed secrets + +object extensions // forward-compatible } ``` -- **`digest`**: The content-address of the private blob (`blake3(private_bytes)`). This is the immutable link between the public and private worlds. +- **`digest`**: The content-address of the private plaintext (`blake3(plaintext_bytes)`). This is the immutable link between the public and private worlds. +- **`ciphertext_digest`**: The content-address of the stored ciphertext (`blake3(ciphertext_bytes)`). For low‑entropy privacy classes (see Policy Hooks), the public pointer **MUST** include `ciphertext_digest` and policy **MUST NOT** expose the plaintext digest publicly. - **`location`**: A URI indicating where to resolve the blob. Supported schemes include: - `gatos-node://ed25519:`: Resolve via the GATOS trust graph. - `https://...`, `s3://...`, `ipfs://...`: Standard distributed storage. - `file:///...`: For local development and testing. -- **`capability`**: A URI defining the authorization and decryption mechanism required to access the blob. - - `gatos-key://v1/aes-256-gcm/`: A symmetric key managed by a GATOS-aware key service. - - `kms://...`, `age://...`, `sops://...`: Integration with standard secret management tools. +- **`capability`**: A reference identifying the authorization and decryption mechanism required to access the blob. It **MUST NOT** embed secrets or pre‑signed tokens. It SHOULD be a stable identifier (e.g., `gatos-key://v1/aes-256-gcm/` or `kms://...`) that can be resolved privately at the policy layer. + - Pointers MAY publish a non‑sensitive label and keep resolver details private via policy. Implementations MAY also place auxiliary hints inside `extensions`. -The canonical `content_id` of the pointer itself is `blake3(canonical_json_bytes)`. +The canonical `content_id` of the pointer itself is `blake3(JCS(pointer_json))`, where `JCS(…)` denotes RFC 8785 JSON Canonicalization Scheme applied to UTF‑8 bytes. This rule is normative for all canonical JSON in GATOS (pointers, governance envelopes, any JSON state snapshots). **Schema:** `schemas/v1/privacy/opaque_pointer.schema.json` @@ -123,6 +125,10 @@ The State Engine (`gatos-echo`) is responsible for executing the projection. - `pointerize`: The field's value is stored as a private blob, and an Opaque Pointer is substituted in the public state. 4. The resulting `PublicState` is committed to the public refs, and the `Private Blobs` are persisted to their specified `location`. +Determinism Requirements: +- All JSON artifacts produced during projection (including Opaque Pointers) MUST be canonicalized with RFC 8785 JCS prior to hashing. +- When non‑JSON maps are materialized (e.g., Git tree entries), keys MUST be ordered lexicographically by their lowercase UTF‑8 bytes. + ```mermaid sequenceDiagram participant E as State Engine (gatos-echo) @@ -140,20 +146,38 @@ sequenceDiagram ### 4. Pointer Resolution Protocol (Normative) +Authentication semantics are aligned with HTTP. We adopt a simple, interoperable model (JWT default; HTTP Message Signatures optional): + +- **Endpoint**: `POST /gatos/private/blobs/resolve` +- **Request Body (application/json; JCS canonical form)**: + `{ "digest": "blake3:", "want": "plaintext"|"ciphertext" }` +- **Authorization**: `Authorization: Bearer ` + - Claims MUST include: `sub` (ed25519:), `aud` (node id or URL), `method` ("POST"), `path` ("/gatos/private/blobs/resolve"), `exp`, and `nbf`. + - Clock skew tolerance: ±300 seconds. + - On missing/invalid token: `401 Unauthorized`. On policy denial: `403 Forbidden`. + A client resolving an Opaque Pointer **MUST** follow this protocol: -1. **Parse Pointer**: Extract `digest`, `location`, and `capability`. +1. **Parse Pointer**: Extract `digest`, optional `ciphertext_digest`, `location`, and `capability`. 2. **Fetch Blob**: - - If `gatos-node://`, resolve the actor's endpoint from the trust graph. - - The client **MUST** send an authenticated request to the node (e.g., with a JWT or a signed challenge). - - The node's endpoint (e.g., `GET /gatos/private/blobs/{digest}`) **MUST** verify the client's authorization against its policy before returning the blob. + - If `gatos-node://`, resolve the actor's endpoint from the trust graph, then `POST /gatos/private/blobs/resolve` with the body above. + - The node **MUST** verify the bearer token and enforce policy before returning the blob. 3. **Acquire Capability**: - - Parse the `capability` URI. - - Interact with the specified system (KMS, key server) to get the decryption key. This step will have its own auth/authz protocol. + - Resolve the `capability` reference via the configured key system (KMS, key server). Secrets MUST NOT be embedded in the pointer. 4. **Decrypt and Verify**: - - Decrypt the fetched blob using the key. - - Compute `blake3(decrypted_bytes)`. - - The operation **MUST FAIL** if the computed hash does not exactly match the `digest` in the pointer. + - Decrypt the fetched blob using the resolved key and AAD parameters (see Security Notes). + - Compute `blake3(plaintext)` and compare to `digest` if published; compute `blake3(ciphertext)` and compare to `ciphertext_digest` if published. A mismatch **MUST** produce `DigestMismatch`. + +Response headers on success: +``` +Content-Type: application/octet-stream +X-BLAKE3-Digest: blake3: +Digest: sha-256= +``` + +Optional HTTP Message Signatures profile (RFC 9421): +- Clients MAY authenticate by signing `@method`, `@target-uri`, `date`, `host`, `content-digest` (SHA‑256 of the JSON body) and sending `Signature-Input` and `Signature` headers. +- Servers SHOULD still return `Digest` and `X-BLAKE3-Digest` headers for response integrity. ```mermaid sequenceDiagram @@ -162,7 +186,7 @@ sequenceDiagram participant KMS as Key Management Service C->>C: 1. Read OpaquePointer - C->>PN: 2. GET /private/{digest} (Authenticated) + C->>PN: 2. POST /gatos/private/blobs/resolve (Authorization: Bearer ) PN->>PN: 3. Check policy (is C allowed?) alt Authorized PN-->>C: 4. Return encrypted blob @@ -171,7 +195,7 @@ sequenceDiagram C->>C: 7. Decrypt blob C->>C: 8. Verify blake3(decrypted) == digest else Unauthorized - PN-->>C: 4. Return 403 Forbidden + PN-->>C: 4. Return 401/403 end ``` @@ -181,9 +205,15 @@ The privacy policy is defined in `.gatos/policy.yaml` and extends the policy eng ```yaml privacy: + classes: + pii_low_entropy: + min_entropy_bits: 40 + publish_plaintext_digest: false + require_ciphertext_digest: true rules: - select: "path.to.sensitive.data" action: "pointerize" + class: "pii_low_entropy" capability: "gatos-key://v1/aes-256-gcm/ops-key-01" location: "gatos-node://ed25519:" - select: "path.to.transient.data" @@ -199,6 +229,7 @@ To make privacy operations transparent and auditable, any commit that creates a ``` Privacy-Redactions: 3 Privacy-Pointers: 12 +Privacy-Pointer-Rotations: 1 ``` This provides a simple, top-level indicator that a projection has occurred, prompting auditors to look deeper if necessary. @@ -223,3 +254,33 @@ This provides a simple, top-level indicator that a projection has occurred, prom - **Large Artifact Management**: Handle large binaries (ML models, videos) without bloating the Git repository. - **Compliant Data Sharing**: Share a public, redacted dataset with third parties while retaining private access to the full, unified view. - **Federated Learning**: Different actors can hold private models locally, referenced by pointers in a public "training plan" shape. + +--- + +## Namespacing and Storage (Normative) + +- Private overlays are actor‑anchored: `refs/gatos/private///` index metadata. The local workspace mirror is `gatos/private///`. +- Private blobs themselves are NOT stored under Git refs. They live in pluggable blob stores and are addressed by their `ciphertext_digest`/`digest`. + +## Security & Privacy Notes (Normative) + +- Capability references in pointers MUST NOT contain secrets or pre‑signed tokens. Use stable identifiers and resolve sensitive data via policy. +- AES‑256‑GCM (if used) MUST include AAD composed of: actor id, pointer `content_id`, and policy version; nonces MUST be 96‑bit, randomly generated, and never reused per key. +- Right‑to‑be‑forgotten: deleting private blobs breaks pointer resolution but does not remove the public pointer. Implement erasure as a tombstone event plus an audit record. + +### Algorithm variants (experimental; private attestations only) + +- Implementations MAY use a keyed BLAKE3 variant for private attestation envelopes (not for public Opaque Pointers): `algo = "blake3-keyed"` with parameters encoded in an envelope or pointer `extensions` field. +- Recommended KDF: `hkdf-sha256`; context string `"gatos:ptr:priv:"`; derive `key = HKDF(policy_key, salt = actor_pubkey, info = context)`. +- Public pointers MUST continue to use `algo = "blake3"` for third‑party verifiability. + +## Error Taxonomy (Normative) + +Implementations SHOULD use a stable set of error codes with JSON problem details: + +- `Unauthorized` (401) +- `Forbidden` (403) +- `NotFound` (404) +- `DigestMismatch` (422) +- `CapabilityUnavailable` (503) +- `PolicyDenied` (403) diff --git a/examples/v1/policy/privacy_min.json b/examples/v1/policy/privacy_min.json new file mode 100644 index 00000000..c510c3b9 --- /dev/null +++ b/examples/v1/policy/privacy_min.json @@ -0,0 +1,21 @@ +{ + "privacy": { + "classes": { + "pii_low_entropy": { + "min_entropy_bits": 40, + "publish_plaintext_digest": false, + "require_ciphertext_digest": true + } + }, + "rules": [ + { + "select": "user.email", + "action": "pointerize", + "class": "pii_low_entropy", + "capability": "gatos-key://v1/aes-256-gcm/ops-key-01", + "location": "gatos-node://ed25519:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" + } + ] + } +} + diff --git a/examples/v1/privacy/opaque_pointer_min.json b/examples/v1/privacy/opaque_pointer_min.json new file mode 100644 index 00000000..eb3cf057 --- /dev/null +++ b/examples/v1/privacy/opaque_pointer_min.json @@ -0,0 +1,9 @@ +{ + "kind": "opaque_pointer", + "algo": "blake3", + "digest": "blake3:0000000000000000000000000000000000000000000000000000000000000000", + "ciphertext_digest": "blake3:1111111111111111111111111111111111111111111111111111111111111111", + "size": 0, + "location": "gatos-node://ed25519:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + "capability": "gatos-key://v1/aes-256-gcm/test-key-01" +} diff --git a/schemas/v1/policy/governance_policy.schema.json b/schemas/v1/policy/governance_policy.schema.json index 5c738b7d..42ff6baa 100644 --- a/schemas/v1/policy/governance_policy.schema.json +++ b/schemas/v1/policy/governance_policy.schema.json @@ -46,5 +46,39 @@ } } } + , + "privacy": { + "type": "object", + "additionalProperties": false, + "properties": { + "classes": { + "type": "object", + "additionalProperties": { + "type": "object", + "additionalProperties": false, + "properties": { + "min_entropy_bits": { "type": "integer", "minimum": 0 }, + "publish_plaintext_digest": { "type": "boolean" }, + "require_ciphertext_digest": { "type": "boolean" } + } + } + }, + "rules": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "properties": { + "select": { "type": "string" }, + "action": { "type": "string", "enum": ["redact", "pointerize"] }, + "class": { "type": "string" }, + "capability": { "type": "string", "format": "uri" }, + "location": { "type": "string", "format": "uri" } + }, + "required": ["select", "action"] + } + } + } + } } } diff --git a/schemas/v1/privacy/opaque_pointer.schema.json b/schemas/v1/privacy/opaque_pointer.schema.json index 78a4d61b..bb8d3475 100644 --- a/schemas/v1/privacy/opaque_pointer.schema.json +++ b/schemas/v1/privacy/opaque_pointer.schema.json @@ -1,46 +1,18 @@ { - "$schema": "http://json-schema.org/draft-07/schema#", + "$schema": "https://json-schema.org/draft/2020-12/schema", "title": "GATOS Opaque Pointer", - "description": "A canonical pointer to a private data blob, used to replace sensitive or large data in a public state projection.", + "description": "Canonical pointer to a private blob used in public projections.", "type": "object", "properties": { - "kind": { - "description": "The object kind, MUST be 'opaque_pointer'.", - "type": "string", - "const": "opaque_pointer" - }, - "algo": { - "description": "The hashing algorithm used for the digest, MUST be 'blake3'.", - "type": "string", - "const": "blake3" - }, - "digest": { - "description": "The content-address of the private blob, prefixed with the algorithm.", - "type": "string", - "pattern": "^blake3:[a-f0-9]{64}$" - }, - "size": { - "description": "Optional: The size of the private blob in bytes.", - "type": "integer", - "minimum": 0 - }, - "location": { - "description": "A URI indicating where the private blob can be resolved.", - "type": "string", - "format": "uri" - }, - "capability": { - "description": "A URI defining the authorization and/or decryption mechanism for the blob.", - "type": "string", - "format": "uri" - } + "kind": { "type": "string", "const": "opaque_pointer" }, + "algo": { "type": "string", "const": "blake3" }, + "digest": { "type": "string", "pattern": "^blake3:[a-f0-9]{64}$" }, + "ciphertext_digest": { "type": "string", "pattern": "^blake3:[a-f0-9]{64}$" }, + "size": { "type": "integer", "minimum": 0 }, + "location": { "type": "string", "format": "uri" }, + "capability": { "type": "string", "format": "uri" }, + "extensions": { "type": "object" } }, - "required": [ - "kind", - "algo", - "digest", - "location", - "capability" - ], + "required": ["kind","algo","digest","location","capability"], "additionalProperties": false } diff --git a/scripts/validate_schemas.sh b/scripts/validate_schemas.sh index 6f0233d1..3771b34b 100755 --- a/scripts/validate_schemas.sh +++ b/scripts/validate_schemas.sh @@ -18,6 +18,7 @@ SCHEMAS=( "schemas/v1/governance/revocation.schema.json" "schemas/v1/governance/proof_of_consensus_envelope.schema.json" "schemas/v1/policy/governance_policy.schema.json" + "schemas/v1/privacy/opaque_pointer.schema.json" ) for schema in "${SCHEMAS[@]}"; do @@ -38,6 +39,7 @@ declare -A EXAMPLES=( ["schemas/v1/governance/grant.schema.json"]="examples/v1/governance/grant_min.json" ["schemas/v1/governance/revocation.schema.json"]="examples/v1/governance/revocation_min.json" ["schemas/v1/governance/proof_of_consensus_envelope.schema.json"]="examples/v1/governance/poc_envelope_min.json" + ["schemas/v1/privacy/opaque_pointer.schema.json"]="examples/v1/privacy/opaque_pointer_min.json" ) for schema in "${!EXAMPLES[@]}"; do @@ -52,6 +54,8 @@ done echo " - ajv validate: examples/v1/policy/governance_min.json against schemas/v1/policy/governance_policy.schema.json" ajv validate "${AJV_BASE_ARGS[@]}" -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.json +echo " - ajv validate: examples/v1/policy/privacy_min.json against schemas/v1/policy/governance_policy.schema.json" +ajv validate "${AJV_BASE_ARGS[@]}" -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/privacy_min.json echo "[schemas] Additional encoding tests (ed25519 base64url forms)…" # Root schemas that reference defs using the canonical $id for proper resolution From 4bbbccb430cb39c4e254d51f3157399192257c03 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 08:30:30 -0800 Subject: [PATCH 25/68] docs: harmonize DigestMismatch to 422 across SPEC/TECH-SPEC; ADR response digest headers reflect response body\nschema(privacy): allow ciphertext-only pointers via anyOf; keep kind/algo/location/capability required\ndocs(ADR-0004): fix colon in class diagram;\ndocs(ADR-0003): PoC storage level to MUST --- docs/SPEC.md | 2 +- docs/TECH-SPEC.md | 2 +- docs/decisions/ADR-0003/DECISION.md | 2 +- docs/decisions/ADR-0004/DECISION.md | 6 +++--- schemas/v1/privacy/opaque_pointer.schema.json | 16 +++++++++++++++- 5 files changed, 21 insertions(+), 7 deletions(-) diff --git a/docs/SPEC.md b/docs/SPEC.md index c1f19549..728c4544 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -344,7 +344,7 @@ Verification Steps: 4. Servers SHOULD return `X-BLAKE3-Digest` and `Digest: sha-256=…` headers for response integrity. Error Taxonomy: -- `Unauthorized` (401), `Forbidden` (403), `NotFound` (404), `DigestMismatch` (409), `CapabilityUnavailable` (503), `PolicyDenied` (403). +- `Unauthorized` (401), `Forbidden` (403), `NotFound` (404), `DigestMismatch` (422), `CapabilityUnavailable` (503), `PolicyDenied` (403). Optional HTTP Message Signatures profile (RFC 9421): - As an alternative to JWT, clients MAY sign `@method`, `@target-uri`, `date`, `host`, `content-digest` and send `Signature-Input`/`Signature` headers. Servers SHOULD still emit `Digest` and `X-BLAKE3-Digest` response headers. diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index dfe0599e..ecddb6b9 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -212,7 +212,7 @@ The `gatosd` daemon exposes a secure endpoint for resolving Opaque Pointers. - Headers: `Digest: sha-256=`, `X-BLAKE3-Digest: blake3:` - Body: requested bytes (ciphertext or plaintext). -Errors: 401 Unauthorized, 403 Forbidden, 404 Not Found, 409 DigestMismatch, 503 CapabilityUnavailable. +Errors: 401 Unauthorized, 403 Forbidden, 404 Not Found, 422 DigestMismatch, 503 CapabilityUnavailable. Optional profile (HTTP Message Signatures, RFC 9421): - Clients MAY authenticate by signing components: `@method`, `@target-uri`, `date`, `host`, `content-digest` (SHA-256 over request body) and sending `Signature-Input: sig1=...` and `Signature: sig1=::`. diff --git a/docs/decisions/ADR-0003/DECISION.md b/docs/decisions/ADR-0003/DECISION.md index d33c69a5..3e4683ee 100644 --- a/docs/decisions/ADR-0003/DECISION.md +++ b/docs/decisions/ADR-0003/DECISION.md @@ -89,7 +89,7 @@ Define a system for gating specific GATOS actions (e.g., locking a file, publish - A lexicographically sorted list of approvals by the lowercase ASCII of each approval's `Signer` value (the `ed25519:` string). Each approval is included by value or via `Approval-Id`. - The governance rule id (`Policy-Rule`) and effective quorum parameters. - Implementations MUST use canonical JSON (UTF‑8, sorted keys, no insignificant whitespace) to build this envelope before hashing. All hex encodings MUST be lowercase. Ordering by signer is an application‑level MUST; JSON Schema cannot enforce sort order. - - Storage: The canonical PoC envelope JSON SHOULD be persisted as a blob referenced under `refs/gatos/audit/proofs/governance/`; the `Proof-Of-Consensus` trailer MUST equal `blake3(envelope_bytes)`. + - Storage: The canonical PoC envelope JSON MUST be persisted as a blob referenced under `refs/gatos/audit/proofs/governance/`; the `Proof-Of-Consensus` trailer MUST equal `blake3(envelope_bytes)`. 8. Governance schema (policy integration) - Extend `.gatos/policy.yaml` to declare governance rules (JSON Schema: [`schemas/v1/policy/governance_policy.schema.json`](../../../schemas/v1/policy/governance_policy.schema.json)): diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index 13804d5e..151b5275 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -93,7 +93,7 @@ classDiagram +string kind: "opaque_pointer" +string algo: "blake3" +string digest: "blake3:" // plaintext digest - +string ciphertext_digest "blake3:" // MAY be present + +string ciphertext_digest: "blake3:" // MAY be present +int size // SHOULD be present (bytes) +string location +string capability // MUST NOT embed secrets @@ -171,8 +171,8 @@ A client resolving an Opaque Pointer **MUST** follow this protocol: Response headers on success: ``` Content-Type: application/octet-stream -X-BLAKE3-Digest: blake3: -Digest: sha-256= +X-BLAKE3-Digest: blake3: +Digest: sha-256= ``` Optional HTTP Message Signatures profile (RFC 9421): diff --git a/schemas/v1/privacy/opaque_pointer.schema.json b/schemas/v1/privacy/opaque_pointer.schema.json index bb8d3475..9e899fc9 100644 --- a/schemas/v1/privacy/opaque_pointer.schema.json +++ b/schemas/v1/privacy/opaque_pointer.schema.json @@ -13,6 +13,20 @@ "capability": { "type": "string", "format": "uri" }, "extensions": { "type": "object" } }, - "required": ["kind","algo","digest","location","capability"], + "required": ["kind","algo","location","capability"], + "anyOf": [ + { + "required": ["digest"], + "properties": { + "digest": { "type": "string", "pattern": "^blake3:[a-f0-9]{64}$" } + } + }, + { + "required": ["ciphertext_digest"], + "properties": { + "ciphertext_digest": { "type": "string", "pattern": "^blake3:[a-f0-9]{64}$" } + } + } + ], "additionalProperties": false } From 79fde02ffeaceb886059fd4278a8a630c5a13c9e Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 10:42:48 -0800 Subject: [PATCH 26/68] docs(ADR-0005, SPEC, TECH-SPEC): formalize Shiplog + shiplog-compat; add trailer/anchors/notes; schemas + examples; regenerate diagrams MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ADR-0005: comprehensive Shiplog spec - Canonicalization: RFC 8785 JCS; Content-Id = blake3(JCS(envelope)) - Namespaces: refs/gatos/shiplog//head; add shiplog-compat (refs/_shiplog) - Commit body: headers + '---' + JSON trailer (v1) - Anchors & Notes: normative sections (+ refs) - Error taxonomy aligned with ledger-kernel (AppendRejected, TemporalOrder, PolicyFail, SigInvalid, DigestMismatch) - Mermaid diagrams and CLI examples - Schemas (+ examples): - shiplog/event_envelope, consumer_checkpoint - shiplog/deployment_trailer (v1), shiplog/anchor - privacy/opaque_pointer examples updated - SPEC Section 8 and TECH-SPEC Section 7 updated to match ADR-0005 - Include compat ingestion (jq -S -> JCS), anchors/notes, importer outline - FEATURES: add F6 — Shiplog Event Stream - CI/scripts: validate new schemas/examples in scripts/validate_schemas.sh - Diagrams: regenerate docs/diagrams/generated (no content changes besides new/updated diagrams) Note: private comparison matrix is in .obsidian/git-tech-matrix.md (gitignored by design). --- docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_1.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_10.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_11.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_12.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_13.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_14.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_15.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_16.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_17.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_18.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_19.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_2.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_20.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_21.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_3.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_4.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_5.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_6.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_7.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_8.svg | 1 + docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_9.svg | 1 + .../diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_1.svg | 1 + .../generated/docs_TECH-SPEC__15850d53f4__mermaid_10.svg | 1 + .../generated/docs_TECH-SPEC__15850d53f4__mermaid_11.svg | 1 + .../generated/docs_TECH-SPEC__15850d53f4__mermaid_12.svg | 1 + .../generated/docs_TECH-SPEC__15850d53f4__mermaid_13.svg | 1 + .../generated/docs_TECH-SPEC__15850d53f4__mermaid_14.svg | 1 + .../generated/docs_TECH-SPEC__15850d53f4__mermaid_15.svg | 1 + .../generated/docs_TECH-SPEC__15850d53f4__mermaid_16.svg | 1 + .../diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_2.svg | 1 + .../diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_3.svg | 1 + .../diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_4.svg | 1 + .../diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_5.svg | 1 + .../diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_6.svg | 1 + .../diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_7.svg | 1 + .../diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_8.svg | 1 + .../diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_9.svg | 1 + .../docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_1.svg | 1 + .../docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_2.svg | 1 + .../docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_3.svg | 1 + .../docs_decisions_ADR-0002_DECISION__a21cc79f92__mermaid_1.svg | 1 + .../docs_decisions_ADR-0002_DECISION__a21cc79f92__mermaid_2.svg | 1 + .../docs_decisions_ADR-0003_DECISION__3c4445d569__mermaid_1.svg | 1 + .../docs_decisions_ADR-0003_DECISION__3c4445d569__mermaid_2.svg | 1 + .../docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_1.svg | 1 + .../docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_2.svg | 1 + .../docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_3.svg | 1 + .../docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_4.svg | 1 + .../docs_diagrams_api_endpoints__578bd81e4d__mermaid_1.svg | 1 + .../docs_diagrams_api_endpoints__578bd81e4d__mermaid_2.svg | 1 + .../docs_diagrams_architecture__105fc24d87__mermaid_1.svg | 1 + .../generated/docs_diagrams_data_flow__559f0c180d__mermaid_1.svg | 1 + .../docs_diagrams_state_management__86fd1ffb65__mermaid_1.svg | 1 + scripts/mermaid/generate_all.sh | 0 54 files changed, 53 insertions(+) create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_1.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_10.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_11.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_12.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_13.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_14.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_15.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_16.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_17.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_18.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_19.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_2.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_20.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_21.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_3.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_4.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_5.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_6.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_7.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_8.svg create mode 100644 docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_9.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_1.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_10.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_11.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_12.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_13.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_14.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_15.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_16.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_2.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_3.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_4.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_5.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_6.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_7.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_8.svg create mode 100644 docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_9.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_1.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_2.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_3.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0002_DECISION__a21cc79f92__mermaid_1.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0002_DECISION__a21cc79f92__mermaid_2.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0003_DECISION__3c4445d569__mermaid_1.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0003_DECISION__3c4445d569__mermaid_2.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_1.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_2.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_3.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_4.svg create mode 100644 docs/diagrams/generated/docs_diagrams_api_endpoints__578bd81e4d__mermaid_1.svg create mode 100644 docs/diagrams/generated/docs_diagrams_api_endpoints__578bd81e4d__mermaid_2.svg create mode 100644 docs/diagrams/generated/docs_diagrams_architecture__105fc24d87__mermaid_1.svg create mode 100644 docs/diagrams/generated/docs_diagrams_data_flow__559f0c180d__mermaid_1.svg create mode 100644 docs/diagrams/generated/docs_diagrams_state_management__86fd1ffb65__mermaid_1.svg mode change 100644 => 100755 scripts/mermaid/generate_all.sh diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_1.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_1.svg new file mode 100644 index 00000000..f24f93ba --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_1.svg @@ -0,0 +1 @@ +
RFC 2119
Keywords
MUST
SHOULD
MAY
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_10.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_10.svg new file mode 100644 index 00000000..08b8917b --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_10.svg @@ -0,0 +1 @@ +
merge
undo
fork
undo
main
commit-a
commit-b
session-2
session-1
main
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_11.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_11.svg new file mode 100644 index 00000000..81877d3d --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_11.svg @@ -0,0 +1 @@ +
ProofEnvelope
+String type
+String ulid
+String inputs_root
+String output_root
+String policy_root
+String proof
+String sig
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_12.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_12.svg new file mode 100644 index 00000000..5de93309 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_12.svg @@ -0,0 +1 @@ +PeerBPeerAPeerBPeerAPeers are offline and make divergent changesalt[Policies are comparable][Policies areincomparable]Reconnect & Exchange EnvelopesValidate Signatures & Policy AncestryPrefer descendant policyAppend governance.conflict event diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_13.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_13.svg new file mode 100644 index 00000000..ef05eede --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_13.svg @@ -0,0 +1 @@ +
GATOS
local
push-gate
saas-hosted
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_14.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_14.svg new file mode 100644 index 00000000..09f7b59b --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_14.svg @@ -0,0 +1 @@ +
Exposes
Scrapes
Diagnoses
gatosd
/metrics
Prometheus
gatos doctor
Ref Invariants
Epoch Continuity
Cache Staleness
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_15.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_15.svg new file mode 100644 index 00000000..7d34e01c --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_15.svg @@ -0,0 +1 @@ +
Access Control
Requests access to
Links
Links
Evaluates request for
Resource
Actor
Capability Grant
Policy
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_16.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_16.svg new file mode 100644 index 00000000..030fa4e5 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_16.svg @@ -0,0 +1 @@ +
Triggers
Prunes
Epoch N
Epoch N+1 Anchor
Compaction
Unreferenced Blobs in Epoch N
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_17.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_17.svg new file mode 100644 index 00000000..cbe16f08 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_17.svg @@ -0,0 +1 @@ +
GATOS Implementation
Certification
Deterministic Fold
Exactly-Once Delivery
Offline Reconcile
Deny Audit
Blob Integrity
Consensus Integrity
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_18.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_18.svg new file mode 100644 index 00000000..9e54f7b0 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_18.svg @@ -0,0 +1 @@ +
git gatos
init
session
event
fold
bus
policy
trust
epoch
prove
doctor
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_19.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_19.svg new file mode 100644 index 00000000..70ebdf51 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_19.svg @@ -0,0 +1 @@ +Workergatos-echogatos-mindgatos-ledgergatosdClientWorkergatos-echogatos-mindgatos-ledgergatosdClientLater, a worker consumes the job...A fold process runs...1. Enqueue Job (Event)2. Append `jobs.enqueue` event3. Success4. Publish `gmb.msg` to topic5. Success6. Job Enqueued7. Subscribe to topic8. Delivers `gmb.msg`9. Report Result (Event)10. Append `jobs.result` event11. Success12. Write `gmb.ack`13. Result Recorded14. Read events from journal15. Compute new state (e.g., update queue view)16. Checkpoint new state diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_2.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_2.svg new file mode 100644 index 00000000..646b6118 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_2.svg @@ -0,0 +1 @@ +
GATOS System
User / Client
Policy Plane
State Plane
Message Plane
Job Plane
Ledger Plane
gatosd (Daemon)
gatos-ledger
gatos-compute (planned)
gatos-mind
gatos-echo
gatos-kv
gatos-policy
gatosd (CLI)
Client SDK
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_20.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_20.svg new file mode 100644 index 00000000..a241d1d0 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_20.svg @@ -0,0 +1 @@ +
Worker claims job (CAS)
Worker begins execution
`jobs.result` (ok)
`jobs.result` (fail)
Canceled by user/policy
Canceled by user/policy
pending
claimed
running
succeeded
failed
aborted
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_21.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_21.svg new file mode 100644 index 00000000..ff575db4 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_21.svg @@ -0,0 +1 @@ +
approval received
additional approvals
ttl elapsed
ttl elapsed
quorum satisfied
revocation committed
proposal
partial
expired
granted
revoked
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_3.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_3.svg new file mode 100644 index 00000000..7bfd07de --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_3.svg @@ -0,0 +1 @@ +
Workspace
policies
gatos
schema
folds
trust
objects
.git
gatos
refs
journal
state
mbus
jobs
sessions
audit
cache
epoch
private
notes
gatos
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_4.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_4.svg new file mode 100644 index 00000000..051c2670 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_4.svg @@ -0,0 +1 @@ +
issuer
1
1
subject
1
1
Grant
+String ulid
+String issuer
+String subject
+String[] caps
+Date exp
+String sig
«enumeration»
Actor
user
agent
service
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_5.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_5.svg new file mode 100644 index 00000000..4a0f3693 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_5.svg @@ -0,0 +1 @@ +
EventEnvelope
+String type
+String ulid
+String actor
+String[] caps
+Object payload
+String policy_root
+String sig
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_6.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_6.svg new file mode 100644 index 00000000..005f6e3d --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_6.svg @@ -0,0 +1 @@ +
Event Stream
Fold Function
Policy
State Root
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_7.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_7.svg new file mode 100644 index 00000000..9c16e272 --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_7.svg @@ -0,0 +1 @@ +PolicyGateGATOSClientPolicyGateGATOSClientalt[Action is Allowed][Action is Denied]Propose Action (Intent)Evaluate(Intent, Context)Decision: AllowBind policy_root to eventSuccessDecision: Deny(reason)Write Audit DecisionFailure(reason) diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_8.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_8.svg new file mode 100644 index 00000000..579db54a --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_8.svg @@ -0,0 +1 @@ +
OpaquePointer
+string kind: "opaque_pointer"
+string algo: "blake3"
+string digest: "blake3:" // plaintext digest
+string ciphertext_digest: "blake3:" // optional
+int size // bytes; SHOULD be present
+string location
+string capability // MUST NOT embed secrets
+object extensions // forward-compatible
 diff --git a/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_9.svg b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_9.svg new file mode 100644 index 00000000..6e2e4e8f --- /dev/null +++ b/docs/diagrams/generated/docs_SPEC__0679d036ea__mermaid_9.svg @@ -0,0 +1 @@ +ConsumerGATOSPublisherConsumerGATOSPublisherPublish Message (QoS: exactly_once)Deliver MessageProcess MessageSend AckObserve Ack QuorumCreate gmb.commit Event diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_1.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_1.svg new file mode 100644 index 00000000..2a48ce81 --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_1.svg @@ -0,0 +1 @@ +
gatos
gatos-ledger-core
crates
gatos-ledger-git
gatos-mind
gatos-echo
gatos-policy
gatos-kv
gatosd
gatos-compute
bindings
wasm
ffi
 diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_10.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_10.svg new file mode 100644 index 00000000..d41c9ca6 --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_10.svg @@ -0,0 +1 @@ +LibsodiumGATOSClientLibsodiumGATOSClientalt[Signature is Valid][Signature is Invalid]Submit Signed EventCanonicalize JSONed25519_verify(signature, payload, pubkey)OKProcess EventFailReject Event diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_11.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_11.svg new file mode 100644 index 00000000..77c3feba --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_11.svg @@ -0,0 +1 @@ +Batch Size Trade-off (Illustrative)11.522.533.544.55Batch Size16014012010080604020Metric diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_12.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_12.svg new file mode 100644 index 00000000..a3c613c3 --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_12.svg @@ -0,0 +1 @@ +
JSONL RPC
JSONL RPC
JSONL RPC
JSONL RPC
gatosd
Go SDK
Python SDK
Rust SDK
Node.js SDK
 diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_13.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_13.svg new file mode 100644 index 00000000..fc80a41d --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_13.svg @@ -0,0 +1 @@ +2025-01-052025-01-122025-01-192025-01-262025-02-022025-02-092025-02-162025-02-232025-03-022025-03-092025-03-162025-03-232025-03-302025-04-06Mirror Mode Shadow Consumers Canary (10%) Full Cutover Phase A: MirrorPhase B: ShadowPhase C: Dual-ReadPhase D: CutoverGATOS Migration Strategy diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_14.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_14.svg new file mode 100644 index 00000000..ee050bcd --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_14.svg @@ -0,0 +1 @@ +
Encodes
«Rust»
BincodeConfig
+standard()
Hash
+[u8; 32]
 diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_15.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_15.svg new file mode 100644 index 00000000..ad6fb484 --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_15.svg @@ -0,0 +1 @@ +WorkerBus (Message Plane)GATOS (Ledger)ClientWorkerBus (Message Plane)GATOS (Ledger)Client1. Create Job Commit2. Publish Job message3. Subscribe to job topic4. Receive Job message5. Atomically create Claim ref6. Claim successful7. Execute Job8. Create Result commit diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_16.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_16.svg new file mode 100644 index 00000000..4af2e2db --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_16.svg @@ -0,0 +1 @@ +Approver (via CLI)Message BusPolicy EngineGATOS (Ledger)ClientApprover (via CLI)Message BusPolicy EngineGATOS (Ledger)Clientloop[Approvals]alt[Quorum satisfied][Not yet satisfied]1. Create Proposal (Action, Target, Quorum)2. Validate proposal3. Accepted4. Publish proposal.created5. Create Approval (Signer, Proposal-Id)6. Verify signature + eligibility7. Approval valid8. Check quorum9. Create Grant (Proof-Of-Consensus)10. Publish grant.createdPending (partial) diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_2.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_2.svg new file mode 100644 index 00000000..12a405ec --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_2.svg @@ -0,0 +1 @@ +
GATOS System
User / Client
Policy Plane
State Plane
Message Plane
Job Plane
Ledger Plane
gatosd (Daemon)
gatos-ledger
gatos-compute
gatos-mind
gatos-echo
gatos-kv
gatos-policy
gatosd (CLI)
Client SDK
 diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_3.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_3.svg new file mode 100644 index 00000000..5d2eb6fc --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_3.svg @@ -0,0 +1 @@ +
uses
blake3
Canonical Events
FoldEngine
rmg-core
Canonical JSON Tree
state_root
 diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_4.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_4.svg new file mode 100644 index 00000000..43a4bc94 --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_4.svg @@ -0,0 +1 @@ +
folded by
produces
stored in
Journal Events
Indexer
Roaring Bitmap
refs/gatos/cache/
 diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_5.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_5.svg new file mode 100644 index 00000000..c92bfdd1 --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_5.svg @@ -0,0 +1 @@ +GATOSUserGATOSUsergatos epoch new <ns>Create new anchor at refs/gatos/epoch/<ns>/<epoch-id>Start CompactorWalk reachability from state_rootPrune unreferenced blobs diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_6.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_6.svg new file mode 100644 index 00000000..7298acaa --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_6.svg @@ -0,0 +1 @@ +LedgerPrivateStorePolicyEcho"StorageBackend (Interface)"gatos-ledgergatos-policygatos-echoLedgerPrivateStorePolicyEcho"StorageBackend (Interface)"gatos-ledgergatos-policygatos-echoalt[rule matches (e.g., "pointerize")]loop[for each field path in the UnifiedState tree]1. Fold event history to produce UnifiedState2. Request privacy rules for the current context3. Return `select` and `action` rules4. Match field path against rules5. Generate Opaque Pointer envelope6. Store original node value as private blob, keyed by its blake3 digest7. Replace node in state tree with pointer8. Commit the final PublicState tree diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_7.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_7.svg new file mode 100644 index 00000000..c227ee9c --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_7.svg @@ -0,0 +1 @@ +gatosdClient (SDK/CLI)gatosdClient (SDK/CLI)loop[Subscription Stream]{"type":"append_event", "id":"01A", "ns":"...", "event":{...}}{"ok":true, "id":"01A", "commit_id":"..."}{"type":"bus.subscribe", "id":"01C", "topic":"..."}{"ack":true, "id":"01C"}{"type":"bus.message", "id":"01C", "topic":"...", "payload":{...}} diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_8.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_8.svg new file mode 100644 index 00000000..cfed5e05 --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_8.svg @@ -0,0 +1 @@ +
Metrics
gatosd
gatos_journal_append_latency_ms
gatos_fold_latency_ms
gatos_bus_ack_lag
Journal
Fold Engine
Message Bus
 diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_9.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_9.svg new file mode 100644 index 00000000..a814a1e4 --- /dev/null +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_9.svg @@ -0,0 +1 @@ +
CI Pipeline
Test Matrix
linux-amd64-glibc
macOS-arm64
Windows-amd64
Test Suites
Golden Vectors
Torture Tests
Reconcile Harness
Projection Determinism
 diff --git a/docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_1.svg b/docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_1.svg new file mode 100644 index 00000000..bc8be047 --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_1.svg @@ -0,0 +1 @@ +
feature: git2-backend
feature: core-only
Consumer Crate
gatos-ledger (meta-crate)
gatos-ledger-git (std)
gatos-ledger-core (no_std)
 diff --git a/docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_2.svg b/docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_2.svg new file mode 100644 index 00000000..5c5b303d --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_2.svg @@ -0,0 +1 @@ +ObjectStore Backendgatos-ledger-coreApplicationObjectStore Backendgatos-ledger-coreApplicationcreate_commit(parent, tree, signature)commit = Commit { ... }hash = compute_commit_id(&commit)put_object(hash, serialize(commit))(persists object)return hash diff --git a/docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_3.svg b/docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_3.svg new file mode 100644 index 00000000..27bb4633 --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0001_DECISION__b1ba19490c__mermaid_3.svg @@ -0,0 +1 @@ +
default-features = true
default-features = false,
features = [core-only]
Consumer Crate
uses gatos-ledger
Feature: 'git2-backend'
Includes `gatos-ledger-git`
Includes `gatos-ledger-core`
Provides `GitStore` impl of `ObjectStore`
Feature: 'core-only'
Includes `gatos-ledger-core` only
Provides `ObjectStore` trait and core types
 diff --git a/docs/diagrams/generated/docs_decisions_ADR-0002_DECISION__a21cc79f92__mermaid_1.svg b/docs/diagrams/generated/docs_decisions_ADR-0002_DECISION__a21cc79f92__mermaid_1.svg new file mode 100644 index 00000000..7b3f5060 --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0002_DECISION__a21cc79f92__mermaid_1.svg @@ -0,0 +1 @@ +
Worker discovers & claims job
Worker begins execution
Execution successful
Execution fails
Canceled by user/policy
Canceled by user/policy
pending
claimed
running
succeeded
failed
aborted
 diff --git a/docs/diagrams/generated/docs_decisions_ADR-0002_DECISION__a21cc79f92__mermaid_2.svg b/docs/diagrams/generated/docs_decisions_ADR-0002_DECISION__a21cc79f92__mermaid_2.svg new file mode 100644 index 00000000..df7a8c48 --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0002_DECISION__a21cc79f92__mermaid_2.svg @@ -0,0 +1 @@ +WorkerBus (Message Plane)GATOS (Ledger)ClientWorkerBus (Message Plane)GATOS (Ledger)Clientalt[Claim already exists by another worker][Claim created]1. Create Job Commit2. Publish Job message3. Subscribe to job topic4. Receive Job message5. Atomically create Claim ref (by job-id)6. Claim failed (CAS)7. Backoff and retry6. Claim successful7. Execute Job8. Create Result commit (with Job-Id trailer) diff --git a/docs/diagrams/generated/docs_decisions_ADR-0003_DECISION__3c4445d569__mermaid_1.svg b/docs/diagrams/generated/docs_decisions_ADR-0003_DECISION__3c4445d569__mermaid_1.svg new file mode 100644 index 00000000..ff575db4 --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0003_DECISION__3c4445d569__mermaid_1.svg @@ -0,0 +1 @@ +
approval received
additional approvals
ttl elapsed
ttl elapsed
quorum satisfied
revocation committed
proposal
partial
expired
granted
revoked
 diff --git a/docs/diagrams/generated/docs_decisions_ADR-0003_DECISION__3c4445d569__mermaid_2.svg b/docs/diagrams/generated/docs_decisions_ADR-0003_DECISION__3c4445d569__mermaid_2.svg new file mode 100644 index 00000000..eeb29207 --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0003_DECISION__3c4445d569__mermaid_2.svg @@ -0,0 +1 @@ +ApproverMessage BusPolicy EngineGATOS (Ledger)ClientApproverMessage BusPolicy EngineGATOS (Ledger)Clientloop[Approvals]alt[Quorum satisfied][Not yet satisfied]1. Create Proposal (Action, Target, Quorum)2. Validate proposal3. Accepted4. Publish proposal.created5. Create Approval (Signer, Proposal-Id)6. Verify signature + eligibility7. Approval valid8. Check quorum9. Create Grant (Proof-Of-Consensus)10. Publish grant.createdPending (partial) diff --git a/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_1.svg b/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_1.svg new file mode 100644 index 00000000..a4c30efe --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_1.svg @@ -0,0 +1 @@ +
Sh_Private
Sh_Public
Sh_Unified
Commit c
Proj(c)
Proj
Proj
Extract
Extract
Private Blobs 1
Private Blobs 2
Public Shape 1
Public Shape 2
Unified Shape 1
Unified Shape 2
 diff --git a/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_2.svg b/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_2.svg new file mode 100644 index 00000000..9faec20a --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_2.svg @@ -0,0 +1 @@ +
OpaquePointer
+string kind: "opaque_pointer"
+string algo: "blake3"
+string digest: "blake3:" // plaintext digest
+string ciphertext_digest: "blake3:" // MAY be present
+string location
+string capability // MUST NOT embed secrets
+object extensions // forward-compatible
+int size // SHOULD be present(bytes)
 diff --git a/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_3.svg b/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_3.svg new file mode 100644 index 00000000..0e9fcfc8 --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_3.svg @@ -0,0 +1 @@ +Private StoreLedger (Git)Policy EngineState Engine (gatos-echo)Private StoreLedger (Git)Policy EngineState Engine (gatos-echo)1. Fold history into UnifiedState2. Fetch privacy rules3. Return rules (redact/pointerize)4. Apply rules to create PublicState + PrivateBlobs5. Commit PublicState to public refs6. Store PrivateBlobs by digest diff --git a/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_4.svg b/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_4.svg new file mode 100644 index 00000000..df225c1c --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0004_DECISION__12d7f53080__mermaid_4.svg @@ -0,0 +1 @@ +Key Management ServicePrivate GATOS NodeClientKey Management ServicePrivate GATOS NodeClientalt[Authorized][Unauthorized]1. Read OpaquePointer2. POST /gatos/private/blobs/resolve (Authorization: Bearer <JWT>)3. Check policy (is C allowed?)4. Return encrypted blob5. Request key for {capability}6. Return decryption key7. Decrypt blob8. Verify blake3(decrypted) == digest4. Return 401/403 diff --git a/docs/diagrams/generated/docs_diagrams_api_endpoints__578bd81e4d__mermaid_1.svg b/docs/diagrams/generated/docs_diagrams_api_endpoints__578bd81e4d__mermaid_1.svg new file mode 100644 index 00000000..e0680fa8 --- /dev/null +++ b/docs/diagrams/generated/docs_diagrams_api_endpoints__578bd81e4d__mermaid_1.svg @@ -0,0 +1 @@ +gatosdClient (SDK/CLI)gatosdClient (SDK/CLI)loop[Subscription Stream]{"type":"append_event", "id":"01A", "ns":"...", "event":{...}}{"ok":true, "id":"01A", "commit_id":"..."}{"type":"bus.publish", "id":"01B", "topic":"...", "payload":{...}}{"ok":true, "id":"01B", "msg_id":"..."}{"type":"bus.subscribe", "id":"01C", "topic":"..."}{"ack":true, "id":"01C"}{"type":"bus.message", "id":"01C", "topic":"...", "payload":{...}}{"type":"fold_state", "id":"01D", "ns":"..."}{"ok":true, "id":"01D", "state_root":"..."} diff --git a/docs/diagrams/generated/docs_diagrams_api_endpoints__578bd81e4d__mermaid_2.svg b/docs/diagrams/generated/docs_diagrams_api_endpoints__578bd81e4d__mermaid_2.svg new file mode 100644 index 00000000..5fe476e7 --- /dev/null +++ b/docs/diagrams/generated/docs_diagrams_api_endpoints__578bd81e4d__mermaid_2.svg @@ -0,0 +1 @@ +gatosdClient (SDK/CLI)gatosdClient (SDK/CLI){"type":"append_event", "id":"02A", "ns":"invalid", "event":{...}}{"ok":false, "id":"02A", "error":{"code":"ERR_INVALID_NS", "message":"namespace not found"}} diff --git a/docs/diagrams/generated/docs_diagrams_architecture__105fc24d87__mermaid_1.svg b/docs/diagrams/generated/docs_diagrams_architecture__105fc24d87__mermaid_1.svg new file mode 100644 index 00000000..12a405ec --- /dev/null +++ b/docs/diagrams/generated/docs_diagrams_architecture__105fc24d87__mermaid_1.svg @@ -0,0 +1 @@ +
GATOS System
User / Client
Policy Plane
State Plane
Message Plane
Job Plane
Ledger Plane
gatosd (Daemon)
gatos-ledger
gatos-compute
gatos-mind
gatos-echo
gatos-kv
gatos-policy
gatosd (CLI)
Client SDK
 diff --git a/docs/diagrams/generated/docs_diagrams_data_flow__559f0c180d__mermaid_1.svg b/docs/diagrams/generated/docs_diagrams_data_flow__559f0c180d__mermaid_1.svg new file mode 100644 index 00000000..70ebdf51 --- /dev/null +++ b/docs/diagrams/generated/docs_diagrams_data_flow__559f0c180d__mermaid_1.svg @@ -0,0 +1 @@ +Workergatos-echogatos-mindgatos-ledgergatosdClientWorkergatos-echogatos-mindgatos-ledgergatosdClientLater, a worker consumes the job...A fold process runs...1. Enqueue Job (Event)2. Append `jobs.enqueue` event3. Success4. Publish `gmb.msg` to topic5. Success6. Job Enqueued7. Subscribe to topic8. Delivers `gmb.msg`9. Report Result (Event)10. Append `jobs.result` event11. Success12. Write `gmb.ack`13. Result Recorded14. Read events from journal15. Compute new state (e.g., update queue view)16. Checkpoint new state diff --git a/docs/diagrams/generated/docs_diagrams_state_management__86fd1ffb65__mermaid_1.svg b/docs/diagrams/generated/docs_diagrams_state_management__86fd1ffb65__mermaid_1.svg new file mode 100644 index 00000000..080036d4 --- /dev/null +++ b/docs/diagrams/generated/docs_diagrams_state_management__86fd1ffb65__mermaid_1.svg @@ -0,0 +1 @@ +
Worker consumes `bus.message`
`jobs.result` (ok) event recorded
`jobs.result` (fail) event recorded
`attempts` < max_retries
`attempts` >= max_retries
Job is re-published
Enqueued
Processing
Succeeded
Failed
Retrying
DeadLetterQueue
 diff --git a/scripts/mermaid/generate_all.sh b/scripts/mermaid/generate_all.sh old mode 100644 new mode 100755 From ea6c357b50e0024bc38f47bf9334a40693ff3472 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 11:01:48 -0800 Subject: [PATCH 27/68] docs/schemas(ADR-0005): add Shiplog ADR text, shiplog schemas and examples; update schema validator --- docs/decisions/ADR-0005/DECISION.md | 184 ++++++++++++++++++ examples/v1/shiplog/anchor_min.json | 8 + examples/v1/shiplog/checkpoint_min.json | 5 + examples/v1/shiplog/event_min.json | 10 + examples/v1/shiplog/trailer_min.json | 9 + schemas/v1/shiplog/anchor.schema.json | 17 ++ .../shiplog/consumer_checkpoint.schema.json | 13 ++ .../v1/shiplog/deployment_trailer.schema.json | 25 +++ schemas/v1/shiplog/event_envelope.schema.json | 21 ++ scripts/validate_schemas.sh | 8 + 10 files changed, 300 insertions(+) create mode 100644 docs/decisions/ADR-0005/DECISION.md create mode 100644 examples/v1/shiplog/anchor_min.json create mode 100644 examples/v1/shiplog/checkpoint_min.json create mode 100644 examples/v1/shiplog/event_min.json create mode 100644 examples/v1/shiplog/trailer_min.json create mode 100644 schemas/v1/shiplog/anchor.schema.json create mode 100644 schemas/v1/shiplog/consumer_checkpoint.schema.json create mode 100644 schemas/v1/shiplog/deployment_trailer.schema.json create mode 100644 schemas/v1/shiplog/event_envelope.schema.json diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md new file mode 100644 index 00000000..b7564115 --- /dev/null +++ b/docs/decisions/ADR-0005/DECISION.md @@ -0,0 +1,184 @@ +--- +Status: Proposed +Date: 2025-11-10 +ADR: ADR-0005 +Authors: [flyingrobots] +Requires: [ADR-0001, ADR-0004] +Related: [ADR-0002, ADR-0003] +Tags: [Shiplog, Event Stream, Consumers, JCS, ULID] +Schemas: + - ../../../../schemas/v1/shiplog/event_envelope.schema.json + - ../../../../schemas/v1/shiplog/consumer_checkpoint.schema.json + - ../../../../schemas/v1/shiplog/deployment_trailer.schema.json + - ../../../../schemas/v1/shiplog/anchor.schema.json + - ../../../../schemas/v1/privacy/opaque_pointer.schema.json +Supersedes: [] +Superseded-By: [] +--- + +# ADR‑0005: Shiplog — A Parallel, Queryable Event Stream + +## Summary / Scope + +Introduce a first‑class, append‑only event stream ("Shiplog") that runs in parallel with deterministic state folds. The Shiplog provides per‑topic ordering, canonical event envelopes, consumer checkpoints, and query APIs. It is privacy‑aware (ADR‑0004) and deterministic (Morphology Calculus). + +## Context / Problem + +Many integrations require an append‑only stream rather than only snapshot state: analytics, external system replay, audit feeds, and incremental ETL. SPEC v0.3 defines append‑only journals conceptually but lacks a normative, queryable stream with consumer checkpoints and a canonical envelope format. This ADR makes the Shiplog normative. + +## Decision (Normative) + +### 1) Canonicalization and Identifiers + +- Envelope canonicalization: RFC 8785 JSON Canonicalization Scheme (JCS). The event Content‑Id is `blake3(JCS(envelope))`. +- ULID: 26‑char Crockford base32, uppercase, excluding I/L/O/U (`^[0-9A-HJKMNP-TV-Z]{26}$`). +- Hashes: content digests are `blake3:<64‑hex>` per `schemas/v1/common/ids.schema.json`. + +```mermaid +classDiagram + class EventEnvelope { + +string ulid + +string ns // topic namespace (e.g., "governance") + +string type // logical event type + +object payload // canonical JSON (JCS) + +map refs // OPTIONAL cross-refs + } +``` + +### 2) Namespaces and Ordering + +- Per‑topic head ref (append‑only, linear): `refs/gatos/shiplog//head` +- Topic naming: `^[a-z][a-z0-9._-]{0,63}$` (ASCII, lowercase start). +- Ordering per topic is the Git parent chain. Appends MUST be fast‑forward (CAS on ref update). On a single node, ULIDs MUST increase strictly per topic. + +```mermaid +graph TD + subgraph "Git Refs" + H1[refs/gatos/shiplog/orders/head]-->C1 + C1((e1))-->C2((e2))-->C3((e3)) + end + C1:::ev; C2:::ev; C3:::ev + classDef ev fill:#cde,stroke:#335; +``` + +### 3) Event Envelope (Schema) + +- Canonical JSON envelope at `schemas/v1/shiplog/event_envelope.schema.json` (draft‑2020‑12). +- Required fields: `ulid`, `ns`, `type`, `payload`. +- Optional `refs` (map) to link related state or IDs. +- Privacy (ADR‑0004): Payload MUST NOT embed private overlay data. Redacted values MUST be replaced by `OpaquePointer` envelopes per `schemas/v1/privacy/opaque_pointer.schema.json`. + +### 4) Commit Message and Trailer + +Each Shiplog commit MUST include headers in the commit message (any order), followed by a single line containing three dashes `---` and then a JSON trailer object: + +``` +Event-Id: ulid: +Content-Id: blake3:<64-hex> +Topic: +Schema: https://gatos.dev/schemas/v1/shiplog/event_envelope.schema.json +--- +{ "version": 1, + "env": "prod", + "who": { "name": "Jane Dev", "email": "jane@example.com" }, + "what": { "service": "web", "artifact": "ghcr.io/acme/web:1.2.3" }, + "where": { "region": "us-east-1", "cluster": "eks-a", "namespace": "prod" }, + "why": { "reason": "canary", "ticket": "OPS-123" }, + "how": { "pipeline": "gha", "run_url": "https://github.com/..." }, + "status": "success", + "when": { "start_ts": "2025-11-10T10:00:00Z", "end_ts": "2025-11-10T10:01:10Z", "dur_s": 70 }, + "seq": 42, + "journal_parent": "", + "trust_oid": "", + "previous_anchor": "", + "repo_head": "" +} +``` + +Trailer schema: `schemas/v1/shiplog/deployment_trailer.schema.json`. + +### 5) Append Semantics + +Append(`topic`, `envelope`): validate schema; compute `content_id = blake3(JCS(envelope))`; enforce monotone ULID per topic on this node; create commit with headers + trailer; CAS update `refs/gatos/shiplog//head`; return `(commit_oid, ulid, content_id)`. + +Errors (normative): +- 400 `InvalidEnvelope`; 409 `UlidOutOfOrder`; 409 `NotFastForward`; 422 `DigestMismatch`. + +### 6) Query Semantics + +- `shiplog.read(topic, since_ulid, limit) -> [ (ulid, content_id, commit_oid, envelope) ]` (increasing ULID order). +- `shiplog.tail(topics[], limit_per_topic)` MAY multiplex without cross‑topic causality guarantees. + +### 7) Consumer Checkpoints + +- `refs/gatos/consumers//` points to the last processed Shiplog commit OID. Portable JSON (optional): `schemas/v1/shiplog/consumer_checkpoint.schema.json`. + +### 8) Privacy Interactions (ADR‑0004) + +- Payloads MUST NOT embed private overlay data. Use Opaque Pointers per privacy schema. For low‑entropy classes, include `ciphertext_digest` and omit plaintext digest in public pointers. + +### 9) Governance and Ledger Interactions + +- Governance (ADR‑0003): Should emit Shiplog events under `topic="governance"`. +- Ledger mirroring: MAY mirror ledger events; must preserve envelope determinism. + +### 10) Security Considerations + +- No secrets in commit messages or payloads. Use capability URIs; notes/logs may be private or pointerized. +- Idempotent appends; checkpoints are advisory. + +### 11) CLI Examples + +```bash +$ gatosd shiplog append --topic governance --file event.json +ok commit=8b1c1e4 content_id=blake3:2a6c... ulid=01HF4Y9Q1SM8Q7K9DK2R3V4AWB + +$ gatosd shiplog read --topic governance --since 01HF4Y9Q1SM8Q7K9DK2R3V4AWB --limit 2 +01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c... 8b1c1e4 {"ulid":"01HF4Y9...","ns":"governance",...} +01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2... 9f0aa21 {"ulid":"01HF4Y9...","ns":"governance",...} + +$ gatosd shiplog checkpoint set --group analytics --topic governance --commit 8b1c1e4 +ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 +``` + +## Consequences + +Pros: clean integration surface; deterministic envelopes; replay + analytics; explicit privacy. +Cons: additional refs to manage; potential duplication if mirroring ledger events. + +## Migration / Rollout + +1. Add schemas + CI wiring. +2. Implement gatos‑mind adapter and gatosd CLI/RPC. +3. Emit governance events. + +## Test Plan (Property + Integration) + +- Determinism; ordering; idempotence; query pagination; checkpoints; privacy envelopes. + +## Documentation Updates + +- SPEC and TECH‑SPEC sections updated; FEATURES include F6 — Shiplog Event Stream. + +## References + +- ADR‑0001, ADR‑0003, ADR‑0004. RFC 8785 JCS. + +--- + +## Compatibility Profile: `shiplog-compat` + +To interoperate with existing bash‑based producers (e.g., `git shiplog`), implementations MUST support a compatibility profile: + +- `ref_root = refs/_shiplog`; Journals: `journal/`; Anchors: `anchors/`; Notes: `notes/logs`; Consumers (optional mirror): `consumers//`. + +Commit body conventions are identical: header lines, a single `---` separator, then a JSON trailer object. Envelopes MAY be present in the commit tree for auditability. + +Canonicalization (ingestion): Content‑Id remains `blake3(JCS(envelope))`. If an existing producer created compact, key‑sorted JSON via `jq -cS .`, readers MUST parse and re‑canonicalize to JCS before hashing. Producers SHOULD emit JCS bytes. + +Anchors and Notes: Anchor commits MAY be written periodically to capture rollup points. Attachments/logs SHOULD be stored as Git notes; redact or pointerize as needed. + +Error taxonomy (aligned with Ledger‑Kernel): `AppendRejected`, `TemporalOrder`, `PolicyFail`, `SigInvalid`, `DigestMismatch`. + +Importer (recommended): mirror from `refs/_shiplog/*` to `refs/gatos/shiplog/*`, re‑canonicalizing to JCS and preserving commit authorship/timestamps. + diff --git a/examples/v1/shiplog/anchor_min.json b/examples/v1/shiplog/anchor_min.json new file mode 100644 index 00000000..2a756c98 --- /dev/null +++ b/examples/v1/shiplog/anchor_min.json @@ -0,0 +1,8 @@ +{ + "ulid": "01HF4Y9Q1SM8Q7K9DK2R3V4AWB", + "topic": "prod", + "head": "8b1c1e4f3c9a0b5d7e2c1a4f6b8d0c3e5f7a9b1c", + "label": "q4-rollup", + "created_at": "2025-11-10T10:05:00Z" +} + diff --git a/examples/v1/shiplog/checkpoint_min.json b/examples/v1/shiplog/checkpoint_min.json new file mode 100644 index 00000000..52d790f2 --- /dev/null +++ b/examples/v1/shiplog/checkpoint_min.json @@ -0,0 +1,5 @@ +{ + "ulid": "01HF4Y9Q1SM8Q7K9DK2R3V4AWB", + "commit_oid": "8b1c1e4f3c9a0b5d7e2c1a4f6b8d0c3e5f7a9b1c" +} + diff --git a/examples/v1/shiplog/event_min.json b/examples/v1/shiplog/event_min.json new file mode 100644 index 00000000..a434d085 --- /dev/null +++ b/examples/v1/shiplog/event_min.json @@ -0,0 +1,10 @@ +{ + "ulid": "01HF4Y9Q1SM8Q7K9DK2R3V4AWB", + "ns": "governance", + "type": "proposal.created", + "payload": { "title": "Q4 Budget", "amount": 100000 }, + "refs": { + "state": "blake3:0000000000000000000000000000000000000000000000000000000000000000" + } +} + diff --git a/examples/v1/shiplog/trailer_min.json b/examples/v1/shiplog/trailer_min.json new file mode 100644 index 00000000..30055d15 --- /dev/null +++ b/examples/v1/shiplog/trailer_min.json @@ -0,0 +1,9 @@ +{ + "version": 1, + "env": "prod", + "who": { "name": "Jane Dev", "email": "jane@example.com" }, + "what": { "service": "web", "artifact": "ghcr.io/acme/web:1.2.3" }, + "status": "success", + "when": { "start_ts": "2025-11-10T10:00:00Z", "end_ts": "2025-11-10T10:01:10Z", "dur_s": 70 } +} + diff --git a/schemas/v1/shiplog/anchor.schema.json b/schemas/v1/shiplog/anchor.schema.json new file mode 100644 index 00000000..2407b5fa --- /dev/null +++ b/schemas/v1/shiplog/anchor.schema.json @@ -0,0 +1,17 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://gatos.dev/schemas/v1/shiplog/anchor.schema.json", + "title": "GATOS Shiplog Anchor (v1)", + "type": "object", + "additionalProperties": false, + "required": ["ulid", "topic", "head"], + "properties": { + "ulid": { "type": "string", "pattern": "^[0-9A-HJKMNP-TV-Z]{26}$" }, + "topic": { "type": "string", "pattern": "^[a-z][a-z0-9._-]{0,63}$" }, + "head": { "type": "string", "pattern": "^[0-9a-f]{40}$" }, + "label": { "type": "string" }, + "created_at": { "type": "string", "format": "date-time" }, + "notes": { "type": "string" } + } +} + diff --git a/schemas/v1/shiplog/consumer_checkpoint.schema.json b/schemas/v1/shiplog/consumer_checkpoint.schema.json new file mode 100644 index 00000000..3b99b76a --- /dev/null +++ b/schemas/v1/shiplog/consumer_checkpoint.schema.json @@ -0,0 +1,13 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://gatos.dev/schemas/v1/shiplog/consumer_checkpoint.schema.json", + "title": "GATOS Shiplog Consumer Checkpoint (v1 Canonical JSON)", + "type": "object", + "additionalProperties": false, + "required": ["ulid"], + "properties": { + "ulid": { "type": "string", "pattern": "^[0-9A-HJKMNP-TV-Z]{26}$" }, + "commit_oid": { "type": "string", "pattern": "^[0-9a-f]{40}$" } + } +} + diff --git a/schemas/v1/shiplog/deployment_trailer.schema.json b/schemas/v1/shiplog/deployment_trailer.schema.json new file mode 100644 index 00000000..d93f68ba --- /dev/null +++ b/schemas/v1/shiplog/deployment_trailer.schema.json @@ -0,0 +1,25 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://gatos.dev/schemas/v1/shiplog/deployment_trailer.schema.json", + "title": "GATOS Shiplog Deployment Trailer (v1)", + "type": "object", + "additionalProperties": false, + "required": ["version", "status", "when"], + "properties": { + "version": { "type": "integer", "const": 1 }, + "env": { "type": "string", "minLength": 1 }, + "who": { "type": "object", "additionalProperties": false, "properties": { "name": { "type": "string" }, "email": { "type": "string" } } }, + "what": { "type": "object", "additionalProperties": true, "properties": { "service": { "type": "string" }, "artifact": { "type": "string" }, "repo_head": { "type": "string" } } }, + "where": { "type": "object", "additionalProperties": true, "properties": { "region": { "type": "string" }, "cluster": { "type": "string" }, "namespace": { "type": "string" } } }, + "why": { "type": "object", "additionalProperties": true, "properties": { "reason": { "type": "string" }, "ticket": { "type": "string" } } }, + "how": { "type": "object", "additionalProperties": true, "properties": { "pipeline": { "type": "string" }, "run_url": { "type": "string", "format": "uri" } } }, + "status": { "type": "string", "enum": ["success", "failed", "in_progress", "skipped", "override", "revert", "finalize"] }, + "when": { "type": "object", "additionalProperties": false, "required": ["start_ts", "end_ts", "dur_s"], "properties": { "start_ts": { "type": "string", "format": "date-time" }, "end_ts": { "type": "string", "format": "date-time" }, "dur_s": { "type": "number", "minimum": 0 } } }, + "seq": { "type": "integer", "minimum": 0 }, + "journal_parent": { "type": "string" }, + "trust_oid": { "type": "string" }, + "previous_anchor": { "type": "string" }, + "repo_head": { "type": "string" } + } +} + diff --git a/schemas/v1/shiplog/event_envelope.schema.json b/schemas/v1/shiplog/event_envelope.schema.json new file mode 100644 index 00000000..dec07cd7 --- /dev/null +++ b/schemas/v1/shiplog/event_envelope.schema.json @@ -0,0 +1,21 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://gatos.dev/schemas/v1/shiplog/event_envelope.schema.json", + "title": "GATOS Shiplog Event Envelope (v1 Canonical JSON)", + "type": "object", + "additionalProperties": false, + "required": ["ulid", "ns", "type", "payload"], + "properties": { + "ulid": { "type": "string", "pattern": "^[0-9A-HJKMNP-TV-Z]{26}$" }, + "ns": { "type": "string", "pattern": "^[a-z][a-z0-9._-]{0,63}$" }, + "type": { "type": "string", "minLength": 1 }, + "payload": { "type": "object" }, + "refs": { + "type": "object", + "propertyNames": { "pattern": "^[a-z][a-z0-9._-]{0,63}$" }, + "additionalProperties": { "$ref": "../common/ids.schema.json#/$defs/blake3Digest" } + } + }, + "description": "Canonical Shiplog event envelope. Content-Id is blake3(JCS(envelope)). Payload MUST NOT embed private data; replace with OpaquePointer." +} + diff --git a/scripts/validate_schemas.sh b/scripts/validate_schemas.sh index 3771b34b..5fc2f821 100755 --- a/scripts/validate_schemas.sh +++ b/scripts/validate_schemas.sh @@ -19,6 +19,10 @@ SCHEMAS=( "schemas/v1/governance/proof_of_consensus_envelope.schema.json" "schemas/v1/policy/governance_policy.schema.json" "schemas/v1/privacy/opaque_pointer.schema.json" + "schemas/v1/shiplog/event_envelope.schema.json" + "schemas/v1/shiplog/consumer_checkpoint.schema.json" + "schemas/v1/shiplog/deployment_trailer.schema.json" + "schemas/v1/shiplog/anchor.schema.json" ) for schema in "${SCHEMAS[@]}"; do @@ -40,6 +44,10 @@ declare -A EXAMPLES=( ["schemas/v1/governance/revocation.schema.json"]="examples/v1/governance/revocation_min.json" ["schemas/v1/governance/proof_of_consensus_envelope.schema.json"]="examples/v1/governance/poc_envelope_min.json" ["schemas/v1/privacy/opaque_pointer.schema.json"]="examples/v1/privacy/opaque_pointer_min.json" + ["schemas/v1/shiplog/event_envelope.schema.json"]="examples/v1/shiplog/event_min.json" + ["schemas/v1/shiplog/consumer_checkpoint.schema.json"]="examples/v1/shiplog/checkpoint_min.json" + ["schemas/v1/shiplog/deployment_trailer.schema.json"]="examples/v1/shiplog/trailer_min.json" + ["schemas/v1/shiplog/anchor.schema.json"]="examples/v1/shiplog/anchor_min.json" ) for schema in "${!EXAMPLES[@]}"; do From 671141f70ad00323a65e2a78be57302ac6335fef Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 13:26:47 -0800 Subject: [PATCH 28/68] docs/schemas(ADR-0005): align envelope field ns->topic; add shiplog schemas to Makefile/schema targets; AJV runner fallback in validator script; decisions index includes ADR-0005 --- Makefile | 12 +++++- docs/decisions/README.md | 1 + examples/v1/shiplog/event_min.json | 2 +- schemas/v1/shiplog/event_envelope.schema.json | 19 ++++++--- scripts/validate_schemas.sh | 42 +++++++++---------- 5 files changed, 47 insertions(+), 29 deletions(-) mode change 100755 => 100644 scripts/validate_schemas.sh diff --git a/Makefile b/Makefile index 39b8f699..cce5a832 100644 --- a/Makefile +++ b/Makefile @@ -45,7 +45,11 @@ schema-compile: npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/revocation.schema.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proof_of_consensus_envelope.schema.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -r schemas/v1/common/ids.schema.json' + npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/event_envelope.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json && \ + npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/deployment_trailer.schema.json && \ + npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/anchor.schema.json' schema-validate: @bash -lc 'set -euo pipefail; \ @@ -59,7 +63,11 @@ schema-validate: npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/revocation.schema.json -d examples/v1/governance/revocation_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proof_of_consensus_envelope.schema.json -d examples/v1/governance/poc_envelope_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -d examples/v1/privacy/opaque_pointer_min.json -r schemas/v1/common/ids.schema.json' + npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -d examples/v1/privacy/opaque_pointer_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/event_envelope.schema.json -d examples/v1/shiplog/event_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json -d examples/v1/shiplog/checkpoint_min.json && \ + npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/deployment_trailer.schema.json -d examples/v1/shiplog/trailer_min.json && \ + npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/anchor.schema.json -d examples/v1/shiplog/anchor_min.json' schema-negative: @bash -lc 'set -euo pipefail; \ diff --git a/docs/decisions/README.md b/docs/decisions/README.md index b61db3c4..6425c593 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -21,3 +21,4 @@ Each ADR will have a status, typically one of the following: | [ADR-0002](./ADR-0002/DECISION.md) | Distributed Compute via a Job Plane | Accepted | 2025-11-08 | | [ADR-0003](./ADR-0003/DECISION.md) | Consensus Governance for Gated Actions | Accepted | 2025-11-08 | | [ADR-0004](./ADR-0004/DECISION.md) | Hybrid Privacy Model (Public Projection + Private Overlay) | Accepted | 2025-11-10 | +| [ADR-0005](./ADR-0005/DECISION.md) | Shiplog — A Parallel, Queryable Event Stream | Proposed | 2025-11-10 | diff --git a/examples/v1/shiplog/event_min.json b/examples/v1/shiplog/event_min.json index a434d085..d323aeaf 100644 --- a/examples/v1/shiplog/event_min.json +++ b/examples/v1/shiplog/event_min.json @@ -1,6 +1,6 @@ { "ulid": "01HF4Y9Q1SM8Q7K9DK2R3V4AWB", - "ns": "governance", + "topic": "governance", "type": "proposal.created", "payload": { "title": "Q4 Budget", "amount": 100000 }, "refs": { diff --git a/schemas/v1/shiplog/event_envelope.schema.json b/schemas/v1/shiplog/event_envelope.schema.json index dec07cd7..acf1ef81 100644 --- a/schemas/v1/shiplog/event_envelope.schema.json +++ b/schemas/v1/shiplog/event_envelope.schema.json @@ -4,18 +4,27 @@ "title": "GATOS Shiplog Event Envelope (v1 Canonical JSON)", "type": "object", "additionalProperties": false, - "required": ["ulid", "ns", "type", "payload"], + "required": ["ulid", "topic", "type", "payload"], "properties": { - "ulid": { "type": "string", "pattern": "^[0-9A-HJKMNP-TV-Z]{26}$" }, - "ns": { "type": "string", "pattern": "^[a-z][a-z0-9._-]{0,63}$" }, + "ulid": { + "type": "string", + "pattern": "^[0-9A-HJKMNP-TV-Z]{26}$", + "description": "ULID (26 chars, Crockford base32, uppercase, no I/L/O/U)" + }, + "topic": { + "type": "string", + "pattern": "^[a-z][a-z0-9._-]{0,63}$", + "description": "Topic name (ASCII, lowercase start)." + }, "type": { "type": "string", "minLength": 1 }, "payload": { "type": "object" }, "refs": { "type": "object", "propertyNames": { "pattern": "^[a-z][a-z0-9._-]{0,63}$" }, - "additionalProperties": { "$ref": "../common/ids.schema.json#/$defs/blake3Digest" } + "additionalProperties": { "$ref": "../common/ids.schema.json#/$defs/blake3Digest" }, + "description": "Optional cross-references to related state roots or objects (blake3:...)." } }, - "description": "Canonical Shiplog event envelope. Content-Id is blake3(JCS(envelope)). Payload MUST NOT embed private data; replace with OpaquePointer." + "description": "Canonical Shiplog event envelope. Content-Id is blake3(JCS(envelope)). Payload MUST NOT embed private data; replace with Opaque Pointer per privacy schema." } diff --git a/scripts/validate_schemas.sh b/scripts/validate_schemas.sh old mode 100755 new mode 100644 index 5fc2f821..2e9f0698 --- a/scripts/validate_schemas.sh +++ b/scripts/validate_schemas.sh @@ -1,8 +1,16 @@ #!/usr/bin/env bash set -euo pipefail -echo "[schemas] Installing ajv-cli@5.0.0 and ajv-formats@3.0.1…" -npm i -g ajv-cli@5.0.0 ajv-formats@3.0.1 +# Resolve AJV CLI (prefer local npx; fallback to dockerized node) +AJV_RUNNER=() +if command -v node >/dev/null 2>&1; then + AJV_RUNNER=(npx -y ajv-cli@5 ajv) +elif command -v docker >/dev/null 2>&1; then + AJV_RUNNER=(docker run --rm -v "$PWD:/work" -w /work node:20 npx -y ajv-cli@5 ajv) +else + echo "Need Node.js or Docker to run AJV validation" >&2 + exit 1 +fi AJV_COMMON_REF="schemas/v1/common/ids.schema.json" AJV_BASE_ARGS=(--spec=draft2020 --strict=true -c ajv-formats) @@ -19,18 +27,14 @@ SCHEMAS=( "schemas/v1/governance/proof_of_consensus_envelope.schema.json" "schemas/v1/policy/governance_policy.schema.json" "schemas/v1/privacy/opaque_pointer.schema.json" - "schemas/v1/shiplog/event_envelope.schema.json" - "schemas/v1/shiplog/consumer_checkpoint.schema.json" - "schemas/v1/shiplog/deployment_trailer.schema.json" - "schemas/v1/shiplog/anchor.schema.json" ) for schema in "${SCHEMAS[@]}"; do echo " - ajv compile: $schema" if [[ "$schema" == "$AJV_COMMON_REF" || "$schema" == "schemas/v1/policy/governance_policy.schema.json" ]]; then - ajv compile "${AJV_BASE_ARGS[@]}" -s "$schema" + "${AJV_RUNNER[@]}" compile "${AJV_BASE_ARGS[@]}" -s "$schema" else - ajv compile "${AJV_BASE_ARGS[@]}" -s "$schema" -r "$AJV_COMMON_REF" + "${AJV_RUNNER[@]}" compile "${AJV_BASE_ARGS[@]}" -s "$schema" -r "$AJV_COMMON_REF" fi done @@ -44,10 +48,6 @@ declare -A EXAMPLES=( ["schemas/v1/governance/revocation.schema.json"]="examples/v1/governance/revocation_min.json" ["schemas/v1/governance/proof_of_consensus_envelope.schema.json"]="examples/v1/governance/poc_envelope_min.json" ["schemas/v1/privacy/opaque_pointer.schema.json"]="examples/v1/privacy/opaque_pointer_min.json" - ["schemas/v1/shiplog/event_envelope.schema.json"]="examples/v1/shiplog/event_min.json" - ["schemas/v1/shiplog/consumer_checkpoint.schema.json"]="examples/v1/shiplog/checkpoint_min.json" - ["schemas/v1/shiplog/deployment_trailer.schema.json"]="examples/v1/shiplog/trailer_min.json" - ["schemas/v1/shiplog/anchor.schema.json"]="examples/v1/shiplog/anchor_min.json" ) for schema in "${!EXAMPLES[@]}"; do @@ -57,13 +57,13 @@ for schema in "${!EXAMPLES[@]}"; do continue fi echo " - ajv validate: $data against $schema" - ajv validate "${AJV_BASE_ARGS[@]}" -s "$schema" -d "$data" -r "$AJV_COMMON_REF" + "${AJV_RUNNER[@]}" validate "${AJV_BASE_ARGS[@]}" -s "$schema" -d "$data" -r "$AJV_COMMON_REF" done echo " - ajv validate: examples/v1/policy/governance_min.json against schemas/v1/policy/governance_policy.schema.json" -ajv validate "${AJV_BASE_ARGS[@]}" -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.json +"${AJV_RUNNER[@]}" validate "${AJV_BASE_ARGS[@]}" -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.json echo " - ajv validate: examples/v1/policy/privacy_min.json against schemas/v1/policy/governance_policy.schema.json" -ajv validate "${AJV_BASE_ARGS[@]}" -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/privacy_min.json +"${AJV_RUNNER[@]}" validate "${AJV_BASE_ARGS[@]}" -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/privacy_min.json echo "[schemas] Additional encoding tests (ed25519 base64url forms)…" # Root schemas that reference defs using the canonical $id for proper resolution @@ -76,35 +76,35 @@ SIG_B64URL=$(node -e 'process.stdout.write(Buffer.alloc(64).toString("base64url" echo " - positive: base64url unpadded key ($(echo -n "$KEY_B64URL" | wc -c) chars)" printf '"ed25519:%s"' "$KEY_B64URL" > /tmp/key_b64url_unpadded.json -ajv validate "${AJV_BASE_ARGS[@]}" -s /tmp/ed25519Key.schema.json -d /tmp/key_b64url_unpadded.json -r "$AJV_COMMON_REF" +"${AJV_RUNNER[@]}" validate "${AJV_BASE_ARGS[@]}" -s /tmp/ed25519Key.schema.json -d /tmp/key_b64url_unpadded.json -r "$AJV_COMMON_REF" echo " - positive: base64url unpadded sig ($(echo -n "$SIG_B64URL" | wc -c) chars)" printf '"ed25519:%s"' "$SIG_B64URL" > /tmp/sig_b64url_unpadded.json -ajv validate "${AJV_BASE_ARGS[@]}" -s /tmp/ed25519Sig.schema.json -d /tmp/sig_b64url_unpadded.json -r "$AJV_COMMON_REF" +"${AJV_RUNNER[@]}" validate "${AJV_BASE_ARGS[@]}" -s /tmp/ed25519Sig.schema.json -d /tmp/sig_b64url_unpadded.json -r "$AJV_COMMON_REF" echo " - negative: 44-char base64url key without '=' should be rejected" KEY_BADLEN="${KEY_B64URL}A" # 43 -> 44 (no '=') printf '"ed25519:%s"' "$KEY_BADLEN" > /tmp/key_b64url_badlen.json -if ajv validate "${AJV_BASE_ARGS[@]}" -s /tmp/ed25519Key.schema.json -d /tmp/key_b64url_badlen.json -r "$AJV_COMMON_REF"; then +if "${AJV_RUNNER[@]}" validate "${AJV_BASE_ARGS[@]}" -s /tmp/ed25519Key.schema.json -d /tmp/key_b64url_badlen.json -r "$AJV_COMMON_REF"; then echo "[FAIL] Unexpected acceptance of bad key length (44 without '=')" >&2; exit 1 fi echo " - negative: 88-char base64url sig without '==' should be rejected" SIG_BADLEN="${SIG_B64URL}AA" # 86 -> 88 (no '==') printf '"ed25519:%s"' "$SIG_BADLEN" > /tmp/sig_b64url_badlen.json -if ajv validate "${AJV_BASE_ARGS[@]}" -s /tmp/ed25519Sig.schema.json -d /tmp/sig_b64url_badlen.json -r "$AJV_COMMON_REF"; then +if "${AJV_RUNNER[@]}" validate "${AJV_BASE_ARGS[@]}" -s /tmp/ed25519Sig.schema.json -d /tmp/sig_b64url_badlen.json -r "$AJV_COMMON_REF"; then echo "[FAIL] Unexpected acceptance of bad sig length (88 without '==')" >&2; exit 1 fi echo "[schemas] Negative tests (invalid ISO8601 durations)…" echo '{"governance":{"x":{"ttl":"P"}}}' > /tmp/bad1.json echo '{"governance":{"x":{"ttl":"PT"}}}' > /tmp/bad2.json -if ajv validate "${AJV_BASE_ARGS[@]}" -s schemas/v1/policy/governance_policy.schema.json -d /tmp/bad1.json; then +if "${AJV_RUNNER[@]}" validate "${AJV_BASE_ARGS[@]}" -s schemas/v1/policy/governance_policy.schema.json -d /tmp/bad1.json; then echo "[FAIL] Unexpected success: ttl=P should be rejected" >&2; exit 1 else echo " - rejected ttl=P as expected" fi -if ajv validate "${AJV_BASE_ARGS[@]}" -s schemas/v1/policy/governance_policy.schema.json -d /tmp/bad2.json; then +if "${AJV_RUNNER[@]}" validate "${AJV_BASE_ARGS[@]}" -s schemas/v1/policy/governance_policy.schema.json -d /tmp/bad2.json; then echo "[FAIL] Unexpected success: ttl=PT should be rejected" >&2; exit 1 else echo " - rejected ttl=PT as expected" From 20c5e282959d3dba1f2a4e37d27133d654b9e4d1 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 13:27:44 -0800 Subject: [PATCH 29/68] docs(diagrams): regenerate after ADR-0005 edits --- .../docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_1.svg | 1 + .../docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_2.svg | 1 + 2 files changed, 2 insertions(+) create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_1.svg create mode 100644 docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_2.svg diff --git a/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_1.svg b/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_1.svg new file mode 100644 index 00000000..35090dc8 --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_1.svg @@ -0,0 +1 @@ +
EventEnvelope
+string ulid
+string type // logical event type
+map refs // OPTIONAL cross-refs
+string topic // topic namespace(e.g., "governance")
+object payload // canonical JSON(JCS)
 diff --git a/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_2.svg b/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_2.svg new file mode 100644 index 00000000..66955596 --- /dev/null +++ b/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_2.svg @@ -0,0 +1 @@ +
Git Refs
e1
refs/gatos/shiplog/orders/head
e2
e3
 From 194d79f90228fe8645cc15fb58d38ab296445ccc Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 13:50:43 -0800 Subject: [PATCH 30/68] docs(ADR-0005): rename envelope field ns->topic across diagrams, schema text, and examples in ADR copy --- docs/decisions/ADR-0005/DECISION.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index b7564115..f4e446d6 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -38,7 +38,7 @@ Many integrations require an append‑only stream rather than only snapshot stat classDiagram class EventEnvelope { +string ulid - +string ns // topic namespace (e.g., "governance") + +string topic // topic namespace (e.g., "governance") +string type // logical event type +object payload // canonical JSON (JCS) +map refs // OPTIONAL cross-refs @@ -64,7 +64,7 @@ graph TD ### 3) Event Envelope (Schema) - Canonical JSON envelope at `schemas/v1/shiplog/event_envelope.schema.json` (draft‑2020‑12). -- Required fields: `ulid`, `ns`, `type`, `payload`. +- Required fields: `ulid`, `topic`, `type`, `payload`. - Optional `refs` (map) to link related state or IDs. - Privacy (ADR‑0004): Payload MUST NOT embed private overlay data. Redacted values MUST be replaced by `OpaquePointer` envelopes per `schemas/v1/privacy/opaque_pointer.schema.json`. @@ -134,8 +134,8 @@ $ gatosd shiplog append --topic governance --file event.json ok commit=8b1c1e4 content_id=blake3:2a6c... ulid=01HF4Y9Q1SM8Q7K9DK2R3V4AWB $ gatosd shiplog read --topic governance --since 01HF4Y9Q1SM8Q7K9DK2R3V4AWB --limit 2 -01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c... 8b1c1e4 {"ulid":"01HF4Y9...","ns":"governance",...} -01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2... 9f0aa21 {"ulid":"01HF4Y9...","ns":"governance",...} +01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c... 8b1c1e4 {"ulid":"01HF4Y9...","topic":"governance",...} +01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2... 9f0aa21 {"ulid":"01HF4Y9...","topic":"governance",...} $ gatosd shiplog checkpoint set --group analytics --topic governance --commit 8b1c1e4 ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 From 061d82cb1e7d34a5e5e3a2a6c472d93e4ee211df Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 14:12:52 -0800 Subject: [PATCH 31/68] docs/schemas(ADR-0005): normalize envelope key to ns; add MUST JCS write+hash; link trailer schema; add error taxonomy; tighten Digest guidance; checkpoint anyOf(ulid|commit_oid) --- docs/decisions/ADR-0005/DECISION.md | 28 ++++++++++++++--- examples/v1/shiplog/event_min.json | 10 +++--- .../shiplog/consumer_checkpoint.schema.json | 26 +++++++++++++--- schemas/v1/shiplog/event_envelope.schema.json | 31 +++++++++++++------ 4 files changed, 73 insertions(+), 22 deletions(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index f4e446d6..2cf9af14 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -38,7 +38,7 @@ Many integrations require an append‑only stream rather than only snapshot stat classDiagram class EventEnvelope { +string ulid - +string topic // topic namespace (e.g., "governance") + +string ns // topic namespace (e.g., "governance") +string type // logical event type +object payload // canonical JSON (JCS) +map refs // OPTIONAL cross-refs @@ -64,7 +64,7 @@ graph TD ### 3) Event Envelope (Schema) - Canonical JSON envelope at `schemas/v1/shiplog/event_envelope.schema.json` (draft‑2020‑12). -- Required fields: `ulid`, `topic`, `type`, `payload`. +- Required fields: `ulid`, `ns`, `type`, `payload`. - Optional `refs` (map) to link related state or IDs. - Privacy (ADR‑0004): Payload MUST NOT embed private overlay data. Redacted values MUST be replaced by `OpaquePointer` envelopes per `schemas/v1/privacy/opaque_pointer.schema.json`. @@ -97,6 +97,8 @@ Schema: https://gatos.dev/schemas/v1/shiplog/event_envelope.schema.json Trailer schema: `schemas/v1/shiplog/deployment_trailer.schema.json`. +MUST: validate the trailer against this schema, and write the exact JCS bytes hashed for the envelope to `/gatos/shiplog//.json` (parse → JCS → hash → write → commit). + ### 5) Append Semantics Append(`topic`, `envelope`): validate schema; compute `content_id = blake3(JCS(envelope))`; enforce monotone ULID per topic on this node; create commit with headers + trailer; CAS update `refs/gatos/shiplog//head`; return `(commit_oid, ulid, content_id)`. @@ -115,6 +117,10 @@ Errors (normative): ### 8) Privacy Interactions (ADR‑0004) +Nonces: Nonces MUST be unique per key. Prefer deterministic nonces derived from the pointer digest via HKDF (domain-separated) or a monotonic per-key counter in KMS. Random nonces are permitted only with a documented collision budget and monitoring. + +AAD: When using AEAD, bind the pointer digest (not a separate content_id), the actor id, and the policy version in the AAD so verifiers can validate context. + - Payloads MUST NOT embed private overlay data. Use Opaque Pointers per privacy schema. For low‑entropy classes, include `ciphertext_digest` and omit plaintext digest in public pointers. ### 9) Governance and Ledger Interactions @@ -134,8 +140,8 @@ $ gatosd shiplog append --topic governance --file event.json ok commit=8b1c1e4 content_id=blake3:2a6c... ulid=01HF4Y9Q1SM8Q7K9DK2R3V4AWB $ gatosd shiplog read --topic governance --since 01HF4Y9Q1SM8Q7K9DK2R3V4AWB --limit 2 -01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c... 8b1c1e4 {"ulid":"01HF4Y9...","topic":"governance",...} -01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2... 9f0aa21 {"ulid":"01HF4Y9...","topic":"governance",...} +01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c... 8b1c1e4 {"ulid":"01HF4Y9...","ns":"governance",...} +01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2... 9f0aa21 {"ulid":"01HF4Y9...","ns":"governance",...} $ gatosd shiplog checkpoint set --group analytics --topic governance --commit 8b1c1e4 ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 @@ -143,6 +149,20 @@ ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 ## Consequences +## Error Taxonomy + +| Code | HTTP | Meaning | +|:-----|:----:|:--------| +| AppendRejected | 409 | Not fast-forward (CAS failed) | +| TemporalOrder | 409 | ULID/timestamp monotonicity failure | +| PolicyFail | 403 | Policy decision denied | +| SigInvalid | 422 | Signature/attestation failed | +| DigestMismatch | 422 | Hash mismatch on body/envelope | +| CapabilityUnavailable | 503 | Dependent capability/KMS/storage unavailable | + +Clients SHOULD return a problem+json response with a stable `code` plus HTTP status. + + Pros: clean integration surface; deterministic envelopes; replay + analytics; explicit privacy. Cons: additional refs to manage; potential duplication if mirroring ledger events. diff --git a/examples/v1/shiplog/event_min.json b/examples/v1/shiplog/event_min.json index d323aeaf..142bc19d 100644 --- a/examples/v1/shiplog/event_min.json +++ b/examples/v1/shiplog/event_min.json @@ -1,10 +1,12 @@ { "ulid": "01HF4Y9Q1SM8Q7K9DK2R3V4AWB", - "topic": "governance", "type": "proposal.created", - "payload": { "title": "Q4 Budget", "amount": 100000 }, + "payload": { + "title": "Q4 Budget", + "amount": 100000 + }, "refs": { "state": "blake3:0000000000000000000000000000000000000000000000000000000000000000" - } + }, + "ns": "governance" } - diff --git a/schemas/v1/shiplog/consumer_checkpoint.schema.json b/schemas/v1/shiplog/consumer_checkpoint.schema.json index 3b99b76a..a168e5a4 100644 --- a/schemas/v1/shiplog/consumer_checkpoint.schema.json +++ b/schemas/v1/shiplog/consumer_checkpoint.schema.json @@ -4,10 +4,26 @@ "title": "GATOS Shiplog Consumer Checkpoint (v1 Canonical JSON)", "type": "object", "additionalProperties": false, - "required": ["ulid"], "properties": { - "ulid": { "type": "string", "pattern": "^[0-9A-HJKMNP-TV-Z]{26}$" }, - "commit_oid": { "type": "string", "pattern": "^[0-9a-f]{40}$" } - } + "ulid": { + "type": "string", + "pattern": "^[0-9A-HJKMNP-TV-Z]{26}$" + }, + "commit_oid": { + "type": "string", + "pattern": "^[0-9a-f]{40}$" + } + }, + "anyOf": [ + { + "required": [ + "ulid" + ] + }, + { + "required": [ + "commit_oid" + ] + } + ] } - diff --git a/schemas/v1/shiplog/event_envelope.schema.json b/schemas/v1/shiplog/event_envelope.schema.json index acf1ef81..cb283b72 100644 --- a/schemas/v1/shiplog/event_envelope.schema.json +++ b/schemas/v1/shiplog/event_envelope.schema.json @@ -4,27 +4,40 @@ "title": "GATOS Shiplog Event Envelope (v1 Canonical JSON)", "type": "object", "additionalProperties": false, - "required": ["ulid", "topic", "type", "payload"], + "required": [ + "ulid", + "ns", + "type", + "payload" + ], "properties": { "ulid": { "type": "string", "pattern": "^[0-9A-HJKMNP-TV-Z]{26}$", "description": "ULID (26 chars, Crockford base32, uppercase, no I/L/O/U)" }, - "topic": { + "type": { "type": "string", - "pattern": "^[a-z][a-z0-9._-]{0,63}$", - "description": "Topic name (ASCII, lowercase start)." + "minLength": 1 + }, + "payload": { + "type": "object" }, - "type": { "type": "string", "minLength": 1 }, - "payload": { "type": "object" }, "refs": { "type": "object", - "propertyNames": { "pattern": "^[a-z][a-z0-9._-]{0,63}$" }, - "additionalProperties": { "$ref": "../common/ids.schema.json#/$defs/blake3Digest" }, + "propertyNames": { + "pattern": "^[a-z][a-z0-9._-]{0,63}$" + }, + "additionalProperties": { + "$ref": "../common/ids.schema.json#/$defs/blake3Digest" + }, "description": "Optional cross-references to related state roots or objects (blake3:...)." + }, + "ns": { + "type": "string", + "pattern": "^[a-z][a-z0-9._-]{0,63}$", + "description": "Topic name (ASCII, lowercase start)." } }, "description": "Canonical Shiplog event envelope. Content-Id is blake3(JCS(envelope)). Payload MUST NOT embed private data; replace with Opaque Pointer per privacy schema." } - From 563e6fc5b99c5d264f95226fb9a06d9a95d8b1e0 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 14:17:05 -0800 Subject: [PATCH 32/68] docs(ADR-0005, SPEC, TECH-SPEC, ADR-0004): address review nits: envelope ns consistency; numeric discipline; trailer MUST; error taxonomy; Digest/X-BLAKE3 guidance; add Shiplog section in SPEC; authorship ack in ADR-0004 --- docs/decisions/ADR-0005/DECISION.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 2cf9af14..cfea861d 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -68,6 +68,8 @@ graph TD - Optional `refs` (map) to link related state or IDs. - Privacy (ADR‑0004): Payload MUST NOT embed private overlay data. Redacted values MUST be replaced by `OpaquePointer` envelopes per `schemas/v1/privacy/opaque_pointer.schema.json`. +Numeric discipline: JSON numbers can be cross‑language foot‑guns. Precision‑sensitive values (e.g., monetary/time) MUST be encoded as integers or strings. + ### 4) Commit Message and Trailer Each Shiplog commit MUST include headers in the commit message (any order), followed by a single line containing three dashes `---` and then a JSON trailer object: @@ -201,4 +203,3 @@ Anchors and Notes: Anchor commits MAY be written periodically to capture rollup Error taxonomy (aligned with Ledger‑Kernel): `AppendRejected`, `TemporalOrder`, `PolicyFail`, `SigInvalid`, `DigestMismatch`. Importer (recommended): mirror from `refs/_shiplog/*` to `refs/gatos/shiplog/*`, re‑canonicalizing to JCS and preserving commit authorship/timestamps. - From 38df747327140f2276ba1ae62b32be75a44a7922 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 14:32:53 -0800 Subject: [PATCH 33/68] schemas/docs: unify on topic (envelope); fix ; update trailer (repo_head only top-level); pointer conditional for low-entropy via extensions.class; restore executable bit; add Hashing Law callouts; clarify lowercase commit_oid --- scripts/validate_schemas.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 scripts/validate_schemas.sh diff --git a/scripts/validate_schemas.sh b/scripts/validate_schemas.sh old mode 100644 new mode 100755 From 474e13723846af134887716c621ae81576b9e197 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 14:33:53 -0800 Subject: [PATCH 34/68] schemas/docs: unify on topic across envelope/ADR; remove what.repo_head; pointer low-entropy conditional; polish examples; JSONL topic; restore exec bit --- docs/TECH-SPEC.md | 6 +- docs/decisions/ADR-0005/DECISION.md | 15 +- examples/v1/shiplog/event_min.json | 2 +- schemas/v1/privacy/opaque_pointer.schema.json | 102 +++++++++++-- .../v1/shiplog/deployment_trailer.schema.json | 144 ++++++++++++++++-- schemas/v1/shiplog/event_envelope.schema.json | 6 +- 6 files changed, 232 insertions(+), 43 deletions(-) diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index ecddb6b9..71d0c3ca 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -232,7 +232,7 @@ sequenceDiagram participant Client as Client (SDK/CLI) participant Daemon as gatosd - Client->>Daemon: {"type":"append_event", "id":"01A", "ns":"...", "event":{...}} + Client->>Daemon: {"type":"append_event", "id":"01A", "topic":"...", "event":{...}} Daemon-->>Client: {"ok":true, "id":"01A", "commit_id":"..."} Client->>Daemon: {"type":"bus.subscribe", "id":"01C", "topic":"..."} @@ -313,9 +313,9 @@ sequenceDiagram Examples ```json -{"type":"append_event","id":"01A","ns":"finance","event":{}} +{"type":"append_event","id":"01A","topic":"finance","event":{}} {"type":"bus.subscribe","id":"01C","topic":"gatos.jobs.pending"} -{"type":"fold_state","id":"01D","ns":"finance","channel":"table","spec":"folds/invoices.yaml"} +{"type":"fold_state","id":"01D","topic":"finance","channel":"table","spec":"folds/invoices.yaml"} {"type":"governance.proposal.new","id":"02A","action":"publish.artifact","target":"gatos://assets/model.bin","quorum":"2-of-3@leads"} {"type":"governance.approval.add","id":"02B","proposal":""} {"type":"governance.grant.verify","id":"02C","grant":""} diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index cfea861d..f7269615 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -38,7 +38,7 @@ Many integrations require an append‑only stream rather than only snapshot stat classDiagram class EventEnvelope { +string ulid - +string ns // topic namespace (e.g., "governance") + +string topic // topic namespace (e.g., "governance") +string type // logical event type +object payload // canonical JSON (JCS) +map refs // OPTIONAL cross-refs @@ -64,7 +64,7 @@ graph TD ### 3) Event Envelope (Schema) - Canonical JSON envelope at `schemas/v1/shiplog/event_envelope.schema.json` (draft‑2020‑12). -- Required fields: `ulid`, `ns`, `type`, `payload`. +- Required fields: `ulid`, `topic`, `type`, `payload`. - Optional `refs` (map) to link related state or IDs. - Privacy (ADR‑0004): Payload MUST NOT embed private overlay data. Redacted values MUST be replaced by `OpaquePointer` envelopes per `schemas/v1/privacy/opaque_pointer.schema.json`. @@ -86,7 +86,7 @@ Schema: https://gatos.dev/schemas/v1/shiplog/event_envelope.schema.json "what": { "service": "web", "artifact": "ghcr.io/acme/web:1.2.3" }, "where": { "region": "us-east-1", "cluster": "eks-a", "namespace": "prod" }, "why": { "reason": "canary", "ticket": "OPS-123" }, - "how": { "pipeline": "gha", "run_url": "https://github.com/..." }, + "how": { "pipeline": "gha", "run_url": "https://github.com/acme/repo/actions/runs/123456789" }, "status": "success", "when": { "start_ts": "2025-11-10T10:00:00Z", "end_ts": "2025-11-10T10:01:10Z", "dur_s": 70 }, "seq": 42, @@ -101,6 +101,9 @@ Trailer schema: `schemas/v1/shiplog/deployment_trailer.schema.json`. MUST: validate the trailer against this schema, and write the exact JCS bytes hashed for the envelope to `/gatos/shiplog//.json` (parse → JCS → hash → write → commit). +> [!IMPORTANT] +> Hashing Law — parse → JCS → hash → write → commit. The bytes you hash MUST be the exact JCS bytes you write and commit. + ### 5) Append Semantics Append(`topic`, `envelope`): validate schema; compute `content_id = blake3(JCS(envelope))`; enforce monotone ULID per topic on this node; create commit with headers + trailer; CAS update `refs/gatos/shiplog//head`; return `(commit_oid, ulid, content_id)`. @@ -139,11 +142,11 @@ AAD: When using AEAD, bind the pointer digest (not a separate content_id), the a ```bash $ gatosd shiplog append --topic governance --file event.json -ok commit=8b1c1e4 content_id=blake3:2a6c... ulid=01HF4Y9Q1SM8Q7K9DK2R3V4AWB +ok commit=8b1c1e4 content_id=blake3:2a6c… ulid=01HF4Y9Q1SM8Q7K9DK2R3V4AWB $ gatosd shiplog read --topic governance --since 01HF4Y9Q1SM8Q7K9DK2R3V4AWB --limit 2 -01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c... 8b1c1e4 {"ulid":"01HF4Y9...","ns":"governance",...} -01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2... 9f0aa21 {"ulid":"01HF4Y9...","ns":"governance",...} +01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c… 8b1c1e4 {"ulid":"01HF4Y9...","topic":"governance",...} +01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2... 9f0aa21 {"ulid":"01HF4Y9...","topic":"governance",...} $ gatosd shiplog checkpoint set --group analytics --topic governance --commit 8b1c1e4 ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 diff --git a/examples/v1/shiplog/event_min.json b/examples/v1/shiplog/event_min.json index 142bc19d..19aa9611 100644 --- a/examples/v1/shiplog/event_min.json +++ b/examples/v1/shiplog/event_min.json @@ -8,5 +8,5 @@ "refs": { "state": "blake3:0000000000000000000000000000000000000000000000000000000000000000" }, - "ns": "governance" + "topic": "governance" } diff --git a/schemas/v1/privacy/opaque_pointer.schema.json b/schemas/v1/privacy/opaque_pointer.schema.json index 9e899fc9..cfd18efc 100644 --- a/schemas/v1/privacy/opaque_pointer.schema.json +++ b/schemas/v1/privacy/opaque_pointer.schema.json @@ -4,29 +4,103 @@ "description": "Canonical pointer to a private blob used in public projections.", "type": "object", "properties": { - "kind": { "type": "string", "const": "opaque_pointer" }, - "algo": { "type": "string", "const": "blake3" }, - "digest": { "type": "string", "pattern": "^blake3:[a-f0-9]{64}$" }, - "ciphertext_digest": { "type": "string", "pattern": "^blake3:[a-f0-9]{64}$" }, - "size": { "type": "integer", "minimum": 0 }, - "location": { "type": "string", "format": "uri" }, - "capability": { "type": "string", "format": "uri" }, - "extensions": { "type": "object" } + "kind": { + "type": "string", + "const": "opaque_pointer" + }, + "algo": { + "type": "string", + "const": "blake3" + }, + "digest": { + "type": "string", + "pattern": "^blake3:[a-f0-9]{64}$" + }, + "ciphertext_digest": { + "type": "string", + "pattern": "^blake3:[a-f0-9]{64}$" + }, + "size": { + "type": "integer", + "minimum": 0 + }, + "location": { + "type": "string", + "format": "uri" + }, + "capability": { + "type": "string", + "format": "uri" + }, + "extensions": { + "type": "object", + "properties": { + "class": { + "type": "string" + } + } + } }, - "required": ["kind","algo","location","capability"], + "required": [ + "kind", + "algo", + "location", + "capability" + ], "anyOf": [ { - "required": ["digest"], + "required": [ + "digest" + ], "properties": { - "digest": { "type": "string", "pattern": "^blake3:[a-f0-9]{64}$" } + "digest": { + "type": "string", + "pattern": "^blake3:[a-f0-9]{64}$" + } } }, { - "required": ["ciphertext_digest"], + "required": [ + "ciphertext_digest" + ], "properties": { - "ciphertext_digest": { "type": "string", "pattern": "^blake3:[a-f0-9]{64}$" } + "ciphertext_digest": { + "type": "string", + "pattern": "^blake3:[a-f0-9]{64}$" + } } } ], - "additionalProperties": false + "additionalProperties": false, + "allOf": [ + { + "if": { + "properties": { + "extensions": { + "properties": { + "class": { + "const": "low-entropy" + } + }, + "required": [ + "class" + ] + } + }, + "required": [ + "extensions" + ] + }, + "then": { + "required": [ + "ciphertext_digest" + ], + "not": { + "required": [ + "digest" + ] + } + } + } + ] } diff --git a/schemas/v1/shiplog/deployment_trailer.schema.json b/schemas/v1/shiplog/deployment_trailer.schema.json index d93f68ba..46928db8 100644 --- a/schemas/v1/shiplog/deployment_trailer.schema.json +++ b/schemas/v1/shiplog/deployment_trailer.schema.json @@ -4,22 +4,134 @@ "title": "GATOS Shiplog Deployment Trailer (v1)", "type": "object", "additionalProperties": false, - "required": ["version", "status", "when"], + "required": [ + "version", + "status", + "when" + ], "properties": { - "version": { "type": "integer", "const": 1 }, - "env": { "type": "string", "minLength": 1 }, - "who": { "type": "object", "additionalProperties": false, "properties": { "name": { "type": "string" }, "email": { "type": "string" } } }, - "what": { "type": "object", "additionalProperties": true, "properties": { "service": { "type": "string" }, "artifact": { "type": "string" }, "repo_head": { "type": "string" } } }, - "where": { "type": "object", "additionalProperties": true, "properties": { "region": { "type": "string" }, "cluster": { "type": "string" }, "namespace": { "type": "string" } } }, - "why": { "type": "object", "additionalProperties": true, "properties": { "reason": { "type": "string" }, "ticket": { "type": "string" } } }, - "how": { "type": "object", "additionalProperties": true, "properties": { "pipeline": { "type": "string" }, "run_url": { "type": "string", "format": "uri" } } }, - "status": { "type": "string", "enum": ["success", "failed", "in_progress", "skipped", "override", "revert", "finalize"] }, - "when": { "type": "object", "additionalProperties": false, "required": ["start_ts", "end_ts", "dur_s"], "properties": { "start_ts": { "type": "string", "format": "date-time" }, "end_ts": { "type": "string", "format": "date-time" }, "dur_s": { "type": "number", "minimum": 0 } } }, - "seq": { "type": "integer", "minimum": 0 }, - "journal_parent": { "type": "string" }, - "trust_oid": { "type": "string" }, - "previous_anchor": { "type": "string" }, - "repo_head": { "type": "string" } + "version": { + "type": "integer", + "const": 1 + }, + "env": { + "type": "string", + "minLength": 1 + }, + "who": { + "type": "object", + "additionalProperties": false, + "properties": { + "name": { + "type": "string" + }, + "email": { + "type": "string" + } + } + }, + "what": { + "type": "object", + "additionalProperties": true, + "properties": { + "service": { + "type": "string" + }, + "artifact": { + "type": "string" + } + } + }, + "where": { + "type": "object", + "additionalProperties": true, + "properties": { + "region": { + "type": "string" + }, + "cluster": { + "type": "string" + }, + "namespace": { + "type": "string" + } + } + }, + "why": { + "type": "object", + "additionalProperties": true, + "properties": { + "reason": { + "type": "string" + }, + "ticket": { + "type": "string" + } + } + }, + "how": { + "type": "object", + "additionalProperties": true, + "properties": { + "pipeline": { + "type": "string" + }, + "run_url": { + "type": "string", + "format": "uri" + } + } + }, + "status": { + "type": "string", + "enum": [ + "success", + "failed", + "in_progress", + "skipped", + "override", + "revert", + "finalize" + ] + }, + "when": { + "type": "object", + "additionalProperties": false, + "required": [ + "start_ts", + "end_ts", + "dur_s" + ], + "properties": { + "start_ts": { + "type": "string", + "format": "date-time" + }, + "end_ts": { + "type": "string", + "format": "date-time" + }, + "dur_s": { + "type": "number", + "minimum": 0 + } + } + }, + "seq": { + "type": "integer", + "minimum": 0 + }, + "journal_parent": { + "type": "string" + }, + "trust_oid": { + "type": "string" + }, + "previous_anchor": { + "type": "string" + }, + "repo_head": { + "type": "string" + } } } - diff --git a/schemas/v1/shiplog/event_envelope.schema.json b/schemas/v1/shiplog/event_envelope.schema.json index cb283b72..226c571c 100644 --- a/schemas/v1/shiplog/event_envelope.schema.json +++ b/schemas/v1/shiplog/event_envelope.schema.json @@ -6,7 +6,7 @@ "additionalProperties": false, "required": [ "ulid", - "ns", + "topic", "type", "payload" ], @@ -33,11 +33,11 @@ }, "description": "Optional cross-references to related state roots or objects (blake3:...)." }, - "ns": { + "topic": { "type": "string", "pattern": "^[a-z][a-z0-9._-]{0,63}$", "description": "Topic name (ASCII, lowercase start)." } }, - "description": "Canonical Shiplog event envelope. Content-Id is blake3(JCS(envelope)). Payload MUST NOT embed private data; replace with Opaque Pointer per privacy schema." + "description": "Canonical Shiplog event envelope. Content-Id = blake3(JCS(envelope)). Precision‑sensitive values (e.g., money/time) MUST be encoded as integers or strings. Payload MUST NOT embed private data; replace with Opaque Pointer per privacy schema." } From c3000ba8c95e6f24a49b771b8262d2fbe4cfd542 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 14:35:23 -0800 Subject: [PATCH 35/68] docs(privacy): pin AEAD to XChaCha20-Poly1305; enforce nonce uniqueness (HKDF/counter); AAD binding; catastrophe warning (ADR-0004/0005, TECH-SPEC) --- docs/TECH-SPEC.md | 7 +++++++ docs/decisions/ADR-0004/DECISION.md | 7 +++++++ docs/decisions/ADR-0005/DECISION.md | 4 +--- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index 71d0c3ca..6b0b9af9 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -193,6 +193,13 @@ loop for each field path in the UnifiedState tree The `PrivateStore` is a pluggable trait, allowing for backends like a local filesystem, S3, or another GATOS node. +#### 6.1.1 Encryption Profile (Normative) + +- AEAD algorithm: XChaCha20-Poly1305. +- Nonces: 24-byte (192-bit) nonces MUST be unique per key. Prefer deterministic nonces derived from the pointer digest via HKDF (domain-separated) or a crash-safe monotonic per-key counter stored in KMS. Random nonces are permitted only with a documented collision budget and monitoring. +- AAD: MUST include the pointer digest, actor id, and effective policy version (or policy_root) to bind authorization context to bytes at rest. +- Reuse is catastrophic and MUST be proven impossible by construction. + ### 6.2 Resolution Implementation The `gatosd` daemon exposes a secure endpoint for resolving Opaque Pointers. diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index 151b5275..add4aa35 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -78,6 +78,13 @@ Private data overlays are fundamentally tied to an actor's identity, not an ephe ``` refs/gatos/private/// ``` + +### 2. Encryption Algorithm & Nonce Discipline (Normative) + +- AEAD algorithm: Implementations MUST use XChaCha20-Poly1305 for encrypting private blobs referenced by Opaque Pointers. +- Nonces: 24-byte (192-bit) nonces MUST be unique per key. Implementations SHOULD use deterministic nonces derived from the pointer digest via HKDF (domain-separated), or a crash-safe, monotonic per-key counter stored in KMS. Random nonces are permitted only with a documented collision budget and active monitoring. +- Catastrophic reuse: Nonce reuse under the same key is catastrophic and MUST be proven impossible by construction (deterministic derivation) or by counter invariants. +- AAD binding: AEAD AAD MUST bind the pointer digest, the requester actor id, and the effective policy version so that verifiers can validate context and detect misuse. - **Public Refs:** The corresponding public projection lives in the main state namespace. ``` refs/gatos/state/public// diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index f7269615..423e0c0b 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -122,9 +122,7 @@ Errors (normative): ### 8) Privacy Interactions (ADR‑0004) -Nonces: Nonces MUST be unique per key. Prefer deterministic nonces derived from the pointer digest via HKDF (domain-separated) or a monotonic per-key counter in KMS. Random nonces are permitted only with a documented collision budget and monitoring. - -AAD: When using AEAD, bind the pointer digest (not a separate content_id), the actor id, and the policy version in the AAD so verifiers can validate context. +AEAD algorithm is pinned by ADR‑0004 to XChaCha20‑Poly1305. Nonces MUST be unique per key; prefer deterministic HKDF‑derived nonces (domain-separated) or crash‑safe per‑key counters in KMS. Random nonces are permitted only with a documented collision budget and monitoring. AAD MUST bind the pointer digest (not a separate content_id), the actor id, and the policy version so verifiers can validate context. - Payloads MUST NOT embed private overlay data. Use Opaque Pointers per privacy schema. For low‑entropy classes, include `ciphertext_digest` and omit plaintext digest in public pointers. From 5c4eecf5d87f92c4e3679f3adf0be9576e13a69b Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 15:18:12 -0800 Subject: [PATCH 36/68] docs(ADR-0005): note that commit_oid in consumer checkpoints MUST be lowercase hex --- docs/decisions/ADR-0005/DECISION.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 423e0c0b..31f1d356 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -118,7 +118,7 @@ Errors (normative): ### 7) Consumer Checkpoints -- `refs/gatos/consumers//` points to the last processed Shiplog commit OID. Portable JSON (optional): `schemas/v1/shiplog/consumer_checkpoint.schema.json`. +- `refs/gatos/consumers//` points to the last processed Shiplog commit OID. Portable JSON (optional): `schemas/v1/shiplog/consumer_checkpoint.schema.json`. The `commit_oid` value MUST be lowercase hex. ### 8) Privacy Interactions (ADR‑0004) From 07802d46b06f89ace05cf31018fd564225da2842 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 15:27:07 -0800 Subject: [PATCH 37/68] gatos-privacy: make digest optional; add validate() + content_id_from_canonical_bytes(); add tests; cleanup deps; schema negatives added; Makefile schema-negative extended; docs nits (USE-CASES newline) --- crates/gatos-privacy/src/lib.rs | 1 - 1 file changed, 1 deletion(-) diff --git a/crates/gatos-privacy/src/lib.rs b/crates/gatos-privacy/src/lib.rs index 28e54abb..9b39119c 100644 --- a/crates/gatos-privacy/src/lib.rs +++ b/crates/gatos-privacy/src/lib.rs @@ -39,4 +39,3 @@ pub enum Kind { pub enum Algo { Blake3, } - From 1edcda43594682c03ac54778b4190c953e6d622f Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 15:29:25 -0800 Subject: [PATCH 38/68] build: fix AJV runner to use 'npx -y ajv-cli@5' (remove redundant 'ajv' token) for compile/validate and script runner --- Makefile | 58 ++++++++++++++++++------------------- scripts/validate_schemas.sh | 4 +-- 2 files changed, 31 insertions(+), 31 deletions(-) diff --git a/Makefile b/Makefile index cce5a832..8c54737c 100644 --- a/Makefile +++ b/Makefile @@ -36,38 +36,38 @@ schema-compile: @bash -lc 'set -euo pipefail; \ if ! command -v node >/dev/null 2>&1; then \ echo "Node.js required (or run in CI)" >&2; exit 1; fi; \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/job/job_manifest.schema.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/job/proof_of_execution_envelope.schema.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proposal.schema.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/approval.schema.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/grant.schema.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/revocation.schema.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proof_of_consensus_envelope.schema.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/event_envelope.schema.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/deployment_trailer.schema.json && \ - npx -y ajv-cli@5 ajv compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/anchor.schema.json' + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/job/job_manifest.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/job/proof_of_execution_envelope.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proposal.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/approval.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/grant.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/revocation.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proof_of_consensus_envelope.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/event_envelope.schema.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/deployment_trailer.schema.json && \ + npx -y ajv-cli@5 compile --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/anchor.schema.json' schema-validate: @bash -lc 'set -euo pipefail; \ if ! command -v node >/dev/null 2>&1; then \ echo "Node.js required (or run in CI)" >&2; exit 1; fi; \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/job/job_manifest.schema.json -d examples/v1/job/manifest_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/job/proof_of_execution_envelope.schema.json -d examples/v1/job/poe_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proposal.schema.json -d examples/v1/governance/proposal_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/approval.schema.json -d examples/v1/governance/approval_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/grant.schema.json -d examples/v1/governance/grant_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/revocation.schema.json -d examples/v1/governance/revocation_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proof_of_consensus_envelope.schema.json -d examples/v1/governance/poc_envelope_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -d examples/v1/privacy/opaque_pointer_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/event_envelope.schema.json -d examples/v1/shiplog/event_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json -d examples/v1/shiplog/checkpoint_min.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/deployment_trailer.schema.json -d examples/v1/shiplog/trailer_min.json && \ - npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/anchor.schema.json -d examples/v1/shiplog/anchor_min.json' + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/job/job_manifest.schema.json -d examples/v1/job/manifest_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/job/proof_of_execution_envelope.schema.json -d examples/v1/job/poe_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proposal.schema.json -d examples/v1/governance/proposal_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/approval.schema.json -d examples/v1/governance/approval_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/grant.schema.json -d examples/v1/governance/grant_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/revocation.schema.json -d examples/v1/governance/revocation_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proof_of_consensus_envelope.schema.json -d examples/v1/governance/poc_envelope_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -d examples/v1/privacy/opaque_pointer_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/event_envelope.schema.json -d examples/v1/shiplog/event_min.json -r schemas/v1/common/ids.schema.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json -d examples/v1/shiplog/checkpoint_min.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/deployment_trailer.schema.json -d examples/v1/shiplog/trailer_min.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/anchor.schema.json -d examples/v1/shiplog/anchor_min.json' schema-negative: @bash -lc 'set -euo pipefail; \ @@ -75,9 +75,9 @@ schema-negative: echo "Node.js required (or run in CI)" >&2; exit 1; fi; \ echo "{\"governance\":{\"x\":{\"ttl\":\"P\"}}}" > /tmp/bad1.json; \ echo "{\"governance\":{\"x\":{\"ttl\":\"PT\"}}}" > /tmp/bad2.json; \ - if npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d /tmp/bad1.json; then \ + if npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d /tmp/bad1.json; then \ echo "Should have rejected ttl=P" >&2; exit 1; else echo "Rejected ttl=P as expected"; fi; \ - if npx -y ajv-cli@5 ajv validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d /tmp/bad2.json; then \ + if npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d /tmp/bad2.json; then \ echo "Should have rejected ttl=PT" >&2; exit 1; else echo "Rejected ttl=PT as expected"; fi' schemas: schema-compile schema-validate schema-negative diff --git a/scripts/validate_schemas.sh b/scripts/validate_schemas.sh index 2e9f0698..59f5a9e8 100755 --- a/scripts/validate_schemas.sh +++ b/scripts/validate_schemas.sh @@ -4,9 +4,9 @@ set -euo pipefail # Resolve AJV CLI (prefer local npx; fallback to dockerized node) AJV_RUNNER=() if command -v node >/dev/null 2>&1; then - AJV_RUNNER=(npx -y ajv-cli@5 ajv) + AJV_RUNNER=(npx -y ajv-cli@5) elif command -v docker >/dev/null 2>&1; then - AJV_RUNNER=(docker run --rm -v "$PWD:/work" -w /work node:20 npx -y ajv-cli@5 ajv) + AJV_RUNNER=(docker run --rm -v "$PWD:/work" -w /work node:20 npx -y ajv-cli@5) else echo "Need Node.js or Docker to run AJV validation" >&2 exit 1 From 66a7c2dbe441fa0f0f89f1b1e76ba5d2e93d696f Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 15:58:05 -0800 Subject: [PATCH 39/68] schemas: strict AJV fixes (types in if/then); consumer_checkpoint anyOf strictRequired safe sub-schemas --- schemas/v1/privacy/opaque_pointer.schema.json | 6 ++++-- schemas/v1/shiplog/consumer_checkpoint.schema.json | 12 ++---------- 2 files changed, 6 insertions(+), 12 deletions(-) diff --git a/schemas/v1/privacy/opaque_pointer.schema.json b/schemas/v1/privacy/opaque_pointer.schema.json index cfd18efc..40dd81c1 100644 --- a/schemas/v1/privacy/opaque_pointer.schema.json +++ b/schemas/v1/privacy/opaque_pointer.schema.json @@ -84,12 +84,14 @@ }, "required": [ "class" - ] + ], + "type": "object" } }, "required": [ "extensions" - ] + ], + "type": "object" }, "then": { "required": [ diff --git a/schemas/v1/shiplog/consumer_checkpoint.schema.json b/schemas/v1/shiplog/consumer_checkpoint.schema.json index a168e5a4..a017c51d 100644 --- a/schemas/v1/shiplog/consumer_checkpoint.schema.json +++ b/schemas/v1/shiplog/consumer_checkpoint.schema.json @@ -15,15 +15,7 @@ } }, "anyOf": [ - { - "required": [ - "ulid" - ] - }, - { - "required": [ - "commit_oid" - ] - } + { "type": "object", "properties": { "ulid": {} }, "required": ["ulid"] }, + { "type": "object", "properties": { "commit_oid": {} }, "required": ["commit_oid"] } ] } From 436bd3377bab83833da0b4bd9535d1c3332048ed Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 16:13:44 -0800 Subject: [PATCH 40/68] =?UTF-8?q?docs(FEATURES):=20add=20F6=20Privacy=20Op?= =?UTF-8?q?aque=20Pointers=20(ADR=E2=80=910004)=20and=20move=20Shiplog=20t?= =?UTF-8?q?o=20F7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/FEATURES.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 5706e3ee..feb8c576 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -212,3 +212,8 @@ See also: [ADR-0004](./decisions/ADR-0004/DECISION.md). - [ ] Golden: project a unified state, resolve pointer, and verify content matches original. - [ ] Edge: attempt to resolve a pointer with an invalid capability URI → DENY. - [ ] Failure: tamper with a private blob → digest mismatch on resolution. + +## F6 — Privacy Opaque Pointers (ADR‑0004) + +- See ADR‑0004 for the normative pointer envelope and privacy projection rules. +- Acceptance: pointers validate against schema; low‑entropy public pointers hide plaintext `digest` and include `ciphertext_digest`. From c06a46df2c113ff56d718484b8f018855c597c8b Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 16:14:08 -0800 Subject: [PATCH 41/68] build: use working AJV invocation (npx ajv-cli@5 ...); drop stray gatos-ledger-core dep from gatos-privacy --- Makefile | 2 +- crates/gatos-privacy/Cargo.toml | 1 - scripts/validate_schemas.sh | 2 +- 3 files changed, 2 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 8c54737c..94932a1d 100644 --- a/Makefile +++ b/Makefile @@ -62,7 +62,7 @@ schema-validate: npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/grant.schema.json -d examples/v1/governance/grant_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/revocation.schema.json -d examples/v1/governance/revocation_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proof_of_consensus_envelope.schema.json -d examples/v1/governance/poc_envelope_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.jsonjson && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -d examples/v1/privacy/opaque_pointer_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/event_envelope.schema.json -d examples/v1/shiplog/event_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json -d examples/v1/shiplog/checkpoint_min.json && \ diff --git a/crates/gatos-privacy/Cargo.toml b/crates/gatos-privacy/Cargo.toml index 631ea04c..579db903 100644 --- a/crates/gatos-privacy/Cargo.toml +++ b/crates/gatos-privacy/Cargo.toml @@ -4,7 +4,6 @@ version = "0.1.0" edition = "2021" [dependencies] -gatos-ledger-core = { path = "../gatos-ledger-core" } serde = { workspace = true, features = ["derive"] } serde_json = { workspace = true } blake3 = { workspace = true } diff --git a/scripts/validate_schemas.sh b/scripts/validate_schemas.sh index 59f5a9e8..64c4366b 100755 --- a/scripts/validate_schemas.sh +++ b/scripts/validate_schemas.sh @@ -6,7 +6,7 @@ AJV_RUNNER=() if command -v node >/dev/null 2>&1; then AJV_RUNNER=(npx -y ajv-cli@5) elif command -v docker >/dev/null 2>&1; then - AJV_RUNNER=(docker run --rm -v "$PWD:/work" -w /work node:20 npx -y ajv-cli@5) + AJV_RUNNER=(docker run --rm -v "$PWD:/work" -w /work node:20 npx -y ajv-cli@5 ajv) else echo "Need Node.js or Docker to run AJV validation" >&2 exit 1 From 69784bb9ba2d9f3c36266929712cd3a1ae4eb0ca Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 16:14:33 -0800 Subject: [PATCH 42/68] build: fix truncated path to governance_min.json in schema-validate target --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 94932a1d..8c54737c 100644 --- a/Makefile +++ b/Makefile @@ -62,7 +62,7 @@ schema-validate: npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/grant.schema.json -d examples/v1/governance/grant_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/revocation.schema.json -d examples/v1/governance/revocation_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/governance/proof_of_consensus_envelope.schema.json -d examples/v1/governance/poc_envelope_min.json -r schemas/v1/common/ids.schema.json && \ - npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.jsonjson && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/policy/governance_policy.schema.json -d examples/v1/policy/governance_min.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -d examples/v1/privacy/opaque_pointer_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/event_envelope.schema.json -d examples/v1/shiplog/event_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json -d examples/v1/shiplog/checkpoint_min.json && \ From bea1c900000d75c6f9284e516096b4db72e059e0 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 16:24:13 -0800 Subject: [PATCH 43/68] docs/schemas: choose ns as canonical envelope field; align schema/examples/ADR; add ns==Topic invariant; clean CLI samples; consolidate error taxonomy --- docs/decisions/ADR-0005/DECISION.md | 22 +++++++++++++++---- examples/v1/shiplog/event_min.json | 2 +- schemas/v1/shiplog/event_envelope.schema.json | 6 ++--- 3 files changed, 22 insertions(+), 8 deletions(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 31f1d356..93317a8f 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -38,7 +38,7 @@ Many integrations require an append‑only stream rather than only snapshot stat classDiagram class EventEnvelope { +string ulid - +string topic // topic namespace (e.g., "governance") + +string ns // topic namespace (e.g., "governance") +string type // logical event type +object payload // canonical JSON (JCS) +map refs // OPTIONAL cross-refs @@ -64,7 +64,7 @@ graph TD ### 3) Event Envelope (Schema) - Canonical JSON envelope at `schemas/v1/shiplog/event_envelope.schema.json` (draft‑2020‑12). -- Required fields: `ulid`, `topic`, `type`, `payload`. +- Required fields: `ulid`, `ns`, `type`, `payload`. - Optional `refs` (map) to link related state or IDs. - Privacy (ADR‑0004): Payload MUST NOT embed private overlay data. Redacted values MUST be replaced by `OpaquePointer` envelopes per `schemas/v1/privacy/opaque_pointer.schema.json`. @@ -106,6 +106,8 @@ MUST: validate the trailer against this schema, and write the exact JCS bytes ha ### 5) Append Semantics +Invariant: envelope.ns MUST equal the commit header `Topic:` value and the per-topic ref segment. + Append(`topic`, `envelope`): validate schema; compute `content_id = blake3(JCS(envelope))`; enforce monotone ULID per topic on this node; create commit with headers + trailer; CAS update `refs/gatos/shiplog//head`; return `(commit_oid, ulid, content_id)`. Errors (normative): @@ -143,13 +145,25 @@ $ gatosd shiplog append --topic governance --file event.json ok commit=8b1c1e4 content_id=blake3:2a6c… ulid=01HF4Y9Q1SM8Q7K9DK2R3V4AWB $ gatosd shiplog read --topic governance --since 01HF4Y9Q1SM8Q7K9DK2R3V4AWB --limit 2 -01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c… 8b1c1e4 {"ulid":"01HF4Y9...","topic":"governance",...} -01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2... 9f0aa21 {"ulid":"01HF4Y9...","topic":"governance",...} +01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c… 8b1c1e4 {"ulid":"01HF4Y9...","ns":"governance",...} +01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2... 9f0aa21 {"ulid":"01HF4Y9...","ns":"governance",...} $ gatosd shiplog checkpoint set --group analytics --topic governance --commit 8b1c1e4 ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 ``` +## Error Taxonomy (Normative) + +| Code | HTTP | Meaning | +|---|---:|---| +| AppendRejected | 409 | Not fast-forward (CAS failed) | +| TemporalOrder | 409 | ULID/timestamp monotonicity failure | +| DigestMismatch | 422 | Hash mismatch (body/envelope/JCS) | +| SigInvalid | 401/403 | Signature/attestation invalid | +| PolicyDenied | 403 | Policy decision denied | +| NotFound | 404 | Missing topic/checkpoint/anchor | +| CapabilityUnavailable | 503 | Dependent capability/KMS/blob store unavailable | + ## Consequences ## Error Taxonomy diff --git a/examples/v1/shiplog/event_min.json b/examples/v1/shiplog/event_min.json index 19aa9611..142bc19d 100644 --- a/examples/v1/shiplog/event_min.json +++ b/examples/v1/shiplog/event_min.json @@ -8,5 +8,5 @@ "refs": { "state": "blake3:0000000000000000000000000000000000000000000000000000000000000000" }, - "topic": "governance" + "ns": "governance" } diff --git a/schemas/v1/shiplog/event_envelope.schema.json b/schemas/v1/shiplog/event_envelope.schema.json index 226c571c..409c1f42 100644 --- a/schemas/v1/shiplog/event_envelope.schema.json +++ b/schemas/v1/shiplog/event_envelope.schema.json @@ -6,7 +6,7 @@ "additionalProperties": false, "required": [ "ulid", - "topic", + "ns", "type", "payload" ], @@ -33,10 +33,10 @@ }, "description": "Optional cross-references to related state roots or objects (blake3:...)." }, - "topic": { + "ns": { "type": "string", "pattern": "^[a-z][a-z0-9._-]{0,63}$", - "description": "Topic name (ASCII, lowercase start)." + "description": "Topic namespace (ASCII, lowercase start)." } }, "description": "Canonical Shiplog event envelope. Content-Id = blake3(JCS(envelope)). Precision‑sensitive values (e.g., money/time) MUST be encoded as integers or strings. Payload MUST NOT embed private data; replace with Opaque Pointer per privacy schema." From 75012bb5180f3f112d703795a3c8d3bcdee1a2ef Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 18:02:21 -0800 Subject: [PATCH 44/68] ADR-0005: ns/topic invariant clarifications; CLI sample cleanup; replace duplicated taxonomy with problem+json example --- docs/decisions/ADR-0005/DECISION.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 93317a8f..d75aea50 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -130,7 +130,7 @@ AEAD algorithm is pinned by ADR‑0004 to XChaCha20‑Poly1305. Nonces MUST be u ### 9) Governance and Ledger Interactions -- Governance (ADR‑0003): Should emit Shiplog events under `topic="governance"`. +- Governance (ADR‑0003): Should emit Shiplog events under `topic="governance"`; envelopes carry `ns="governance"` and the commit header sets `Topic: governance`. - Ledger mirroring: MAY mirror ledger events; must preserve envelope determinism. ### 10) Security Considerations @@ -145,8 +145,8 @@ $ gatosd shiplog append --topic governance --file event.json ok commit=8b1c1e4 content_id=blake3:2a6c… ulid=01HF4Y9Q1SM8Q7K9DK2R3V4AWB $ gatosd shiplog read --topic governance --since 01HF4Y9Q1SM8Q7K9DK2R3V4AWB --limit 2 -01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c… 8b1c1e4 {"ulid":"01HF4Y9...","ns":"governance",...} -01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2... 9f0aa21 {"ulid":"01HF4Y9...","ns":"governance",...} +01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2A6C… 8b1c1e4 {"ulid":"01HF4Y9Q1SM8Q7K9DK2R4V5CXD","ns":"governance","type":"proposal.created","payload":{…}} +01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:C1D2… 9f0aa21 {"ulid":"01HF4Y9Q1SM8Q7K9DK2R4V5CXE","ns":"governance","type":"proposal.created","payload":{…}} $ gatosd shiplog checkpoint set --group analytics --topic governance --commit 8b1c1e4 ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 @@ -166,18 +166,18 @@ ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 ## Consequences -## Error Taxonomy +Clients SHOULD return a problem+json response with a stable `code` plus HTTP status. Example: -| Code | HTTP | Meaning | -|:-----|:----:|:--------| -| AppendRejected | 409 | Not fast-forward (CAS failed) | -| TemporalOrder | 409 | ULID/timestamp monotonicity failure | -| PolicyFail | 403 | Policy decision denied | -| SigInvalid | 422 | Signature/attestation failed | -| DigestMismatch | 422 | Hash mismatch on body/envelope | -| CapabilityUnavailable | 503 | Dependent capability/KMS/storage unavailable | - -Clients SHOULD return a problem+json response with a stable `code` plus HTTP status. +```json +{ + "type": "https://gatos.dev/problems/append-rejected", + "title": "AppendRejected", + "status": 409, + "code": "AppendRejected", + "detail": "Not fast-forward (CAS failed)", + "instance": "urn:commit:8b1c1e4" +} +``` Pros: clean integration surface; deterministic envelopes; replay + analytics; explicit privacy. From 31276bd21bc1d43c13177c03264c00609fb0c839 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 18:36:09 -0800 Subject: [PATCH 45/68] ADR-0005: polish per review - Change sample ref path to demo/head and label subgraph as (sample). - Make CLI read example copy-pasteable (valid JSON in-line; ellipsis outside JSON). - Add CLI invariant: --topic MUST equal envelope.ns and ref segment. - Note trailer repo_head is top-level only. - Add numeric discipline MUST to canonicalization section. - Fix schema refs description to use blake3:<64-hex>. --- docs/decisions/ADR-0005/DECISION.md | 9 +++++---- .../generated/docs_TECH-SPEC__15850d53f4__mermaid_13.svg | 2 +- .../generated/docs_TECH-SPEC__15850d53f4__mermaid_7.svg | 2 +- ...ecisions_ADR-0005_DECISION__03ffbed862__mermaid_1.svg | 2 +- ...ecisions_ADR-0005_DECISION__03ffbed862__mermaid_2.svg | 2 +- schemas/v1/shiplog/event_envelope.schema.json | 2 +- 6 files changed, 10 insertions(+), 9 deletions(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index d75aea50..6e36f09e 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -33,6 +33,7 @@ Many integrations require an append‑only stream rather than only snapshot stat - Envelope canonicalization: RFC 8785 JSON Canonicalization Scheme (JCS). The event Content‑Id is `blake3(JCS(envelope))`. - ULID: 26‑char Crockford base32, uppercase, excluding I/L/O/U (`^[0-9A-HJKMNP-TV-Z]{26}$`). - Hashes: content digests are `blake3:<64‑hex>` per `schemas/v1/common/ids.schema.json`. +- Numeric discipline: precision‑sensitive values (e.g., money/time) MUST be encoded as integers or strings. ```mermaid classDiagram @@ -53,8 +54,8 @@ classDiagram ```mermaid graph TD - subgraph "Git Refs" - H1[refs/gatos/shiplog/orders/head]-->C1 + subgraph "Git Refs (sample)" + H1[refs/gatos/shiplog/demo/head]-->C1 C1((e1))-->C2((e2))-->C3((e3)) end C1:::ev; C2:::ev; C3:::ev @@ -145,8 +146,8 @@ $ gatosd shiplog append --topic governance --file event.json ok commit=8b1c1e4 content_id=blake3:2a6c… ulid=01HF4Y9Q1SM8Q7K9DK2R3V4AWB $ gatosd shiplog read --topic governance --since 01HF4Y9Q1SM8Q7K9DK2R3V4AWB --limit 2 -01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2A6C… 8b1c1e4 {"ulid":"01HF4Y9Q1SM8Q7K9DK2R4V5CXD","ns":"governance","type":"proposal.created","payload":{…}} -01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:C1D2… 9f0aa21 {"ulid":"01HF4Y9Q1SM8Q7K9DK2R4V5CXE","ns":"governance","type":"proposal.created","payload":{…}} +01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2A6C… 8b1c1e4 {"ulid":"01HF4Y9Q1SM8Q7K9DK2R4V5CXD","ns":"governance","type":"proposal.created","payload":{}} +01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:C1D2… 9f0aa21 {"ulid":"01HF4Y9Q1SM8Q7K9DK2R4V5CXE","ns":"governance","type":"proposal.approved","payload":{}} $ gatosd shiplog checkpoint set --group analytics --topic governance --commit 8b1c1e4 ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_13.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_13.svg index fc80a41d..119564aa 100644 --- a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_13.svg +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_13.svg @@ -1 +1 @@ -2025-01-052025-01-122025-01-192025-01-262025-02-022025-02-092025-02-162025-02-232025-03-022025-03-092025-03-162025-03-232025-03-302025-04-06Mirror Mode Shadow Consumers Canary (10%) Full Cutover Phase A: MirrorPhase B: ShadowPhase C: Dual-ReadPhase D: CutoverGATOS Migration Strategy +2025-01-052025-01-122025-01-192025-01-262025-02-022025-02-092025-02-162025-02-232025-03-022025-03-092025-03-162025-03-232025-03-302025-04-06Mirror Mode Shadow Consumers Canary (10%) Full Cutover Phase A: MirrorPhase B: ShadowPhase C: Dual-ReadPhase D: CutoverGATOS Migration Strategy diff --git a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_7.svg b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_7.svg index c227ee9c..477574ad 100644 --- a/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_7.svg +++ b/docs/diagrams/generated/docs_TECH-SPEC__15850d53f4__mermaid_7.svg @@ -1 +1 @@ -gatosdClient (SDK/CLI)gatosdClient (SDK/CLI)loop[Subscription Stream]{"type":"append_event", "id":"01A", "ns":"...", "event":{...}}{"ok":true, "id":"01A", "commit_id":"..."}{"type":"bus.subscribe", "id":"01C", "topic":"..."}{"ack":true, "id":"01C"}{"type":"bus.message", "id":"01C", "topic":"...", "payload":{...}} +gatosdClient (SDK/CLI)gatosdClient (SDK/CLI)loop[Subscription Stream]{"type":"append_event", "id":"01A", "topic":"...", "event":{...}}{"ok":true, "id":"01A", "commit_id":"..."}{"type":"bus.subscribe", "id":"01C", "topic":"..."}{"ack":true, "id":"01C"}{"type":"bus.message", "id":"01C", "topic":"...", "payload":{...}} diff --git a/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_1.svg b/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_1.svg index 35090dc8..c5bf78ea 100644 --- a/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_1.svg +++ b/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_1.svg @@ -1 +1 @@ -
EventEnvelope
+string ulid
+string type // logical event type
+map refs // OPTIONAL cross-refs
+string topic // topic namespace(e.g., "governance")
+object payload // canonical JSON(JCS)
 +
EventEnvelope
+string ulid
+string type // logical event type
+map refs // OPTIONAL cross-refs
+string ns // topic namespace(e.g., "governance")
+object payload // canonical JSON(JCS)
 diff --git a/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_2.svg b/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_2.svg index 66955596..9fd28dcf 100644 --- a/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_2.svg +++ b/docs/diagrams/generated/docs_decisions_ADR-0005_DECISION__03ffbed862__mermaid_2.svg @@ -1 +1 @@ -
Git Refs
e1
refs/gatos/shiplog/orders/head
e2
e3
 +
Git Refs (sample)
e1
refs/gatos/shiplog/demo/head
e2
e3
 diff --git a/schemas/v1/shiplog/event_envelope.schema.json b/schemas/v1/shiplog/event_envelope.schema.json index 409c1f42..fbd93d76 100644 --- a/schemas/v1/shiplog/event_envelope.schema.json +++ b/schemas/v1/shiplog/event_envelope.schema.json @@ -31,7 +31,7 @@ "additionalProperties": { "$ref": "../common/ids.schema.json#/$defs/blake3Digest" }, - "description": "Optional cross-references to related state roots or objects (blake3:...)." + "description": "Optional cross-references to related state roots or objects (blake3:<64-hex>)." }, "ns": { "type": "string", From 3618a2c64359398103dc58f44ec57f4c2d097c14 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 18:40:25 -0800 Subject: [PATCH 46/68] docs: markdownlint fixes in ADR-0004 and ADR-0005 fences/blanks --- docs/FEATURES.md | 1 + docs/SPEC.md | 23 ++-- docs/TECH-SPEC.md | 26 +++-- docs/USE-CASES.md | 2 +- docs/decisions/ADR-0004/DECISION.md | 167 ++++++++++++++-------------- docs/decisions/ADR-0005/DECISION.md | 6 +- 6 files changed, 121 insertions(+), 104 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index feb8c576..fafd7571 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -185,6 +185,7 @@ Each feature includes user stories per relevant stakeholders (format requested), - [ ] Golden: metrics show non-zero counters post workload - [ ] Edge: cache stale → doctor recommends rebuild - [ ] Failure: FF-only violation → doctor flags critical + --- ## F9 — Hybrid Privacy Model diff --git a/docs/SPEC.md b/docs/SPEC.md index 728c4544..783a4df4 100644 --- a/docs/SPEC.md +++ b/docs/SPEC.md @@ -295,8 +295,8 @@ GATOS supports a hybrid privacy model where state can be separated into a verifi The State Engine (`gatos-echo`) can be configured with privacy rules. When folding history, it first computes a `UnifiedState` containing all data. It then applies the privacy rules to produce a `PublicState` and a set of `PrivateBlobs`. -- **`PublicState`**: Contains only public data and Opaque Pointers. This is committed to the public `refs/gatos/state/public/...` namespace and is globally verifiable. -- **`PrivateBlobs`**: The raw data that was redacted or pointerized. This data is stored in a separate, private store (e.g., a local directory, a private object store) and is addressed by its content hash. +- **`PublicState`**: Contains only public data and Opaque Pointers. This is committed to the public `refs/gatos/state/public/...` namespace and is globally verifiable. +- **`PrivateBlobs`**: The raw data that was redacted or pointerized. This data is stored in a separate, private store (e.g., a local directory, a private object store) and is addressed by its content hash. Any commit that is the result of a privacy projection **MUST** include trailers indicating the number of redactions and pointers created. @@ -323,39 +323,46 @@ classDiagram } ``` -- `digest`: The **REQUIRED** `blake3` hash of the plaintext. For low‑entropy privacy classes, the public pointer MUST NOT expose this value. -- `ciphertext_digest`: The `blake3` hash of the stored ciphertext. For low‑entropy privacy classes, this field MUST be present in the public pointer. -- `size`: The size of the private blob in bytes (RECOMMENDED). -- `location`: A **REQUIRED** stable URI indicating where the blob can be fetched (e.g., `gatos-node://ed25519:`, `s3://bucket/key`). Do not embed pre‑signed tokens. -- `capability`: A **REQUIRED** reference to the authn/z + decryption mechanism (e.g., `gatos-key://...`, `kms://...`). It MUST NOT embed secrets; resolution occurs at the policy layer. +- `digest`: The **REQUIRED** `blake3` hash of the plaintext. For low‑entropy privacy classes, the public pointer MUST NOT expose this value. +- `ciphertext_digest`: The `blake3` hash of the stored ciphertext. For low‑entropy privacy classes, this field MUST be present in the public pointer. +- `size`: The size of the private blob in bytes (RECOMMENDED). +- `location`: A **REQUIRED** stable URI indicating where the blob can be fetched (e.g., `gatos-node://ed25519:`, `s3://bucket/key`). Do not embed pre‑signed tokens. +- `capability`: A **REQUIRED** reference to the authn/z + decryption mechanism (e.g., `gatos-key://...`, `kms://...`). It MUST NOT embed secrets; resolution occurs at the policy layer. The pointer itself is canonicalized via RFC 8785 JCS and its `content_id` is `blake3(JCS(pointer_json))`. ### 7.3 Pointer Resolution Endpoint and AuthN: + - Clients MUST resolve via `POST /gatos/private/blobs/resolve` with body `{ "digest": "blake3:", "want": "plaintext"|"ciphertext" }` and `Authorization: Bearer `. - Tokens MUST include standard claims (`sub`, `aud`, `method`, `path`, `exp`, `nbf`); skew tolerance ±300s. 401 for authn failures; 403 for policy denials. Verification Steps: + 1. Fetch the ciphertext blob from `location` via the node’s resolver endpoint. 2. Acquire the necessary keys via the `capability` reference (policy-driven; no secrets in the pointer). 3. Decrypt. Compute `blake3(ciphertext)` and compare with `ciphertext_digest` when present; compute `blake3(plaintext)` and compare with `digest` when exposed. Any mismatch MUST yield `DigestMismatch`. 4. Servers SHOULD return `X-BLAKE3-Digest` and `Digest: sha-256=…` headers for response integrity. Error Taxonomy: + - `Unauthorized` (401), `Forbidden` (403), `NotFound` (404), `DigestMismatch` (422), `CapabilityUnavailable` (503), `PolicyDenied` (403). - + Optional HTTP Message Signatures profile (RFC 9421): + - As an alternative to JWT, clients MAY sign `@method`, `@target-uri`, `date`, `host`, `content-digest` and send `Signature-Input`/`Signature` headers. Servers SHOULD still emit `Digest` and `X-BLAKE3-Digest` response headers. Pointer Rotation (Rekey): + 1) fetch ciphertext; 2) decrypt; 3) re‑encrypt per new capability; 4) store new ciphertext; 5) emit rotation event updating pointer fields (capability/location). `digest` (plaintext) MUST remain stable. Add trailer `Privacy-Pointer-Rotations: `. Namespacing: + - `refs/gatos/private//…` holds private overlay indices/metadata only; workspace mirror is `gatos/private//…`. Blobs live in external stores keyed by digest. Canonicalization: + - All JSON labeled as canonical MUST use RFC 8785 JCS; non‑JSON maps MUST be ordered lexicographically by lowercase UTF‑8 keys. This process guarantees that even though the data is stored privately, its integrity is verifiable against the public ledger. diff --git a/docs/TECH-SPEC.md b/docs/TECH-SPEC.md index 6b0b9af9..a31c35c7 100644 --- a/docs/TECH-SPEC.md +++ b/docs/TECH-SPEC.md @@ -204,28 +204,32 @@ The `PrivateStore` is a pluggable trait, allowing for backends like a local file The `gatosd` daemon exposes a secure endpoint for resolving Opaque Pointers. -- Endpoint: `POST /gatos/private/blobs/resolve` -- Content-Type: `application/json` -- Request body (JCS canonical JSON): +- Endpoint: `POST /gatos/private/blobs/resolve` +- Content-Type: `application/json` +- Request body (JCS canonical JSON): + ```json { "digest": "blake3:", "want": "plaintext" } ``` - - `want` OPTIONAL: `"plaintext" | "ciphertext"` (default `"plaintext"`). -- Authentication: `Authorization: Bearer ` - - Claims (example): `iss`, `sub` (ed25519:), `aud` ("gatos-node:"), `exp`, `nbf`, `jti`, `method` ("POST"), `path` ("/gatos/private/blobs/resolve"), `digest` (MUST match body.digest). - - Clock skew tolerance: ±300 seconds. -- Authorization: Node evaluates policy for `` on ``. -- Response (200 OK): - - Headers: `Digest: sha-256=`, `X-BLAKE3-Digest: blake3:` - - Body: requested bytes (ciphertext or plaintext). + + - `want` OPTIONAL: `"plaintext" | "ciphertext"` (default `"plaintext"`). +- Authentication: `Authorization: Bearer ` + - Claims (example): `iss`, `sub` (ed25519:), `aud` ("gatos-node:"), `exp`, `nbf`, `jti`, `method` ("POST"), `path` ("/gatos/private/blobs/resolve"), `digest` (MUST match body.digest). + - Clock skew tolerance: ±300 seconds. +- Authorization: Node evaluates policy for `` on ``. +- Response (200 OK): + - Headers: `Digest: sha-256=`, `X-BLAKE3-Digest: blake3:` + - Body: requested bytes (ciphertext or plaintext). Errors: 401 Unauthorized, 403 Forbidden, 404 Not Found, 422 DigestMismatch, 503 CapabilityUnavailable. Optional profile (HTTP Message Signatures, RFC 9421): + - Clients MAY authenticate by signing components: `@method`, `@target-uri`, `date`, `host`, `content-digest` (SHA-256 over request body) and sending `Signature-Input: sig1=...` and `Signature: sig1=::`. - Servers STILL apply policy and SHOULD return `Digest` and `X-BLAKE3-Digest` headers. Pointer Rotation (Rekey): + - Implement a rotation that: (1) fetches; (2) decrypts; (3) re‑encrypts; (4) stores; (5) emits an audit event updating pointer fields while keeping plaintext `digest` stable. Add trailer `Privacy-Pointer-Rotations: ` when a projection commit includes rotations. --- diff --git a/docs/USE-CASES.md b/docs/USE-CASES.md index 26e7100c..7b56a6d7 100644 --- a/docs/USE-CASES.md +++ b/docs/USE-CASES.md @@ -92,4 +92,4 @@ This document illustrates practical scenarios where GATOS provides unique value. |---|---| |**Goal** | Manage customer data (PII) in a way that is both auditable and privacy-preserving. | | **How** | A privacy policy projects the unified state into a public state with PII replaced by Opaque Pointers. The private data lives in an actor-anchored, encrypted blob store. | -| **Why GATOS** | Provides a verifiable public audit trail ("a user's data was accessed") without ever exposing the private data ("the user's address is...") to the public ledger. Access is gated by cryptographic capabilities. | \ No newline at end of file +| **Why GATOS** | Provides a verifiable public audit trail ("a user's data was accessed") without ever exposing the private data ("the user's address is...") to the public ledger. Access is gated by cryptographic capabilities. | diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index add4aa35..ece63bc5 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -26,12 +26,12 @@ This ADR makes the hybrid model **normative, deterministic, and provable**. It e This model is a direct application of the GATOS Morphology Calculus. -1. **Shape Categories**: We define three categories of shapes: - * `Sh_Unified`: The category of shapes containing both public and private data. - * `Sh_Public`: The category of shapes containing only public data and opaque pointers. - * `Sh_Private`: The category of shapes containing only the private data blobs. +1. **Shape Categories**: We define three categories of shapes: + * `Sh_Unified`: The category of shapes containing both public and private data. + * `Sh_Public`: The category of shapes containing only public data and opaque pointers. + * `Sh_Private`: The category of shapes containing only the private data blobs. -2. **Projection as a Functor**: The privacy model is implemented as a functor, `Proj`, which maps shapes and morphisms from the unified category to the public category. +2. **Projection as a Functor**: The privacy model is implemented as a functor, `Proj`, which maps shapes and morphisms from the unified category to the public category. `Proj: Sh_Unified -> Sh_Public` This functor applies the privacy policy rules (`redact`, `pointerize`) to transform a unified shape into its public projection. The private data is extracted into `Sh_Private` during this process. @@ -63,7 +63,7 @@ This model is a direct application of the GATOS Morphology Calculus. style P1 fill:#cde,stroke:#333 style P2 fill:#cde,stroke:#333 - ``` + ```text This ensures that the transformation is structure-preserving and that the public history remains a valid, deterministic projection of the complete history. @@ -73,20 +73,22 @@ This ensures that the transformation is structure-preserving and that the public Private data overlays are fundamentally tied to an actor's identity, not an ephemeral session. This anchors private data within the GATOS trust graph. -- **Actor ID:** The canonical identifier for an actor, e.g., `ed25519:`. -- **Private Refs:** Private data is stored under refs namespaced by the actor ID. - ``` +* **Actor ID:** The canonical identifier for an actor, e.g., `ed25519:`. +* **Private Refs:** Private data is stored under refs namespaced by the actor ID. + + ```text refs/gatos/private/// - ``` + ```text ### 2. Encryption Algorithm & Nonce Discipline (Normative) -- AEAD algorithm: Implementations MUST use XChaCha20-Poly1305 for encrypting private blobs referenced by Opaque Pointers. -- Nonces: 24-byte (192-bit) nonces MUST be unique per key. Implementations SHOULD use deterministic nonces derived from the pointer digest via HKDF (domain-separated), or a crash-safe, monotonic per-key counter stored in KMS. Random nonces are permitted only with a documented collision budget and active monitoring. -- Catastrophic reuse: Nonce reuse under the same key is catastrophic and MUST be proven impossible by construction (deterministic derivation) or by counter invariants. -- AAD binding: AEAD AAD MUST bind the pointer digest, the requester actor id, and the effective policy version so that verifiers can validate context and detect misuse. -- **Public Refs:** The corresponding public projection lives in the main state namespace. - ``` +* AEAD algorithm: Implementations MUST use XChaCha20-Poly1305 for encrypting private blobs referenced by Opaque Pointers. +* Nonces: 24-byte (192-bit) nonces MUST be unique per key. Implementations SHOULD use deterministic nonces derived from the pointer digest via HKDF (domain-separated), or a crash-safe, monotonic per-key counter stored in KMS. Random nonces are permitted only with a documented collision budget and active monitoring. +* Catastrophic reuse: Nonce reuse under the same key is catastrophic and MUST be proven impossible by construction (deterministic derivation) or by counter invariants. +* AAD binding: AEAD AAD MUST bind the pointer digest, the requester actor id, and the effective policy version so that verifiers can validate context and detect misuse. +* **Public Refs:** The corresponding public projection lives in the main state namespace. + + ```text refs/gatos/state/public// ``` @@ -106,16 +108,16 @@ classDiagram +string capability // MUST NOT embed secrets +object extensions // forward-compatible } -``` +```text -- **`digest`**: The content-address of the private plaintext (`blake3(plaintext_bytes)`). This is the immutable link between the public and private worlds. -- **`ciphertext_digest`**: The content-address of the stored ciphertext (`blake3(ciphertext_bytes)`). For low‑entropy privacy classes (see Policy Hooks), the public pointer **MUST** include `ciphertext_digest` and policy **MUST NOT** expose the plaintext digest publicly. -- **`location`**: A URI indicating where to resolve the blob. Supported schemes include: - - `gatos-node://ed25519:`: Resolve via the GATOS trust graph. - - `https://...`, `s3://...`, `ipfs://...`: Standard distributed storage. - - `file:///...`: For local development and testing. -- **`capability`**: A reference identifying the authorization and decryption mechanism required to access the blob. It **MUST NOT** embed secrets or pre‑signed tokens. It SHOULD be a stable identifier (e.g., `gatos-key://v1/aes-256-gcm/` or `kms://...`) that can be resolved privately at the policy layer. - - Pointers MAY publish a non‑sensitive label and keep resolver details private via policy. Implementations MAY also place auxiliary hints inside `extensions`. +* **`digest`**: The content-address of the private plaintext (`blake3(plaintext_bytes)`). This is the immutable link between the public and private worlds. +* **`ciphertext_digest`**: The content-address of the stored ciphertext (`blake3(ciphertext_bytes)`). For low‑entropy privacy classes (see Policy Hooks), the public pointer **MUST** include `ciphertext_digest` and policy **MUST NOT** expose the plaintext digest publicly. +* **`location`**: A URI indicating where to resolve the blob. Supported schemes include: + * `gatos-node://ed25519:`: Resolve via the GATOS trust graph. + * `https://...`, `s3://...`, `ipfs://...`: Standard distributed storage. + * `file:///...`: For local development and testing. +* **`capability`**: A reference identifying the authorization and decryption mechanism required to access the blob. It **MUST NOT** embed secrets or pre‑signed tokens. It SHOULD be a stable identifier (e.g., `gatos-key://v1/aes-256-gcm/` or `kms://...`) that can be resolved privately at the policy layer. + * Pointers MAY publish a non‑sensitive label and keep resolver details private via policy. Implementations MAY also place auxiliary hints inside `extensions`. The canonical `content_id` of the pointer itself is `blake3(JCS(pointer_json))`, where `JCS(…)` denotes RFC 8785 JSON Canonicalization Scheme applied to UTF‑8 bytes. This rule is normative for all canonical JSON in GATOS (pointers, governance envelopes, any JSON state snapshots). @@ -125,16 +127,17 @@ The canonical `content_id` of the pointer itself is `blake3(JCS(pointer_json))`, The State Engine (`gatos-echo`) is responsible for executing the projection. -1. It computes a **UnifiedState** by folding the complete event history. -2. It consults the **Privacy Policy** (`.gatos/policy.yaml`). -3. It traverses the `UnifiedState` tree, applying `redact` or `pointerize` rules. - - `redact`: The field is removed from the public state. - - `pointerize`: The field's value is stored as a private blob, and an Opaque Pointer is substituted in the public state. -4. The resulting `PublicState` is committed to the public refs, and the `Private Blobs` are persisted to their specified `location`. +1. It computes a **UnifiedState** by folding the complete event history. +2. It consults the **Privacy Policy** (`.gatos/policy.yaml`). +3. It traverses the `UnifiedState` tree, applying `redact` or `pointerize` rules. + * `redact`: The field is removed from the public state. + * `pointerize`: The field's value is stored as a private blob, and an Opaque Pointer is substituted in the public state. +4. The resulting `PublicState` is committed to the public refs, and the `Private Blobs` are persisted to their specified `location`. Determinism Requirements: -- All JSON artifacts produced during projection (including Opaque Pointers) MUST be canonicalized with RFC 8785 JCS prior to hashing. -- When non‑JSON maps are materialized (e.g., Git tree entries), keys MUST be ordered lexicographically by their lowercase UTF‑8 bytes. + +* All JSON artifacts produced during projection (including Opaque Pointers) MUST be canonicalized with RFC 8785 JCS prior to hashing. +* When non‑JSON maps are materialized (e.g., Git tree entries), keys MUST be ordered lexicographically by their lowercase UTF‑8 bytes. ```mermaid sequenceDiagram @@ -149,42 +152,44 @@ sequenceDiagram E->>E: 4. Apply rules to create PublicState + PrivateBlobs E->>L: 5. Commit PublicState to public refs E->>PS: 6. Store PrivateBlobs by digest -``` +```text ### 4. Pointer Resolution Protocol (Normative) Authentication semantics are aligned with HTTP. We adopt a simple, interoperable model (JWT default; HTTP Message Signatures optional): -- **Endpoint**: `POST /gatos/private/blobs/resolve` -- **Request Body (application/json; JCS canonical form)**: +* **Endpoint**: `POST /gatos/private/blobs/resolve` +* **Request Body (application/json; JCS canonical form)**: `{ "digest": "blake3:", "want": "plaintext"|"ciphertext" }` -- **Authorization**: `Authorization: Bearer ` - - Claims MUST include: `sub` (ed25519:), `aud` (node id or URL), `method` ("POST"), `path` ("/gatos/private/blobs/resolve"), `exp`, and `nbf`. - - Clock skew tolerance: ±300 seconds. - - On missing/invalid token: `401 Unauthorized`. On policy denial: `403 Forbidden`. +* **Authorization**: `Authorization: Bearer ` + * Claims MUST include: `sub` (ed25519:), `aud` (node id or URL), `method` ("POST"), `path` ("/gatos/private/blobs/resolve"), `exp`, and `nbf`. + * Clock skew tolerance: ±300 seconds. + * On missing/invalid token: `401 Unauthorized`. On policy denial: `403 Forbidden`. A client resolving an Opaque Pointer **MUST** follow this protocol: -1. **Parse Pointer**: Extract `digest`, optional `ciphertext_digest`, `location`, and `capability`. -2. **Fetch Blob**: - - If `gatos-node://`, resolve the actor's endpoint from the trust graph, then `POST /gatos/private/blobs/resolve` with the body above. - - The node **MUST** verify the bearer token and enforce policy before returning the blob. -3. **Acquire Capability**: - - Resolve the `capability` reference via the configured key system (KMS, key server). Secrets MUST NOT be embedded in the pointer. -4. **Decrypt and Verify**: - - Decrypt the fetched blob using the resolved key and AAD parameters (see Security Notes). - - Compute `blake3(plaintext)` and compare to `digest` if published; compute `blake3(ciphertext)` and compare to `ciphertext_digest` if published. A mismatch **MUST** produce `DigestMismatch`. +1. **Parse Pointer**: Extract `digest`, optional `ciphertext_digest`, `location`, and `capability`. +2. **Fetch Blob**: + * If `gatos-node://`, resolve the actor's endpoint from the trust graph, then `POST /gatos/private/blobs/resolve` with the body above. + * The node **MUST** verify the bearer token and enforce policy before returning the blob. +3. **Acquire Capability**: + * Resolve the `capability` reference via the configured key system (KMS, key server). Secrets MUST NOT be embedded in the pointer. +4. **Decrypt and Verify**: + * Decrypt the fetched blob using the resolved key and AAD parameters (see Security Notes). + * Compute `blake3(plaintext)` and compare to `digest` if published; compute `blake3(ciphertext)` and compare to `ciphertext_digest` if published. A mismatch **MUST** produce `DigestMismatch`. Response headers on success: -``` + +```text Content-Type: application/octet-stream X-BLAKE3-Digest: blake3: Digest: sha-256= -``` +```text Optional HTTP Message Signatures profile (RFC 9421): -- Clients MAY authenticate by signing `@method`, `@target-uri`, `date`, `host`, `content-digest` (SHA‑256 of the JSON body) and sending `Signature-Input` and `Signature` headers. -- Servers SHOULD still return `Digest` and `X-BLAKE3-Digest` headers for response integrity. + +* Clients MAY authenticate by signing `@method`, `@target-uri`, `date`, `host`, `content-digest` (SHA‑256 of the JSON body) and sending `Signature-Input` and `Signature` headers. +* Servers SHOULD still return `Digest` and `X-BLAKE3-Digest` headers for response integrity. ```mermaid sequenceDiagram @@ -204,7 +209,7 @@ sequenceDiagram else Unauthorized PN-->>C: 4. Return 401/403 end -``` +```text ### 5. Policy Hooks (Normative) @@ -225,7 +230,7 @@ privacy: location: "gatos-node://ed25519:" - select: "path.to.transient.data" action: "redact" -``` +```text The `select` syntax will use a simple path-matching language (e.g., glob patterns) defined by the policy engine. @@ -233,11 +238,11 @@ The `select` syntax will use a simple path-matching language (e.g., glob pattern To make privacy operations transparent and auditable, any commit that creates a `PublicState` from a projection **MUST** include the following trailers: -``` +```text Privacy-Redactions: 3 Privacy-Pointers: 12 Privacy-Pointer-Rotations: 1 -``` +```text This provides a simple, top-level indicator that a projection has occurred, prompting auditors to look deeper if necessary. @@ -245,49 +250,49 @@ This provides a simple, top-level indicator that a projection has occurred, prom ### Pros -- **Provable Privacy**: The model is grounded in the Morphology Calculus, making it verifiable. -- **Decoupled Storage**: Private data can live in any storage system (S3, IPFS, local disk) without affecting the public ledger's logic. -- **Integrated Auth/Authz**: By tying pointers to actor identities and capabilities, access to private data is governed by the existing GATOS trust and policy model. -- **Preserves Verifiability**: The `PublicState` remains globally verifiable, as pointers are just content-addressed links. +* **Provable Privacy**: The model is grounded in the Morphology Calculus, making it verifiable. +* **Decoupled Storage**: Private data can live in any storage system (S3, IPFS, local disk) without affecting the public ledger's logic. +* **Integrated Auth/Authz**: By tying pointers to actor identities and capabilities, access to private data is governed by the existing GATOS trust and policy model. +* **Preserves Verifiability**: The `PublicState` remains globally verifiable, as pointers are just content-addressed links. ### Cons -- **Increased Complexity**: Resolution requires network requests and interaction with key management systems, adding latency and potential points of failure. -- **Operational Overhead**: Operators must manage the private blob stores and ensure their availability and security. +* **Increased Complexity**: Resolution requires network requests and interaction with key management systems, adding latency and potential points of failure. +* **Operational Overhead**: Operators must manage the private blob stores and ensure their availability and security. ## Feature Payoff -- **Secure PII/Secret Storage**: Store sensitive data off-chain while retaining an auditable link to it. -- **Large Artifact Management**: Handle large binaries (ML models, videos) without bloating the Git repository. -- **Compliant Data Sharing**: Share a public, redacted dataset with third parties while retaining private access to the full, unified view. -- **Federated Learning**: Different actors can hold private models locally, referenced by pointers in a public "training plan" shape. +* **Secure PII/Secret Storage**: Store sensitive data off-chain while retaining an auditable link to it. +* **Large Artifact Management**: Handle large binaries (ML models, videos) without bloating the Git repository. +* **Compliant Data Sharing**: Share a public, redacted dataset with third parties while retaining private access to the full, unified view. +* **Federated Learning**: Different actors can hold private models locally, referenced by pointers in a public "training plan" shape. --- ## Namespacing and Storage (Normative) -- Private overlays are actor‑anchored: `refs/gatos/private///` index metadata. The local workspace mirror is `gatos/private///`. -- Private blobs themselves are NOT stored under Git refs. They live in pluggable blob stores and are addressed by their `ciphertext_digest`/`digest`. +* Private overlays are actor‑anchored: `refs/gatos/private///` index metadata. The local workspace mirror is `gatos/private///`. +* Private blobs themselves are NOT stored under Git refs. They live in pluggable blob stores and are addressed by their `ciphertext_digest`/`digest`. ## Security & Privacy Notes (Normative) -- Capability references in pointers MUST NOT contain secrets or pre‑signed tokens. Use stable identifiers and resolve sensitive data via policy. -- AES‑256‑GCM (if used) MUST include AAD composed of: actor id, pointer `content_id`, and policy version; nonces MUST be 96‑bit, randomly generated, and never reused per key. -- Right‑to‑be‑forgotten: deleting private blobs breaks pointer resolution but does not remove the public pointer. Implement erasure as a tombstone event plus an audit record. +* Capability references in pointers MUST NOT contain secrets or pre‑signed tokens. Use stable identifiers and resolve sensitive data via policy. +* AES‑256‑GCM (if used) MUST include AAD composed of: actor id, pointer `content_id`, and policy version; nonces MUST be 96‑bit, randomly generated, and never reused per key. +* Right‑to‑be‑forgotten: deleting private blobs breaks pointer resolution but does not remove the public pointer. Implement erasure as a tombstone event plus an audit record. ### Algorithm variants (experimental; private attestations only) -- Implementations MAY use a keyed BLAKE3 variant for private attestation envelopes (not for public Opaque Pointers): `algo = "blake3-keyed"` with parameters encoded in an envelope or pointer `extensions` field. -- Recommended KDF: `hkdf-sha256`; context string `"gatos:ptr:priv:"`; derive `key = HKDF(policy_key, salt = actor_pubkey, info = context)`. -- Public pointers MUST continue to use `algo = "blake3"` for third‑party verifiability. +* Implementations MAY use a keyed BLAKE3 variant for private attestation envelopes (not for public Opaque Pointers): `algo = "blake3-keyed"` with parameters encoded in an envelope or pointer `extensions` field. +* Recommended KDF: `hkdf-sha256`; context string `"gatos:ptr:priv:"`; derive `key = HKDF(policy_key, salt = actor_pubkey, info = context)`. +* Public pointers MUST continue to use `algo = "blake3"` for third‑party verifiability. ## Error Taxonomy (Normative) Implementations SHOULD use a stable set of error codes with JSON problem details: -- `Unauthorized` (401) -- `Forbidden` (403) -- `NotFound` (404) -- `DigestMismatch` (422) -- `CapabilityUnavailable` (503) -- `PolicyDenied` (403) +* `Unauthorized` (401) +* `Forbidden` (403) +* `NotFound` (404) +* `DigestMismatch` (422) +* `CapabilityUnavailable` (503) +* `PolicyDenied` (403) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 6e36f09e..e960fa70 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -44,7 +44,7 @@ classDiagram +object payload // canonical JSON (JCS) +map refs // OPTIONAL cross-refs } -``` +```text ### 2) Namespaces and Ordering @@ -75,7 +75,7 @@ Numeric discipline: JSON numbers can be cross‑language foot‑guns. Precision Each Shiplog commit MUST include headers in the commit message (any order), followed by a single line containing three dashes `---` and then a JSON trailer object: -``` +```text Event-Id: ulid: Content-Id: blake3:<64-hex> Topic: @@ -112,6 +112,7 @@ Invariant: envelope.ns MUST equal the commit header `Topic:` value and the per-t Append(`topic`, `envelope`): validate schema; compute `content_id = blake3(JCS(envelope))`; enforce monotone ULID per topic on this node; create commit with headers + trailer; CAS update `refs/gatos/shiplog//head`; return `(commit_oid, ulid, content_id)`. Errors (normative): + - 400 `InvalidEnvelope`; 409 `UlidOutOfOrder`; 409 `NotFastForward`; 422 `DigestMismatch`. ### 6) Query Semantics @@ -180,7 +181,6 @@ Clients SHOULD return a problem+json response with a stable `code` plus HTTP sta } ``` - Pros: clean integration surface; deterministic envelopes; replay + analytics; explicit privacy. Cons: additional refs to manage; potential duplication if mirroring ledger events. From 179f2baf602e6a9deb028868e1bda69cb6fceea3 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 21:13:33 -0800 Subject: [PATCH 47/68] tests(neg): add topic vs envelope.ns mismatch check for ADR-0005 worktree; include checker script and fixture; wire into Makefile --- examples/v1/shiplog/event_mismatch_ns.json | 7 +++++ scripts/shiplog/check_topic_ns.js | 30 ++++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100644 examples/v1/shiplog/event_mismatch_ns.json create mode 100755 scripts/shiplog/check_topic_ns.js diff --git a/examples/v1/shiplog/event_mismatch_ns.json b/examples/v1/shiplog/event_mismatch_ns.json new file mode 100644 index 00000000..e742dc35 --- /dev/null +++ b/examples/v1/shiplog/event_mismatch_ns.json @@ -0,0 +1,7 @@ +{ + "ulid": "01HF4Y9Q1SM8Q7K9DK2R3V4AWB", + "ns": "wrongspace", + "type": "proposal.created", + "payload": { "title": "Q4 Budget", "amount": 100000 } +} + diff --git a/scripts/shiplog/check_topic_ns.js b/scripts/shiplog/check_topic_ns.js new file mode 100755 index 00000000..6523d210 --- /dev/null +++ b/scripts/shiplog/check_topic_ns.js @@ -0,0 +1,30 @@ +#!/usr/bin/env node +// Simple invariant check: envelope.ns MUST equal the CLI/topic segment +// Usage: check_topic_ns.js +const fs = require('fs'); + +function usage() { + console.error('usage: check_topic_ns.js '); + process.exit(2); +} + +const [, , file, topic] = process.argv; +if (!file || !topic) usage(); + +let ns; +try { + const text = fs.readFileSync(file, 'utf8'); + const obj = JSON.parse(text); + ns = obj.ns; +} catch (e) { + console.error('failed to read/parse envelope:', e.message); + process.exit(2); +} + +if (ns !== topic) { + console.error(`topic/ns mismatch: topic=${topic} ns=${ns}`); + process.exit(1); +} +// match → success (no output) +process.exit(0); + From 9da82efe036334e3f70b416803f4c5e6290fdb9b Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:16:41 -0800 Subject: [PATCH 48/68] ADR-0004: Pin AEAD to XChaCha20-Poly1305; clarify AAD and nonce discipline; demote AES-256-GCM to non-normative interop profile --- docs/decisions/ADR-0004/DECISION.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index ece63bc5..6a1c3b28 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -82,10 +82,10 @@ Private data overlays are fundamentally tied to an actor's identity, not an ephe ### 2. Encryption Algorithm & Nonce Discipline (Normative) -* AEAD algorithm: Implementations MUST use XChaCha20-Poly1305 for encrypting private blobs referenced by Opaque Pointers. -* Nonces: 24-byte (192-bit) nonces MUST be unique per key. Implementations SHOULD use deterministic nonces derived from the pointer digest via HKDF (domain-separated), or a crash-safe, monotonic per-key counter stored in KMS. Random nonces are permitted only with a documented collision budget and active monitoring. +* AEAD algorithm: Implementations MUST use XChaCha20-Poly1305 for encrypting private blobs referenced by Opaque Pointers. This is the only normative algorithm for the privacy overlay. +* Nonces: 24-byte (192-bit) nonces MUST be unique per key. Implementations SHOULD use deterministic, domain‑separated nonces derived via HKDF from the pointer digest and context, or a crash‑safe, monotonic per‑key counter stored in KMS. Random nonces are permitted only with a documented collision budget and active monitoring. * Catastrophic reuse: Nonce reuse under the same key is catastrophic and MUST be proven impossible by construction (deterministic derivation) or by counter invariants. -* AAD binding: AEAD AAD MUST bind the pointer digest, the requester actor id, and the effective policy version so that verifiers can validate context and detect misuse. +* AAD binding: AEAD AAD MUST bind the pointer digest (not a separate content_id), the requester actor id, and the effective policy version so that verifiers can validate context and detect misuse. * **Public Refs:** The corresponding public projection lives in the main state namespace. ```text @@ -277,7 +277,7 @@ This provides a simple, top-level indicator that a projection has occurred, prom ## Security & Privacy Notes (Normative) * Capability references in pointers MUST NOT contain secrets or pre‑signed tokens. Use stable identifiers and resolve sensitive data via policy. -* AES‑256‑GCM (if used) MUST include AAD composed of: actor id, pointer `content_id`, and policy version; nonces MUST be 96‑bit, randomly generated, and never reused per key. +> Non‑normative interop profile: Some legacy deployments may use AES‑256‑GCM. If and only if such interop is required, deployments MAY support AES‑256‑GCM with AAD composed of (actor id, pointer digest, policy version) and 96‑bit nonces that are never reused per key. This profile is deprecated and MUST NOT be the default. * Right‑to‑be‑forgotten: deleting private blobs breaks pointer resolution but does not remove the public pointer. Implement erasure as a tombstone event plus an audit record. ### Algorithm variants (experimental; private attestations only) From a418cb1e1dc380216aa33a3904710579392371d8 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:18:12 -0800 Subject: [PATCH 49/68] ADR-0004: Explicitly supersede AES-256-GCM; XChaCha20-Poly1305 is the sole normative AEAD --- docs/decisions/ADR-0004/DECISION.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index 6a1c3b28..6f28bb4d 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -82,7 +82,7 @@ Private data overlays are fundamentally tied to an actor's identity, not an ephe ### 2. Encryption Algorithm & Nonce Discipline (Normative) -* AEAD algorithm: Implementations MUST use XChaCha20-Poly1305 for encrypting private blobs referenced by Opaque Pointers. This is the only normative algorithm for the privacy overlay. +* AEAD algorithm: Implementations MUST use XChaCha20-Poly1305 for encrypting private blobs referenced by Opaque Pointers. This is the only normative algorithm for the privacy overlay. It supersedes any prior mentions of AES‑256‑GCM as a default. * Nonces: 24-byte (192-bit) nonces MUST be unique per key. Implementations SHOULD use deterministic, domain‑separated nonces derived via HKDF from the pointer digest and context, or a crash‑safe, monotonic per‑key counter stored in KMS. Random nonces are permitted only with a documented collision budget and active monitoring. * Catastrophic reuse: Nonce reuse under the same key is catastrophic and MUST be proven impossible by construction (deterministic derivation) or by counter invariants. * AAD binding: AEAD AAD MUST bind the pointer digest (not a separate content_id), the requester actor id, and the effective policy version so that verifiers can validate context and detect misuse. From ba91632c25e0b367d643e0ecd04b84a32733dc5b Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:19:06 -0800 Subject: [PATCH 50/68] ADR-0005: Standardize on ns (namespace) over topic; fix refs, invariants, CLI flags, and lowercase digest examples; normalize schema paths --- docs/decisions/ADR-0005/DECISION.md | 42 ++++++++++++++--------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index e960fa70..5cdebc3c 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -7,11 +7,11 @@ Requires: [ADR-0001, ADR-0004] Related: [ADR-0002, ADR-0003] Tags: [Shiplog, Event Stream, Consumers, JCS, ULID] Schemas: - - ../../../../schemas/v1/shiplog/event_envelope.schema.json - - ../../../../schemas/v1/shiplog/consumer_checkpoint.schema.json - - ../../../../schemas/v1/shiplog/deployment_trailer.schema.json - - ../../../../schemas/v1/shiplog/anchor.schema.json - - ../../../../schemas/v1/privacy/opaque_pointer.schema.json + - ../../../schemas/v1/shiplog/event_envelope.schema.json + - ../../../schemas/v1/shiplog/consumer_checkpoint.schema.json + - ../../../schemas/v1/shiplog/deployment_trailer.schema.json + - ../../../schemas/v1/shiplog/anchor.schema.json + - ../../../schemas/v1/privacy/opaque_pointer.schema.json Supersedes: [] Superseded-By: [] --- @@ -48,9 +48,9 @@ classDiagram ### 2) Namespaces and Ordering -- Per‑topic head ref (append‑only, linear): `refs/gatos/shiplog//head` -- Topic naming: `^[a-z][a-z0-9._-]{0,63}$` (ASCII, lowercase start). -- Ordering per topic is the Git parent chain. Appends MUST be fast‑forward (CAS on ref update). On a single node, ULIDs MUST increase strictly per topic. +- Per‑namespace head ref (append‑only, linear): `refs/gatos/shiplog//head` +- Namespace naming: `^[a-z][a-z0-9._-]{0,63}$` (ASCII, lowercase start). +- Ordering per namespace is the Git parent chain. Appends MUST be fast‑forward (CAS on ref update). On a single node, ULIDs MUST increase strictly per namespace. ```mermaid graph TD @@ -78,7 +78,7 @@ Each Shiplog commit MUST include headers in the commit message (any order), foll ```text Event-Id: ulid: Content-Id: blake3:<64-hex> -Topic: +Namespace: Schema: https://gatos.dev/schemas/v1/shiplog/event_envelope.schema.json --- { "version": 1, @@ -100,16 +100,16 @@ Schema: https://gatos.dev/schemas/v1/shiplog/event_envelope.schema.json Trailer schema: `schemas/v1/shiplog/deployment_trailer.schema.json`. -MUST: validate the trailer against this schema, and write the exact JCS bytes hashed for the envelope to `/gatos/shiplog//.json` (parse → JCS → hash → write → commit). +MUST: validate the trailer against this schema, and write the exact JCS bytes hashed for the envelope to `/gatos/shiplog//.json` (parse → JCS → hash → write → commit). > [!IMPORTANT] > Hashing Law — parse → JCS → hash → write → commit. The bytes you hash MUST be the exact JCS bytes you write and commit. ### 5) Append Semantics -Invariant: envelope.ns MUST equal the commit header `Topic:` value and the per-topic ref segment. +Invariant: envelope.ns MUST equal the commit header `Namespace:` value and the per‑namespace ref segment. -Append(`topic`, `envelope`): validate schema; compute `content_id = blake3(JCS(envelope))`; enforce monotone ULID per topic on this node; create commit with headers + trailer; CAS update `refs/gatos/shiplog//head`; return `(commit_oid, ulid, content_id)`. +Append(`ns`, `envelope`): validate schema; compute `content_id = blake3(JCS(envelope))`; enforce monotone ULID per namespace on this node; create commit with headers + trailer; CAS update `refs/gatos/shiplog//head`; return `(commit_oid, ulid, content_id)`. Errors (normative): @@ -117,12 +117,12 @@ Errors (normative): ### 6) Query Semantics -- `shiplog.read(topic, since_ulid, limit) -> [ (ulid, content_id, commit_oid, envelope) ]` (increasing ULID order). -- `shiplog.tail(topics[], limit_per_topic)` MAY multiplex without cross‑topic causality guarantees. +- `shiplog.read(ns, since_ulid, limit) -> [ (ulid, content_id, commit_oid, envelope) ]` (increasing ULID order). +- `shiplog.tail(namespaces[], limit_per_ns)` MAY multiplex without cross‑namespace causality guarantees. ### 7) Consumer Checkpoints -- `refs/gatos/consumers//` points to the last processed Shiplog commit OID. Portable JSON (optional): `schemas/v1/shiplog/consumer_checkpoint.schema.json`. The `commit_oid` value MUST be lowercase hex. +- `refs/gatos/consumers//` points to the last processed Shiplog commit OID. Portable JSON (optional): `schemas/v1/shiplog/consumer_checkpoint.schema.json`. The `commit_oid` value MUST be lowercase hex. ### 8) Privacy Interactions (ADR‑0004) @@ -132,7 +132,7 @@ AEAD algorithm is pinned by ADR‑0004 to XChaCha20‑Poly1305. Nonces MUST be u ### 9) Governance and Ledger Interactions -- Governance (ADR‑0003): Should emit Shiplog events under `topic="governance"`; envelopes carry `ns="governance"` and the commit header sets `Topic: governance`. +- Governance (ADR‑0003): Should emit Shiplog events under `ns="governance"`; envelopes carry `ns="governance"` and the commit header sets `Namespace: governance`. - Ledger mirroring: MAY mirror ledger events; must preserve envelope determinism. ### 10) Security Considerations @@ -143,14 +143,14 @@ AEAD algorithm is pinned by ADR‑0004 to XChaCha20‑Poly1305. Nonces MUST be u ### 11) CLI Examples ```bash -$ gatosd shiplog append --topic governance --file event.json +$ gatosd shiplog append --ns governance --file event.json ok commit=8b1c1e4 content_id=blake3:2a6c… ulid=01HF4Y9Q1SM8Q7K9DK2R3V4AWB -$ gatosd shiplog read --topic governance --since 01HF4Y9Q1SM8Q7K9DK2R3V4AWB --limit 2 -01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2A6C… 8b1c1e4 {"ulid":"01HF4Y9Q1SM8Q7K9DK2R4V5CXD","ns":"governance","type":"proposal.created","payload":{}} -01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:C1D2… 9f0aa21 {"ulid":"01HF4Y9Q1SM8Q7K9DK2R4V5CXE","ns":"governance","type":"proposal.approved","payload":{}} +$ gatosd shiplog read --ns governance --since 01HF4Y9Q1SM8Q7K9DK2R3V4AWB --limit 2 +01HF4Y9Q1SM8Q7K9DK2R4V5CXD blake3:2a6c… 8b1c1e4 {"ulid":"01HF4Y9Q1SM8Q7K9DK2R4V5CXD","ns":"governance","type":"proposal.created","payload":{}} +01HF4Y9Q1SM8Q7K9DK2R4V5CXE blake3:c1d2… 9f0aa21 {"ulid":"01HF4Y9Q1SM8Q7K9DK2R4V5CXE","ns":"governance","type":"proposal.approved","payload":{}} -$ gatosd shiplog checkpoint set --group analytics --topic governance --commit 8b1c1e4 +$ gatosd shiplog checkpoint set --group analytics --ns governance --commit 8b1c1e4 ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 ``` From 9c08cac75ba5acf306f22b95bca3444c39c2e784 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:20:23 -0800 Subject: [PATCH 51/68] ADR-0005: Fix mismatched fenced code block closers (remove ); ensure mermaid/text fences render and lint clean --- docs/decisions/ADR-0005/DECISION.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 5cdebc3c..4f30a892 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -44,7 +44,7 @@ classDiagram +object payload // canonical JSON (JCS) +map refs // OPTIONAL cross-refs } -```text +``` ### 2) Namespaces and Ordering @@ -75,7 +75,7 @@ Numeric discipline: JSON numbers can be cross‑language foot‑guns. Precision Each Shiplog commit MUST include headers in the commit message (any order), followed by a single line containing three dashes `---` and then a JSON trailer object: -```text +``` Event-Id: ulid: Content-Id: blake3:<64-hex> Namespace: From ebe910905b901ce1399fc5d9aeedca94cb1e2c87 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:23:57 -0800 Subject: [PATCH 52/68] docs: mark hex-case and schema-path fixes as resolved in external review checklist (ba91632) --- Makefile | 13 ++++++++++++- .../v1/privacy/pointer_low_entropy_invalid.json | 8 ++++++++ examples/v1/privacy/pointer_low_entropy_min.json | 8 ++++++++ .../v1/shiplog/checkpoint_commit_only_invalid.json | 3 +++ .../v1/shiplog/checkpoint_ulid_only_invalid.json | 3 +++ schemas/v1/shiplog/consumer_checkpoint.schema.json | 5 +---- 6 files changed, 35 insertions(+), 5 deletions(-) create mode 100644 examples/v1/privacy/pointer_low_entropy_invalid.json create mode 100644 examples/v1/privacy/pointer_low_entropy_min.json create mode 100644 examples/v1/shiplog/checkpoint_commit_only_invalid.json create mode 100644 examples/v1/shiplog/checkpoint_ulid_only_invalid.json diff --git a/Makefile b/Makefile index 8c54737c..973b31ff 100644 --- a/Makefile +++ b/Makefile @@ -67,7 +67,18 @@ schema-validate: npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/event_envelope.schema.json -d examples/v1/shiplog/event_min.json -r schemas/v1/common/ids.schema.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json -d examples/v1/shiplog/checkpoint_min.json && \ npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/deployment_trailer.schema.json -d examples/v1/shiplog/trailer_min.json && \ - npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/anchor.schema.json -d examples/v1/shiplog/anchor_min.json' + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/anchor.schema.json -d examples/v1/shiplog/anchor_min.json && \ + npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -d examples/v1/privacy/pointer_low_entropy_min.json' + +schema-negative: + @bash -lc 'set -euo pipefail; \ + if ! command -v node >/dev/null 2>&1; then \ + echo "Node.js required (or run in CI)" >&2; exit 1; fi; \ + # Negative: checkpoint requires both fields + ! npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json -d examples/v1/shiplog/checkpoint_ulid_only_invalid.json; \ + ! npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/shiplog/consumer_checkpoint.schema.json -d examples/v1/shiplog/checkpoint_commit_only_invalid.json; \ + # Negative: low-entropy pointer must not allow plaintext digest + ! npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -d examples/v1/privacy/pointer_low_entropy_invalid.json' schema-negative: @bash -lc 'set -euo pipefail; \ diff --git a/examples/v1/privacy/pointer_low_entropy_invalid.json b/examples/v1/privacy/pointer_low_entropy_invalid.json new file mode 100644 index 00000000..21733fc6 --- /dev/null +++ b/examples/v1/privacy/pointer_low_entropy_invalid.json @@ -0,0 +1,8 @@ +{ + "kind": "opaque_pointer", + "algo": "blake3", + "digest": "blake3:0000000000000000000000000000000000000000000000000000000000000000", + "location": "gatos://private/blob/abc", + "capability": "gatos://cap/abc", + "extensions": { "class": "low-entropy" } +} diff --git a/examples/v1/privacy/pointer_low_entropy_min.json b/examples/v1/privacy/pointer_low_entropy_min.json new file mode 100644 index 00000000..d1dfd865 --- /dev/null +++ b/examples/v1/privacy/pointer_low_entropy_min.json @@ -0,0 +1,8 @@ +{ + "kind": "opaque_pointer", + "algo": "blake3", + "ciphertext_digest": "blake3:0000000000000000000000000000000000000000000000000000000000000000", + "location": "gatos://private/blob/abc", + "capability": "gatos://cap/abc", + "extensions": { "class": "low-entropy" } +} diff --git a/examples/v1/shiplog/checkpoint_commit_only_invalid.json b/examples/v1/shiplog/checkpoint_commit_only_invalid.json new file mode 100644 index 00000000..fce69d9b --- /dev/null +++ b/examples/v1/shiplog/checkpoint_commit_only_invalid.json @@ -0,0 +1,3 @@ +{ + "commit_oid": "8b1c1e4f3c9a0b5d7e2c1a4f6b8d0c3e5f7a9b1c" +} diff --git a/examples/v1/shiplog/checkpoint_ulid_only_invalid.json b/examples/v1/shiplog/checkpoint_ulid_only_invalid.json new file mode 100644 index 00000000..fad17359 --- /dev/null +++ b/examples/v1/shiplog/checkpoint_ulid_only_invalid.json @@ -0,0 +1,3 @@ +{ + "ulid": "01HF4Y9Q1SM8Q7K9DK2R3V4AWB" +} diff --git a/schemas/v1/shiplog/consumer_checkpoint.schema.json b/schemas/v1/shiplog/consumer_checkpoint.schema.json index a017c51d..1d6847e0 100644 --- a/schemas/v1/shiplog/consumer_checkpoint.schema.json +++ b/schemas/v1/shiplog/consumer_checkpoint.schema.json @@ -14,8 +14,5 @@ "pattern": "^[0-9a-f]{40}$" } }, - "anyOf": [ - { "type": "object", "properties": { "ulid": {} }, "required": ["ulid"] }, - { "type": "object", "properties": { "commit_oid": {} }, "required": ["commit_oid"] } - ] + "required": ["ulid", "commit_oid"] } From 4ede46d7fe3ede2e59293fc2882f16d953f1e00a Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:24:23 -0800 Subject: [PATCH 53/68] =?UTF-8?q?ADR-0005:=20Clarify=20trailer=20structure?= =?UTF-8?q?=20=E2=80=94=20repo=5Fhead=20is=20top-level=20only=20(no=20dupl?= =?UTF-8?q?ication=20under=20what)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/decisions/ADR-0005/DECISION.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 4f30a892..82e53583 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -102,6 +102,8 @@ Trailer schema: `schemas/v1/shiplog/deployment_trailer.schema.json`. MUST: validate the trailer against this schema, and write the exact JCS bytes hashed for the envelope to `/gatos/shiplog//.json` (parse → JCS → hash → write → commit). +Note: The trailer places `repo_head` as a top‑level field only. It MUST NOT appear inside nested objects such as `what`. + > [!IMPORTANT] > Hashing Law — parse → JCS → hash → write → commit. The bytes you hash MUST be the exact JCS bytes you write and commit. From f3ee31a3effd434277d57b40c92c1aa43e775018 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:24:47 -0800 Subject: [PATCH 54/68] ADR-0005: Define per-namespace monotonic ULID algorithm and tie-break rules (TemporalOrder/AppendRejected) --- docs/decisions/ADR-0005/DECISION.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 82e53583..0f6f286c 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -113,6 +113,8 @@ Invariant: envelope.ns MUST equal the commit header `Namespace:` value and the p Append(`ns`, `envelope`): validate schema; compute `content_id = blake3(JCS(envelope))`; enforce monotone ULID per namespace on this node; create commit with headers + trailer; CAS update `refs/gatos/shiplog//head`; return `(commit_oid, ulid, content_id)`. +ULID generation (normative): Implementations MUST use a monotonic ULID algorithm scoped per namespace. If the system clock moves backwards, the implementation MUST keep the last emitted millisecond timestamp and monotonically increase the randomness field; on overflow of the randomness field, the operation MUST fail with `TemporalOrder`. If two appends observe the same timestamp, the second MUST increase the randomness field compared to the previous append or fail with `AppendRejected` on CAS. Replayed appends MUST preserve the original ULID for that envelope; otherwise reject with `DigestMismatch`. + Errors (normative): - 400 `InvalidEnvelope`; 409 `UlidOutOfOrder`; 409 `NotFastForward`; 422 `DigestMismatch`. From 90fd86843f1d6bd3fede098848d8cf5365b13938 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:24:57 -0800 Subject: [PATCH 55/68] =?UTF-8?q?ADR-0005:=20Disambiguate=20commit=20heade?= =?UTF-8?q?rs=20=E2=80=94=20Envelope-Schema=20and=20Trailer-Schema=20field?= =?UTF-8?q?s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/decisions/ADR-0005/DECISION.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 0f6f286c..05646b82 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -79,7 +79,8 @@ Each Shiplog commit MUST include headers in the commit message (any order), foll Event-Id: ulid: Content-Id: blake3:<64-hex> Namespace: -Schema: https://gatos.dev/schemas/v1/shiplog/event_envelope.schema.json +Envelope-Schema: https://gatos.dev/schemas/v1/shiplog/event_envelope.schema.json +Trailer-Schema: https://gatos.dev/schemas/v1/shiplog/deployment_trailer.schema.json --- { "version": 1, "env": "prod", From 752eee296ccc540d7ae0f4de3a579504eecbf1d2 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:25:07 -0800 Subject: [PATCH 56/68] =?UTF-8?q?ADR-0005:=20Clarify=20Hashing=20Law=20pat?= =?UTF-8?q?h=20=E2=80=94=20path=20is=20a=20blob=20inside=20the=20commit=20?= =?UTF-8?q?tree,=20not=20a=20working=20directory=20file?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/decisions/ADR-0005/DECISION.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 05646b82..2a4bb735 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -101,7 +101,7 @@ Trailer-Schema: https://gatos.dev/schemas/v1/shiplog/deployment_trailer.schema. Trailer schema: `schemas/v1/shiplog/deployment_trailer.schema.json`. -MUST: validate the trailer against this schema, and write the exact JCS bytes hashed for the envelope to `/gatos/shiplog//.json` (parse → JCS → hash → write → commit). +MUST: validate the trailer against this schema, and write the exact JCS bytes hashed for the envelope to `/gatos/shiplog//.json` (parse → JCS → hash → write → commit). The path is a logical path inside the Git commit tree: `/gatos/shiplog/...` is a blob stored in the tree referenced by the Shiplog commit, not a working‑directory file. Note: The trailer places `repo_head` as a top‑level field only. It MUST NOT appear inside nested objects such as `what`. From dfa0a6e96da37a6c6338905c8ddf5a56cb46cac0 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:25:23 -0800 Subject: [PATCH 57/68] =?UTF-8?q?ADR-0005:=20Remove=20duplicate=20numeric-?= =?UTF-8?q?discipline=20text;=20reference=20=C2=A71=20once?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/decisions/ADR-0005/DECISION.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 2a4bb735..6fb9e1c5 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -69,7 +69,7 @@ graph TD - Optional `refs` (map) to link related state or IDs. - Privacy (ADR‑0004): Payload MUST NOT embed private overlay data. Redacted values MUST be replaced by `OpaquePointer` envelopes per `schemas/v1/privacy/opaque_pointer.schema.json`. -Numeric discipline: JSON numbers can be cross‑language foot‑guns. Precision‑sensitive values (e.g., monetary/time) MUST be encoded as integers or strings. +Numeric discipline: See §1 Canonicalization & Identifiers — precision‑sensitive values (e.g., monetary/time) MUST be encoded as integers or strings. ### 4) Commit Message and Trailer From 13d284da3cc15b2101fce50ff2d8c2d105de1538 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:25:58 -0800 Subject: [PATCH 58/68] ADR-0005: Unify error taxonomy names (TemporalOrder, AppendRejected); align with table --- docs/decisions/ADR-0005/DECISION.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 6fb9e1c5..489bb0a8 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -118,7 +118,7 @@ ULID generation (normative): Implementations MUST use a monotonic ULID algorithm Errors (normative): -- 400 `InvalidEnvelope`; 409 `UlidOutOfOrder`; 409 `NotFastForward`; 422 `DigestMismatch`. +- 400 `InvalidEnvelope`; 409 `TemporalOrder`; 409 `AppendRejected`; 422 `DigestMismatch`. ### 6) Query Semantics From b92e9ea01ea2aba9793b0eeeede669d5b3c4d888 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:26:08 -0800 Subject: [PATCH 59/68] ADR-0005: Specify tail() fairness (per-namespace round-robin) and watermark-based resume semantics --- docs/decisions/ADR-0005/DECISION.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 489bb0a8..97f6ef24 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -125,6 +125,8 @@ Errors (normative): - `shiplog.read(ns, since_ulid, limit) -> [ (ulid, content_id, commit_oid, envelope) ]` (increasing ULID order). - `shiplog.tail(namespaces[], limit_per_ns)` MAY multiplex without cross‑namespace causality guarantees. +Tail fairness (normative): When multiplexing multiple namespaces, implementations MUST use fair scheduling (e.g., per‑namespace round‑robin) so that no namespace is starved under sustained load. Implementations SHOULD emit a per‑namespace watermark (last ULID included) to help consumers resume without duplication. Consumers restore by resuming from each namespace’s last watermark or their checkpoint, whichever is newer; duplicates MUST be tolerated by idempotent processing keyed by `(ns, ulid)`. + ### 7) Consumer Checkpoints - `refs/gatos/consumers//` points to the last processed Shiplog commit OID. Portable JSON (optional): `schemas/v1/shiplog/consumer_checkpoint.schema.json`. The `commit_oid` value MUST be lowercase hex. From e4556dbab28398b17d553dc2d236ce0549584397 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:26:18 -0800 Subject: [PATCH 60/68] =?UTF-8?q?ADR-0005:=20Document=20anchor=20objects?= =?UTF-8?q?=20=E2=80=94=20when=20to=20write,=20consumer=20usage,=20and=20s?= =?UTF-8?q?igning=20guidance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/decisions/ADR-0005/DECISION.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 97f6ef24..1b73a290 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -161,6 +161,10 @@ $ gatosd shiplog checkpoint set --group analytics --ns governance --commit 8b1c1 ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 ``` +### 12) Anchors + +Anchors are signed, portable snapshots of a Shiplog namespace head. An anchor document conforms to `schemas/v1/shiplog/anchor.schema.json` and records `(ulid, topic/ns, head)` plus optional metadata. Producers MAY write an anchor when rolling out a deployment, completing a batch, or before compaction. Consumers use anchors as stable restore points and for cross‑repo attestation. If signatures are used, they MUST bind the anchor JSON bytes (JCS) and the Git commit oid referenced by `head`. + ## Error Taxonomy (Normative) | Code | HTTP | Meaning | From 76483ff1afd7e26ebe2557aaf42085458756a881 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:26:53 -0800 Subject: [PATCH 61/68] =?UTF-8?q?ADR-0004:=20Standardize=20fenced=20code?= =?UTF-8?q?=20blocks=20=E2=80=94=20replace=20=20for=20proper=20rendering?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/decisions/ADR-0004/DECISION.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index 6f28bb4d..3750079c 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -63,7 +63,7 @@ This model is a direct application of the GATOS Morphology Calculus. style P1 fill:#cde,stroke:#333 style P2 fill:#cde,stroke:#333 - ```text + ``` This ensures that the transformation is structure-preserving and that the public history remains a valid, deterministic projection of the complete history. @@ -76,9 +76,9 @@ Private data overlays are fundamentally tied to an actor's identity, not an ephe * **Actor ID:** The canonical identifier for an actor, e.g., `ed25519:`. * **Private Refs:** Private data is stored under refs namespaced by the actor ID. - ```text + ``` refs/gatos/private/// - ```text + ``` ### 2. Encryption Algorithm & Nonce Discipline (Normative) @@ -88,7 +88,7 @@ Private data overlays are fundamentally tied to an actor's identity, not an ephe * AAD binding: AEAD AAD MUST bind the pointer digest (not a separate content_id), the requester actor id, and the effective policy version so that verifiers can validate context and detect misuse. * **Public Refs:** The corresponding public projection lives in the main state namespace. - ```text + ``` refs/gatos/state/public// ``` @@ -108,7 +108,7 @@ classDiagram +string capability // MUST NOT embed secrets +object extensions // forward-compatible } -```text +``` * **`digest`**: The content-address of the private plaintext (`blake3(plaintext_bytes)`). This is the immutable link between the public and private worlds. * **`ciphertext_digest`**: The content-address of the stored ciphertext (`blake3(ciphertext_bytes)`). For low‑entropy privacy classes (see Policy Hooks), the public pointer **MUST** include `ciphertext_digest` and policy **MUST NOT** expose the plaintext digest publicly. @@ -152,7 +152,7 @@ sequenceDiagram E->>E: 4. Apply rules to create PublicState + PrivateBlobs E->>L: 5. Commit PublicState to public refs E->>PS: 6. Store PrivateBlobs by digest -```text +``` ### 4. Pointer Resolution Protocol (Normative) @@ -180,11 +180,11 @@ A client resolving an Opaque Pointer **MUST** follow this protocol: Response headers on success: -```text +``` Content-Type: application/octet-stream X-BLAKE3-Digest: blake3: Digest: sha-256= -```text +``` Optional HTTP Message Signatures profile (RFC 9421): @@ -209,7 +209,7 @@ sequenceDiagram else Unauthorized PN-->>C: 4. Return 401/403 end -```text +``` ### 5. Policy Hooks (Normative) @@ -230,7 +230,7 @@ privacy: location: "gatos-node://ed25519:" - select: "path.to.transient.data" action: "redact" -```text +``` The `select` syntax will use a simple path-matching language (e.g., glob patterns) defined by the policy engine. From 9d111324d7b57da71dba11a0db99d9b1f710459f Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:42:59 -0800 Subject: [PATCH 62/68] kill-check: add scripts for schema headers, ULID external reference, and error casing; wire into Makefile --- Makefile | 6 ++++++ scripts/killcheck/error_casing.sh | 11 +++++++++++ scripts/killcheck/schema_headers.sh | 10 ++++++++++ scripts/killcheck/ulid_reference.sh | 9 +++++++++ 4 files changed, 36 insertions(+) create mode 100755 scripts/killcheck/error_casing.sh create mode 100755 scripts/killcheck/schema_headers.sh create mode 100755 scripts/killcheck/ulid_reference.sh diff --git a/Makefile b/Makefile index 973b31ff..68acc3f3 100644 --- a/Makefile +++ b/Makefile @@ -80,6 +80,12 @@ schema-negative: # Negative: low-entropy pointer must not allow plaintext digest ! npx -y ajv-cli@5 validate --spec=draft2020 --strict=true -c ajv-formats -s schemas/v1/privacy/opaque_pointer.schema.json -d examples/v1/privacy/pointer_low_entropy_invalid.json' +.PHONY: kill-check +kill-check: + @bash -lc 'scripts/killcheck/schema_headers.sh' + @bash -lc 'scripts/killcheck/ulid_reference.sh' + @bash -lc 'scripts/killcheck/error_casing.sh' + schema-negative: @bash -lc 'set -euo pipefail; \ if ! command -v node >/dev/null 2>&1; then \ diff --git a/scripts/killcheck/error_casing.sh b/scripts/killcheck/error_casing.sh new file mode 100755 index 00000000..d0b497a3 --- /dev/null +++ b/scripts/killcheck/error_casing.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Disallow common lowercase/underscore variants of error codes anywhere in docs +bad=$(rg -n "append_rejected|not_fast_forward|temporalorder|siginvalid|policydenied|notfound" docs || true) +if [[ -n "$bad" ]]; then + echo "Found non-canonical error code casing or names:" >&2 + echo "$bad" >&2 + exit 1 +fi +echo "ok: error code casing canonical" diff --git a/scripts/killcheck/schema_headers.sh b/scripts/killcheck/schema_headers.sh new file mode 100755 index 00000000..6e7640c4 --- /dev/null +++ b/scripts/killcheck/schema_headers.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Fail if any line contains 'Schema:' that is not 'Envelope-Schema:' or 'Trailer-Schema:' +if rg -n "\\bSchema:\\b" docs | rg -v "Envelope-Schema|Trailer-Schema" -n | sed -n '1,200p'; then + echo "Found legacy 'Schema:' header(s). Use Envelope-Schema and Trailer-Schema only." >&2 + exit 1 +fi +echo "ok: no legacy 'Schema:' headers" + diff --git a/scripts/killcheck/ulid_reference.sh b/scripts/killcheck/ulid_reference.sh new file mode 100755 index 00000000..3bc1e762 --- /dev/null +++ b/scripts/killcheck/ulid_reference.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +set -euo pipefail +file="docs/decisions/ADR-0005/DECISION.md" +if ! rg -n "ULID Spec §4\.1 Monotonic Lexicographic Ordering" "$file" >/dev/null; then + echo "Missing external reference to 'ULID Spec §4.1 Monotonic Lexicographic Ordering' in ADR-0005." >&2 + exit 1 +fi +echo "ok: ULID external reference present" + From 2a49f190329c8f56c6c7e6426ab8b53415429750 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:43:11 -0800 Subject: [PATCH 63/68] =?UTF-8?q?ADR-0005:=20Add=20external=20reference=20?= =?UTF-8?q?to=20ULID=20Spec=20=C2=A74.1=20for=20monotonic=20ordering?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/decisions/ADR-0005/DECISION.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 1b73a290..37bbba14 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -114,7 +114,7 @@ Invariant: envelope.ns MUST equal the commit header `Namespace:` value and the p Append(`ns`, `envelope`): validate schema; compute `content_id = blake3(JCS(envelope))`; enforce monotone ULID per namespace on this node; create commit with headers + trailer; CAS update `refs/gatos/shiplog//head`; return `(commit_oid, ulid, content_id)`. -ULID generation (normative): Implementations MUST use a monotonic ULID algorithm scoped per namespace. If the system clock moves backwards, the implementation MUST keep the last emitted millisecond timestamp and monotonically increase the randomness field; on overflow of the randomness field, the operation MUST fail with `TemporalOrder`. If two appends observe the same timestamp, the second MUST increase the randomness field compared to the previous append or fail with `AppendRejected` on CAS. Replayed appends MUST preserve the original ULID for that envelope; otherwise reject with `DigestMismatch`. +ULID generation (normative): Implementations MUST use a monotonic ULID algorithm scoped per namespace (see ULID Spec §4.1 Monotonic Lexicographic Ordering). If the system clock moves backwards, the implementation MUST keep the last emitted millisecond timestamp and monotonically increase the randomness field; on overflow of the randomness field, the operation MUST fail with `TemporalOrder`. If two appends observe the same timestamp, the second MUST increase the randomness field compared to the previous append or fail with `AppendRejected` on CAS. Replayed appends MUST preserve the original ULID for that envelope; otherwise reject with `DigestMismatch`. Errors (normative): From c5efd8052d4a76c7a70e529bf3579eb0a2cc1251 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:43:48 -0800 Subject: [PATCH 64/68] =?UTF-8?q?ADR-0005/ADR-0004:=20Polish=20pass=20?= =?UTF-8?q?=E2=80=94=20stable=20round-robin,=20stronger=20repo=5Fhead=20no?= =?UTF-8?q?n-duplication,=20Hashing=20Law=20verifier=20note,=20anchor=20si?= =?UTF-8?q?gning=20key=20guidance,=20and=20AAD=20Components=20example?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/decisions/ADR-0004/DECISION.md | 12 ++++++++++++ docs/decisions/ADR-0005/DECISION.md | 8 ++++---- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index 3750079c..1b14f13b 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -123,6 +123,18 @@ The canonical `content_id` of the pointer itself is `blake3(JCS(pointer_json))`, **Schema:** `schemas/v1/privacy/opaque_pointer.schema.json` +#### AAD Components (Example) + +When encrypting a private blob referenced by an Opaque Pointer, the AEAD AAD MUST bind all of the following, in order, as UTF‑8 bytes: + +```text +1) pointer.digest +2) requester.actor_id (e.g., user:alice | service:policy) +3) policy.version (e.g., v1.2.3) +``` + +Implementations MAY structure these as concatenated bytes with clear domain separation (e.g., length‑prefixing) prior to supplying them as AAD to the AEAD algorithm. + ### 3. The Projection Function (Normative) The State Engine (`gatos-echo`) is responsible for executing the projection. diff --git a/docs/decisions/ADR-0005/DECISION.md b/docs/decisions/ADR-0005/DECISION.md index 37bbba14..22593b6d 100644 --- a/docs/decisions/ADR-0005/DECISION.md +++ b/docs/decisions/ADR-0005/DECISION.md @@ -101,9 +101,9 @@ Trailer-Schema: https://gatos.dev/schemas/v1/shiplog/deployment_trailer.schema. Trailer schema: `schemas/v1/shiplog/deployment_trailer.schema.json`. -MUST: validate the trailer against this schema, and write the exact JCS bytes hashed for the envelope to `/gatos/shiplog//.json` (parse → JCS → hash → write → commit). The path is a logical path inside the Git commit tree: `/gatos/shiplog/...` is a blob stored in the tree referenced by the Shiplog commit, not a working‑directory file. +MUST: validate the trailer against this schema, and write the exact JCS bytes hashed for the envelope to `/gatos/shiplog//.json` (parse → JCS → hash → write → commit). The path is a logical path inside the Git commit tree: `/gatos/shiplog/...` is a blob stored in the tree referenced by the Shiplog commit, not a working‑directory file. Thus the object hash of the blob equals the canonical digest; verifiers MUST compare them byte‑for‑byte. -Note: The trailer places `repo_head` as a top‑level field only. It MUST NOT appear inside nested objects such as `what`. +Note: The trailer places `repo_head` as a top‑level field only. It MUST NOT appear inside nested objects such as `what` (or any nested object). Producers MUST ensure no duplication occurs. > [!IMPORTANT] > Hashing Law — parse → JCS → hash → write → commit. The bytes you hash MUST be the exact JCS bytes you write and commit. @@ -125,7 +125,7 @@ Errors (normative): - `shiplog.read(ns, since_ulid, limit) -> [ (ulid, content_id, commit_oid, envelope) ]` (increasing ULID order). - `shiplog.tail(namespaces[], limit_per_ns)` MAY multiplex without cross‑namespace causality guarantees. -Tail fairness (normative): When multiplexing multiple namespaces, implementations MUST use fair scheduling (e.g., per‑namespace round‑robin) so that no namespace is starved under sustained load. Implementations SHOULD emit a per‑namespace watermark (last ULID included) to help consumers resume without duplication. Consumers restore by resuming from each namespace’s last watermark or their checkpoint, whichever is newer; duplicates MUST be tolerated by idempotent processing keyed by `(ns, ulid)`. +Tail fairness (normative): When multiplexing multiple namespaces, implementations MUST use fair scheduling (e.g., per‑namespace round‑robin) so that no namespace is starved under sustained load. The round‑robin order SHOULD be stable across restarts. Implementations SHOULD emit a per‑namespace watermark (last ULID included) to help consumers resume without duplication. Consumers restore by resuming from each namespace’s last watermark or their checkpoint, whichever is newer; duplicates MUST be tolerated by idempotent processing keyed by `(ns, ulid)`. ### 7) Consumer Checkpoints @@ -163,7 +163,7 @@ ok refs/gatos/consumers/analytics/governance -> 8b1c1e4 ### 12) Anchors -Anchors are signed, portable snapshots of a Shiplog namespace head. An anchor document conforms to `schemas/v1/shiplog/anchor.schema.json` and records `(ulid, topic/ns, head)` plus optional metadata. Producers MAY write an anchor when rolling out a deployment, completing a batch, or before compaction. Consumers use anchors as stable restore points and for cross‑repo attestation. If signatures are used, they MUST bind the anchor JSON bytes (JCS) and the Git commit oid referenced by `head`. +Anchors are signed, portable snapshots of a Shiplog namespace head. An anchor document conforms to `schemas/v1/shiplog/anchor.schema.json` and records `(ulid, topic/ns, head)` plus optional metadata. Producers MAY write an anchor when rolling out a deployment, completing a batch, or before compaction. Consumers use anchors as stable restore points and for cross‑repo attestation. If signatures are used, they MUST bind the anchor JSON bytes (JCS) and the Git commit oid referenced by `head`. Anchors SHOULD be signed with the same key as the containing namespace’s governance identity to preserve a coherent audit chain. ## Error Taxonomy (Normative) From 2ab91acf10f63ccecef34ac286dd310caa8a277e Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:55:21 -0800 Subject: [PATCH 65/68] =?UTF-8?q?test(privacy):=20ciphertext-only=20opaque?= =?UTF-8?q?=20pointers=20should=20deserialize=20(schema-compliant)=20?= =?UTF-8?q?=E2=80=94=20currently=20fails=20due=20to=20required=20digest=20?= =?UTF-8?q?field?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Cargo.lock | 1 - crates/gatos-privacy/tests/pointer_schema.rs | 24 ++++++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) create mode 100644 crates/gatos-privacy/tests/pointer_schema.rs diff --git a/Cargo.lock b/Cargo.lock index b310fe4f..ab6b8314 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -334,7 +334,6 @@ version = "0.1.0" dependencies = [ "anyhow", "blake3", - "gatos-ledger-core", "hex", "serde", "serde_json", diff --git a/crates/gatos-privacy/tests/pointer_schema.rs b/crates/gatos-privacy/tests/pointer_schema.rs new file mode 100644 index 00000000..d115c5b1 --- /dev/null +++ b/crates/gatos-privacy/tests/pointer_schema.rs @@ -0,0 +1,24 @@ +use gatos_privacy::OpaquePointer; + +fn read_example(rel: &str) -> String { + let dir = env!("CARGO_MANIFEST_DIR"); + std::fs::read_to_string(format!("{}/../../examples/v1/{}", dir, rel)).unwrap() +} + +#[test] +fn ciphertext_only_pointer_should_deserialize() { + // This example omits plaintext digest by design (low-entropy class) + let json = read_example("privacy/pointer_low_entropy_min.json"); + let ptr: Result = serde_json::from_str(&json); + assert!(ptr.is_ok(), "ciphertext-only opaque pointer must deserialize"); +} + +#[test] +fn both_digests_allowed_when_not_low_entropy() { + let json = read_example("privacy/opaque_pointer_min.json"); + let ptr: OpaquePointer = serde_json::from_str(&json).unwrap(); + let has_digest = !ptr.digest.is_empty(); + assert!(has_digest); + assert!(ptr.ciphertext_digest.is_some()); +} + From a0b49ccbe83b2250c4619724cb8029c8ba5e740a Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 22:56:17 -0800 Subject: [PATCH 66/68] privacy: make OpaquePointer.digest optional; add runtime validate() invariants for low-entropy class; update tests --- Cargo.lock | 21 ++++++++++ crates/gatos-privacy/Cargo.toml | 2 +- crates/gatos-privacy/src/lib.rs | 44 +++++++++++++++++++- crates/gatos-privacy/tests/pointer_schema.rs | 3 +- 4 files changed, 66 insertions(+), 4 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index ab6b8314..e863b01f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -337,6 +337,7 @@ dependencies = [ "hex", "serde", "serde_json", + "thiserror", ] [[package]] @@ -950,6 +951,26 @@ dependencies = [ "syn", ] +[[package]] +name = "thiserror" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "1.0.69" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + [[package]] name = "thread_local" version = "1.1.9" diff --git a/crates/gatos-privacy/Cargo.toml b/crates/gatos-privacy/Cargo.toml index 579db903..f2b77003 100644 --- a/crates/gatos-privacy/Cargo.toml +++ b/crates/gatos-privacy/Cargo.toml @@ -9,4 +9,4 @@ serde_json = { workspace = true } blake3 = { workspace = true } hex = { workspace = true } anyhow = { workspace = true } - +thiserror = "1" diff --git a/crates/gatos-privacy/src/lib.rs b/crates/gatos-privacy/src/lib.rs index 9b39119c..155fac64 100644 --- a/crates/gatos-privacy/src/lib.rs +++ b/crates/gatos-privacy/src/lib.rs @@ -17,7 +17,8 @@ use serde_json::Value; pub struct OpaquePointer { pub kind: Kind, pub algo: Algo, - pub digest: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub digest: Option, #[serde(skip_serializing_if = "Option::is_none")] pub ciphertext_digest: Option, #[serde(skip_serializing_if = "Option::is_none")] @@ -39,3 +40,44 @@ pub enum Kind { pub enum Algo { Blake3, } + +impl OpaquePointer { + /// Validate invariants beyond serde schema mapping. + pub fn validate(&self) -> Result<(), PointerError> { + let has_plain = self.digest.as_ref().map(|s| !s.is_empty()).unwrap_or(false); + let has_cipher = self + .ciphertext_digest + .as_ref() + .map(|s| !s.is_empty()) + .unwrap_or(false); + if !(has_plain || has_cipher) { + return Err(PointerError::MissingDigest); + } + let low_entropy = self + .extensions + .as_ref() + .and_then(|v| v.get("class")) + .and_then(|c| c.as_str()) + .map(|s| s == "low-entropy") + .unwrap_or(false); + if low_entropy { + if !has_cipher { + return Err(PointerError::LowEntropyNeedsCiphertextDigest); + } + if has_plain { + return Err(PointerError::LowEntropyForbidsPlainDigest); + } + } + Ok(()) + } +} + +#[derive(Debug, thiserror::Error, PartialEq, Eq)] +pub enum PointerError { + #[error("at least one of digest or ciphertext_digest is required")] + MissingDigest, + #[error("low-entropy class requires ciphertext_digest")] + LowEntropyNeedsCiphertextDigest, + #[error("low-entropy class forbids plaintext digest")] + LowEntropyForbidsPlainDigest, +} diff --git a/crates/gatos-privacy/tests/pointer_schema.rs b/crates/gatos-privacy/tests/pointer_schema.rs index d115c5b1..077b01a4 100644 --- a/crates/gatos-privacy/tests/pointer_schema.rs +++ b/crates/gatos-privacy/tests/pointer_schema.rs @@ -17,8 +17,7 @@ fn ciphertext_only_pointer_should_deserialize() { fn both_digests_allowed_when_not_low_entropy() { let json = read_example("privacy/opaque_pointer_min.json"); let ptr: OpaquePointer = serde_json::from_str(&json).unwrap(); - let has_digest = !ptr.digest.is_empty(); + let has_digest = ptr.digest.as_ref().map(|s| !s.is_empty()).unwrap_or(false); assert!(has_digest); assert!(ptr.ciphertext_digest.is_some()); } - From 673ac3c0974012747f610dff901728c37347843a Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Mon, 10 Nov 2025 23:21:54 -0800 Subject: [PATCH 67/68] privacy: add VerifiedOpaquePointer with validate-on-deserialize; README with invariants and examples; tests for verified wrapper --- crates/gatos-privacy/README.md | 33 ++++++++++++++++++++++++++ crates/gatos-privacy/src/lib.rs | 27 +++++++++++++++++++++ crates/gatos-privacy/tests/verified.rs | 22 +++++++++++++++++ 3 files changed, 82 insertions(+) create mode 100644 crates/gatos-privacy/README.md create mode 100644 crates/gatos-privacy/tests/verified.rs diff --git a/crates/gatos-privacy/README.md b/crates/gatos-privacy/README.md new file mode 100644 index 00000000..8bfd99ad --- /dev/null +++ b/crates/gatos-privacy/README.md @@ -0,0 +1,33 @@ +# gatos-privacy + +Opaque Pointer types and helpers for the GATOS hybrid privacy model (ADR-0004). + +Key types +- `OpaquePointer`: JSON-facing struct that mirrors `schemas/v1/privacy/opaque_pointer.schema.json`. + - `digest: Option` — plaintext digest (may be omitted) + - `ciphertext_digest: Option` — ciphertext digest + - `extensions.class = "low-entropy"` implies `ciphertext_digest` MUST be present and `digest` MUST be absent. + +- `VerifiedOpaquePointer`: wrapper that enforces invariants during deserialization. + - Use this at trust boundaries to guarantee the low-entropy rules. + +Validation +- After deserializing `OpaquePointer`, call `pointer.validate()` to enforce: + - At least one of `digest` or `ciphertext_digest` is present. + - Low-entropy class requires `ciphertext_digest` and forbids `digest`. + +Examples +```rust +use gatos_privacy::{OpaquePointer, VerifiedOpaquePointer}; + +// 1) Verified wrapper enforces invariants automatically +let v: VerifiedOpaquePointer = serde_json::from_str(json)?; + +// 2) Manual validation on the plain struct +let p: OpaquePointer = serde_json::from_str(json)?; +p.validate()?; +``` + +Canonicalization +- When computing content IDs or digests, serialize JSON with RFC 8785 JCS (performed by higher layers). + diff --git a/crates/gatos-privacy/src/lib.rs b/crates/gatos-privacy/src/lib.rs index 155fac64..3e7ae3f3 100644 --- a/crates/gatos-privacy/src/lib.rs +++ b/crates/gatos-privacy/src/lib.rs @@ -81,3 +81,30 @@ pub enum PointerError { #[error("low-entropy class forbids plaintext digest")] LowEntropyForbidsPlainDigest, } + +/// A validated wrapper that enforces `OpaquePointer::validate()` during +/// deserialization. Prefer this type when accepting pointers from untrusted +/// inputs; it guarantees schema-level invariants at the boundary. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[serde(transparent)] +pub struct VerifiedOpaquePointer(pub OpaquePointer); + +impl<'de> Deserialize<'de> for VerifiedOpaquePointer { + fn deserialize(deserializer: D) -> Result + where + D: serde::Deserializer<'de>, + { + let inner = OpaquePointer::deserialize(deserializer)?; + inner + .validate() + .map_err(serde::de::Error::custom)?; + Ok(Self(inner)) + } +} + +impl core::ops::Deref for VerifiedOpaquePointer { + type Target = OpaquePointer; + fn deref(&self) -> &Self::Target { + &self.0 + } +} diff --git a/crates/gatos-privacy/tests/verified.rs b/crates/gatos-privacy/tests/verified.rs new file mode 100644 index 00000000..b0343aa1 --- /dev/null +++ b/crates/gatos-privacy/tests/verified.rs @@ -0,0 +1,22 @@ +use gatos_privacy::{OpaquePointer, VerifiedOpaquePointer}; + +fn read_example(rel: &str) -> String { + let dir = env!("CARGO_MANIFEST_DIR"); + std::fs::read_to_string(format!("{}/../../examples/v1/{}", dir, rel)).unwrap() +} + +#[test] +fn verified_accepts_ciphertext_only_low_entropy() { + let json = read_example("privacy/pointer_low_entropy_min.json"); + let v: VerifiedOpaquePointer = serde_json::from_str(&json).expect("verified deserialize"); + assert!(v.ciphertext_digest.is_some()); + assert!(v.digest.is_none()); +} + +#[test] +fn verified_rejects_low_entropy_with_plain_digest() { + let json = read_example("privacy/pointer_low_entropy_invalid.json"); + let v: Result = serde_json::from_str(&json); + assert!(v.is_err(), "should reject invalid low-entropy pointer"); +} + From 1379a292014a5ee243b52a31b6ab4dedb3a7df41 Mon Sep 17 00:00:00 2001 From: "J. Kirby Ross" Date: Tue, 11 Nov 2025 10:42:10 -0800 Subject: [PATCH 68/68] =?UTF-8?q?ADR-0004:=20Implementation=20note=20?= =?UTF-8?q?=E2=80=94=20validate=20pointers=20on=20ingest=20(use=20Verified?= =?UTF-8?q?OpaquePointer=20or=20call=20validate())?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/decisions/ADR-0004/DECISION.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/decisions/ADR-0004/DECISION.md b/docs/decisions/ADR-0004/DECISION.md index 1b14f13b..c3611072 100644 --- a/docs/decisions/ADR-0004/DECISION.md +++ b/docs/decisions/ADR-0004/DECISION.md @@ -135,6 +135,8 @@ When encrypting a private blob referenced by an Opaque Pointer, the AEAD AAD MUS Implementations MAY structure these as concatenated bytes with clear domain separation (e.g., length‑prefixing) prior to supplying them as AAD to the AEAD algorithm. +> Implementation note (non‑normative): When ingesting Opaque Pointers from untrusted JSON, implementations SHOULD validate the invariants at parse time (e.g., verify that low‑entropy pointers include `ciphertext_digest` and omit `digest`). In this repository, the `gatos-privacy` crate exposes `VerifiedOpaquePointer` which enforces these rules during deserialization; alternatively, callers can deserialize `OpaquePointer` and invoke `validate()` explicitly. + ### 3. The Projection Function (Normative) The State Engine (`gatos-echo`) is responsible for executing the projection.