Cycle 0008: Stream architecture + P5 codec dissolution

CHANGELOG.md

@@ -7,8 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **WarpStream** — composable async stream primitive built on `AsyncIterable<T>`. Domain concept for "data flow over time." Supports `pipe`, `tee`, `mux`, `demux`, `drain`, `reduce`, `forEach`, `collect`. Natural backpressure via `for await`, error propagation via async iterator protocol, cooperative cancellation via `AbortSignal`.
+- **Stream transforms** — `CborEncodeTransform`, `CborDecodeTransform`, `GitBlobWriteTransform`, `TreeAssemblerSink`, `IndexShardEncodeTransform` — composable infrastructure pipeline stages for encode → blobWrite → treeAssemble.
+- **Artifact record classes** — `CheckpointArtifact` family (`StateArtifact`, `FrontierArtifact`, `AppliedVVArtifact`), `IndexShard` family (`MetaShard`, `EdgeShard`, `LabelShard`, `PropertyShard`, `ReceiptShard`), `PatchEntry`, `ProvenanceEntry`. Runtime-backed domain nouns with constructor validation and `instanceof` dispatch.
+- **`PatchJournalPort.scanPatchRange()`** — streaming alternative to `loadPatchRange()`. Returns `WarpStream<PatchEntry>` for incremental patch consumption. Commit walking moved from SyncProtocol into the adapter.
+- **`StateHashService`** — standalone canonical state hash computation. Separately callable by checkpoint creation, comparison, materialization, and verification.
+- **Index builder `yieldShards()`** — `LogicalBitmapIndexBuilder` and `PropertyIndexBuilder` yield `IndexShard` record instances via generators. Proven byte-identical to legacy `serialize()` path.
+- **`LogicalIndexBuildService.buildStream()`** — returns `WarpStream<IndexShard>` merging both builders via `mux()`.
+- **Hex tripwire test** — 36 automated checks scanning domain files for forbidden codec imports/usage. Fails loud if domain touches `defaultCodec`, `cbor-x`, or `codec.encode()`/`codec.decode()`.
+- **Golden fixtures** — known CBOR bytes for patches, checkpoints, VV, frontier, index shards. Wire format stability proven across refactor.
+
+### Fixed
+
+- **CborPatchJournalAdapter.readPatch()** — now throws `EncryptionError` when `encrypted=true` but no `patchBlobStorage` is configured. Previously fell through to the unencrypted `blobPort.readBlob()` path, returning wrong data.
+- **Writer constructor** — `patchJournal` is now a required parameter. Previously optional, which let `beginPatch()` succeed but `commit()` hard-fail with a confusing `PersistenceError` from `PatchBuilderV2`.
+
 ### Changed
 
+- **CheckpointStorePort collapsed** — 7 micro-methods (`writeState`, `readState`, etc.) replaced with 2 semantic operations: `writeCheckpoint(record)` and `readCheckpoint(treeOids)`. A checkpoint is one domain event, not a bag of individual blob writes.
+- **SyncProtocol uses stream scan** — `processSyncRequest()` prefers `patchJournal.scanPatchRange()` (streaming) over `loadPatchRange()` (array). Falls back to legacy path when unavailable.
+- **PatchBuilderV2, SyncProtocol, Writer** — codec-free. Patch persistence goes through `PatchJournalPort`; domain never imports `defaultCodec` or calls `codec.encode()`/`codec.decode()`.
+- **CheckpointService** — routes through `CheckpointStorePort` when available. Legacy codec-based paths remain as fallback.
 - **The Method** — introduced `METHOD.md` as the development process framework. Filesystem-native backlog (`docs/method/backlog/`) with lane directories (`inbox/`, `asap/`, `up-next/`, `cool-ideas/`, `bad-code/`). Legend-prefixed filenames (`PROTO_`, `TRUST_`, `VIZ_`, `TUI_`, `DX_`, `PERF_`). Sequential cycle numbering (`docs/design/<NNNN-slug>/`). Dual-audience design docs (sponsor human + sponsor agent). Replaced B-number system entirely.
 - **Backlog migration** — all 49 B-number and OG items migrated from `BACKLOG/` to `docs/method/backlog/` lanes. Tech debt journal (`.claude/bad_code.md`) split into 10 individual files in `bad-code/`. Cool ideas journal split into 13 individual files in `cool-ideas/`. `docs/release.md` moved to `docs/method/release.md`. `BACKLOG/` directory removed.
 

contracts/type-surface.m8.json

@@ -1237,6 +1237,18 @@
     "WriterError": {
       "kind": "class"
     },
+    "WorldlineSelector": {
+      "kind": "class"
+    },
+    "LiveSelector": {
+      "kind": "class"
+    },
+    "CoordinateSelector": {
+      "kind": "class"
+    },
+    "StrandSelector": {
+      "kind": "class"
+    },
     "buildWarpStateIndex": {
       "kind": "function",
       "async": true

docs/BEARING.md

@@ -2,6 +2,18 @@
 
 Updated at cycle boundaries. Not mid-cycle.
 
+## Invariants
+
+1. **HEXAGONAL** — domain never imports infrastructure
+2. **DETERMINISTIC** — same patches, any order → same materialized state
+3. **APPEND-ONLY** — Git history never rewritten
+4. **MULTI-WRITER** — each writer owns its own ref, no coordination
+5. **RUNTIME-TRUTH** — domain concepts are classes with validated invariants (SSJS P1)
+6. **BOUNDARY-HONESTY** — untrusted input validated at boundaries
+7. **TRAVERSAL-TRUTH** — unbounded data flows through streams (traversal);
+   bounded truth crosses ports (contracts). Never conflated. Persistence
+   ordering is canonical regardless of stream timing.
+
 ## Where are we going?
 
 Structural decomposition of `domain/services/` — 83 files in a flat
@@ -15,9 +27,13 @@ analysis, 10 cohesive groups identified, no circular dependencies.
 
 ## What feels wrong?
 
-- WorldlineSource is still a tagged object, not a subclass hierarchy.
-- `defaultCodec.js` lives in `domain/utils/` but imports `cbor-x`
-  directly — a hexagonal boundary violation.
+- ~~WorldlineSource~~ Shipped as WorldlineSelector hierarchy (cycle 0007).
+- 20 domain services do serialization directly (`codec.encode()`/
+  `codec.decode()`). The fix is a two-stage boundary: artifact-level
+  ports (PatchJournalPort, CheckpointStorePort, etc.) that speak
+  domain types, backed by codec-owning adapters over the raw Git
+  ports. Strangler refactor, patches first.
+  See `NDNM_defaultcodec-to-infrastructure.md`.
 - The two legends (CLEAN_CODE, NO_DOGS_NO_MASTERS) overlap
   significantly. May need consolidation or clearer boundaries.
 - JoinReducer is imported by 8 of 10 service clusters — it is the

docs/design/0007-viewpoint-design/viewpoint-design.md

@@ -414,7 +414,7 @@ change shape.
 The inbound boundary factory. Accepts plain objects with a `kind`
 discriminant and returns the appropriate class instance:
 
-```javascript
+```text
 WorldlineSelector.from({ kind: 'live' })
   → new LiveSelector()
 
@@ -526,31 +526,31 @@ const selector = WorldlineSelector.from(source).clone();
 
 **Before (materializeSource dispatch):**
 
-```javascript
+```text
 if (source.kind === 'live') { ... }
 if (source.kind === 'coordinate') { ... }
 return await materializeStrandSource(...);
 ```
 
 **After:**
 
-```javascript
+```text
 if (source instanceof LiveSelector) { ... }
 if (source instanceof CoordinateSelector) { ... }
 return await materializeStrandSource(...);
 ```
 
 **Before (Worldline.source getter):**
 
-```javascript
+```text
 get source() {
   return cloneWorldlineSource(this._source);
 }
 ```
 
 **After:**
 
-```javascript
+```text
 get source() {
   return this._source.toDTO();  // plain object for public API
 }
@@ -611,7 +611,7 @@ export {
 Add the selector class declarations. Keep the existing
 `WorldlineSource` type and interfaces unchanged:
 
-```typescript
+```text
 // NEW — selector classes
 export class WorldlineSelector {
   clone(): WorldlineSelector;

docs/design/0008-stream-architecture/stream-architecture.md

@@ -0,0 +1,234 @@
+# Cycle 0008 — Stream Architecture
+
+**Sponsor (human):** James
+**Sponsor (agent):** Claude
+**Status:** DESIGN
+
+## Hill
+
+A developer can pipe domain objects through a composable stream
+pipeline where encoding, persistence, and tree assembly are transforms
+and sinks — never called directly by domain code. Semantic ports
+remain for bounded single-artifact operations. Artifact records carry
+runtime identity. The system is memory-bounded for unbounded datasets.
+
+## The Rule
+
+Streams are for scale. Ports are for meaning. Artifacts are the nouns.
+Paths are infrastructure.
+
+## Playback Questions
+
+1. Does the pipeline produce byte-identical output to the legacy path?
+2. Does a constrained-heap test complete for a dataset that would
+   otherwise OOM?
+3. Do semantic ports still tell you WHAT is being persisted and WHAT
+   lifecycle rules apply?
+4. Is CBOR vocabulary absent from domain nouns?
+5. Does every artifact record class add runtime identity, not just a name?
+
+## Non-Goals
+
+- CborStream or any codec-named class in the domain
+- Marker stream subclasses that don't add flow behavior
+- Melting separate ports/services into one generic pipe
+- Replacing bounded single-artifact reads with streams
+
+---
+
+## Architecture
+
+### One Stream Container
+
+```text
+WarpStream<T>                — domain primitive
+  pipe / tee / mux / demux / drain / reduce / forEach / collect
+  [Symbol.asyncIterator]()
+```
+
+No domain subclasses. Identity lives on elements, not the container.
+
+### Semantic Ports
+
+Ports define what is being persisted and what lifecycle rules apply.
+Bounded operations stay `Promise<T>`. Unbounded operations return or
+accept `WarpStream<SemanticUnit>`.
+
+**PatchJournalPort** (keep, extend)
+```text
+writePatch(patch) → Promise<string>              bounded write
+readPatch(oid) → Promise<PatchV2>                bounded read
+scanPatchRange(...) → WarpStream<PatchEntry>     unbounded scan (NEW)
+```
+
+**CheckpointStorePort** (collapse micro-methods)
+```text
+writeCheckpoint(record) → Promise<CheckpointWriteResult>   one call
+readCheckpoint(sha) → Promise<CheckpointData>              bounded read
+```
+Adapter internally streams artifacts through the pipeline.
+
+**IndexStorePort** (NEW, streaming)
+```text
+writeShards(stream) → Promise<string>            WarpStream<IndexShard> → tree OID
+scanShards(...) → WarpStream<IndexShard>         unbounded read
+```
+
+**ProvenanceStorePort** (NEW, separate concept)
+```text
+scanEntries(...) → WarpStream<ProvenanceEntry>
+writeIndex(index) → Promise<string>
+```
+Own port. Physical colocation under checkpoint tree ≠ semantic
+ownership. Checkpoint = recovery. Provenance = causal/query/verification.
+Different jobs, different lifecycle, different consumers.
+
+**StateHashService** (separate callable, not buried in adapter)
+```text
+compute(state) → Promise<string>
+```
+Used by verification, comparison, detached checks, AND checkpoint
+creation. Not exclusively inside writeCheckpoint().
+
+### Artifact Records
+
+Runtime identity on elements, not containers (P1/P7).
+
+**CheckpointArtifact** — closed subclass family
+```text
+CheckpointArtifact (abstract base)
+  common: checkpointRef, schemaVersion
+
+StateArtifact extends CheckpointArtifact
+  payload: { state: WarpStateV5 }
+
+FrontierArtifact extends CheckpointArtifact
+  payload: { frontier: Map<string, string> }
+
+AppliedVVArtifact extends CheckpointArtifact
+  payload: { appliedVV: VersionVector }
+```
+No paths. No CBOR. No blob OIDs. No adapter trivia.
+
+**IndexShard** — subtype family (not one generic class)
+```text
+IndexShard (base)
+  common: indexFamily, shardId, schemaVersion
+
+MetaShard extends IndexShard
+  payload: { nodeToGlobal, alive, nextLocalId }
+
+EdgeShard extends IndexShard
+  payload: { direction, shardKey, buckets }
+
+LabelShard extends IndexShard
+  payload: { labels: [string, number][] }
+
+PropertyShard extends IndexShard
+  payload: { entries: [string, Record][] }
+```
+The code already treats shard families differently (isMetaShard,
+isEdgeShard, classifyShards). One mega-shard class is just `any`
+with better PR.
+
+**PatchEntry** — `{ patch: PatchV2, sha: string }`
+
+**ProvenanceEntry** — `{ nodeId, patchShas }`
+
+### Path Mapping
+
+Adapter owns it. Full stop. Domain produces artifact records.
+Adapter maps to Git tree paths at the last responsible moment.
+
+```text
+StateArtifact   → 'state.cbor'
+FrontierArtifact → 'frontier.cbor'
+MetaShard       → 'meta_XX.cbor'
+EdgeShard       → '{fwd|rev}_XX.cbor'
+```
+
+Static mapping table or instanceof dispatcher in the adapter.
+No `.path()` on domain objects. Paths are storage convention.
+
+Domain owns meaning. Adapter owns layout.
+
+### Infrastructure Transforms
+
+```text
+CborEncodeTransform   artifact → [path, bytes]
+CborDecodeTransform   [path, bytes] → artifact
+GitBlobWriteTransform [path, bytes] → [path, oid]
+TreeAssemblerSink     [path, oid] → finalize → treeOid
+```
+
+Encode → blobWrite → treeAssemble stays entirely in infrastructure.
+CBOR is boundary vocabulary — never a domain noun.
+
+### Pipeline Examples
+
+```js
+// Index write (unbounded, streaming)
+await indexStore.writeShards(
+  WarpStream.from(builder.yieldShards())
+);
+// Adapter internally: stream → encode → blobWrite → treeAssemble
+
+// Checkpoint write (bounded, one call)
+await checkpointStore.writeCheckpoint({
+  state, frontier, appliedVV, stateHash, provenanceIndex
+});
+// Adapter internally: yield artifacts → encode → blobWrite → tree
+
+// Patch scan (unbounded)
+const patches = patchJournal.scanPatchRange(writerRef, fromSha, toSha);
+for await (const entry of patches) {
+  reducer.apply(entry.patch);
+}
+```
+
+### Ordering Guarantee
+
+`WarpStream.mux()` interleaves by arrival order. Async completion
+timing must not bleed into tree assembly. `TreeAssemblerSink` sorts
+entries before `writeTree()`. Deterministic Git trees don't care
+which blob write finished first.
+
+---
+
+## Migration Plan
+
+### Phase 1 — Artifact records + streaming ports
+
+- CheckpointArtifact family (StateArtifact, FrontierArtifact,
+  AppliedVVArtifact)
+- IndexShard family (MetaShard, EdgeShard, LabelShard, PropertyShard)
+- PatchEntry, ProvenanceEntry records
+- IndexStorePort with writeShards/scanShards
+- PatchJournalPort.scanPatchRange()
+- StateHashService
+- ProvenanceStorePort
+
+### Phase 2 — Wire write paths
+
+- CheckpointStorePort collapse → writeCheckpoint(record)
+- Index builders: yieldShards() returns IndexShard subclass instances
+- SyncProtocol: consume scanPatchRange() instead of loadPatchRange()
+
+### Phase 3 — P5 cleanup
+
+- Remove defaultCodec from all domain files
+- Delete defaultCodec.js, canonicalCbor.js
+- Expand tripwire to all migrated files
+
+### Phase 4 — Memory-bounded witnesses
+
+- Constrained-heap tests
+- Naming audit for slurp APIs
+
+---
+
+## Accessibility / Localization / Agent-Inspectability
+
+- **Agent-Inspectability**: Artifact records are `instanceof`-
+  dispatchable. WarpStream carries AbortSignal. Sink.consume()
+  returns typed results.