From 2c6b8c81481b76542049eab7138326edf3fae0b6 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 13 Mar 2026 21:10:07 +0000 Subject: [PATCH 1/3] Initial plan From ea0e43512e8004e54ad6604e8cfb9f0c17caade8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 13 Mar 2026 21:16:34 +0000 Subject: [PATCH 2/3] Remove broken TODO-to-issue sync system and set up GitHub-native project management - Remove scripts/sync-github-project.mjs and its workflow - Remove TODO.md and PLAN.md (replaced by GitHub Issues/Projects) - Remove sync:github-project npm script - Add issue templates (bug, feature, task) with structured forms - Add PR template with checklist - Add auto-label workflow using actions/labeler with path-based rules - Add close-legacy-issues workflow to close all 29 existing issues - Add stale issue/PR management workflow - Update copilot-instructions.md, README.md, DESIGN.md, docs/development.md Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug.yml | 71 ++ .github/ISSUE_TEMPLATE/config.yml | 5 + .github/ISSUE_TEMPLATE/feature.yml | 60 ++ .github/ISSUE_TEMPLATE/task.yml | 77 ++ .github/PULL_REQUEST_TEMPLATE.md | 19 + .github/copilot-instructions.md | 22 +- .github/labeler.yml | 60 ++ .github/workflows/auto-label.yml | 19 + .github/workflows/close-legacy-issues.yml | 67 ++ .github/workflows/stale.yml | 37 + .github/workflows/sync-github-project.yml | 37 - DESIGN.md | 2 +- PLAN.md | 493 ------------ README.md | 12 +- TODO.md | 892 ---------------------- docs/development.md | 8 +- package.json | 3 +- scripts/sync-github-project.mjs | 495 ------------ 18 files changed, 448 insertions(+), 1931 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug.yml create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/feature.yml create mode 100644 .github/ISSUE_TEMPLATE/task.yml create mode 100644 .github/PULL_REQUEST_TEMPLATE.md create mode 100644 .github/labeler.yml create mode 100644 .github/workflows/auto-label.yml create mode 100644 .github/workflows/close-legacy-issues.yml create mode 100644 .github/workflows/stale.yml delete mode 100644 .github/workflows/sync-github-project.yml delete mode 100644 PLAN.md delete mode 100644 TODO.md delete mode 100644 scripts/sync-github-project.mjs diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml new file mode 100644 index 0000000..67a9954 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -0,0 +1,71 @@ +name: "πŸ› Bug Report" +description: Report a bug in CORTEX +labels: ["bug"] +body: + - type: markdown + attributes: + value: | + Thanks for reporting a bug. Please fill in the details below. + + - type: textarea + id: description + attributes: + label: Description + description: A clear and concise description of the bug. + validations: + required: true + + - type: textarea + id: reproduction + attributes: + label: Steps to Reproduce + description: Minimal steps to reproduce the behavior. + placeholder: | + 1. Call `ingestText(...)` with ... + 2. Then call `query(...)` with ... + 3. Observe ... + validations: + required: true + + - type: textarea + id: expected + attributes: + label: Expected Behavior + description: What you expected to happen. + validations: + required: true + + - type: textarea + id: actual + attributes: + label: Actual Behavior + description: What actually happened. Include error messages or screenshots if applicable. + validations: + required: true + + - type: dropdown + id: layer + attributes: + label: Affected Layer + description: Which CORTEX layer does this bug affect? + multiple: true + options: + - Foundation (core/) + - Storage (storage/) + - Vector Compute (root backends) + - Embeddings (embeddings/) + - Hippocampus (hippocampus/) + - Cortex (cortex/) + - Daydreamer + - Runtime Harness (runtime/) + - CI / Build + - Documentation + validations: + required: true + + - type: input + id: browser + attributes: + label: Browser / Runtime + description: Browser version, Electron version, or Node/Bun version. + placeholder: "Chrome 120, Electron 28, Bun 1.3.10" diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..ab8c2fc --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,5 @@ +blank_issues_enabled: true +contact_links: + - name: "πŸ“– Architecture Reference" + url: https://github.com/devlux76/cortex/blob/main/DESIGN.md + about: Review DESIGN.md before proposing architectural changes. diff --git a/.github/ISSUE_TEMPLATE/feature.yml b/.github/ISSUE_TEMPLATE/feature.yml new file mode 100644 index 0000000..137c437 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature.yml @@ -0,0 +1,60 @@ +name: "✨ Feature Request" +description: Propose a new feature or enhancement for CORTEX +labels: ["enhancement"] +body: + - type: markdown + attributes: + value: | + Describe the feature you'd like to see in CORTEX. + + - type: textarea + id: summary + attributes: + label: Summary + description: A concise description of the proposed feature. + validations: + required: true + + - type: textarea + id: motivation + attributes: + label: Motivation + description: Why is this feature needed? What problem does it solve? + validations: + required: true + + - type: textarea + id: design + attributes: + label: Proposed Design + description: How should this feature work? Reference DESIGN.md sections if applicable. + + - type: dropdown + id: layer + attributes: + label: Target Layer + description: Which CORTEX layer would this feature primarily affect? + multiple: true + options: + - Foundation (core/) + - Storage (storage/) + - Vector Compute (root backends) + - Embeddings (embeddings/) + - Hippocampus (hippocampus/) + - Cortex (cortex/) + - Daydreamer + - Runtime Harness (runtime/) + - CI / Build + - Documentation + validations: + required: true + + - type: textarea + id: exit-criteria + attributes: + label: Exit Criteria + description: What must be true for this feature to be considered complete? + placeholder: | + - [ ] Unit tests pass + - [ ] Integration test covers end-to-end flow + - [ ] DESIGN.md updated if architecture changed diff --git a/.github/ISSUE_TEMPLATE/task.yml b/.github/ISSUE_TEMPLATE/task.yml new file mode 100644 index 0000000..788955f --- /dev/null +++ b/.github/ISSUE_TEMPLATE/task.yml @@ -0,0 +1,77 @@ +name: "πŸ”§ Implementation Task" +description: Define a specific implementation task +labels: ["task"] +body: + - type: markdown + attributes: + value: | + Define a concrete implementation task. Reference DESIGN.md for architectural context. + + - type: textarea + id: objective + attributes: + label: Objective + description: What needs to be built or changed? + validations: + required: true + + - type: dropdown + id: priority + attributes: + label: Priority + options: + - "P0: critical β€” blocks dependent work" + - "P1: high β€” targets next milestone" + - "P2: medium β€” important but not blocking" + - "P3: low β€” polish or nice-to-have" + validations: + required: true + + - type: dropdown + id: layer + attributes: + label: Target Layer + description: Which CORTEX layer does this task target? + multiple: true + options: + - Foundation (core/) + - Storage (storage/) + - Vector Compute (root backends) + - Embeddings (embeddings/) + - Hippocampus (hippocampus/) + - Cortex (cortex/) + - Daydreamer + - Runtime Harness (runtime/) + - CI / Build + - Documentation + validations: + required: true + + - type: textarea + id: files + attributes: + label: Files to Create or Modify + description: List the files this task will touch. + placeholder: | + - `cortex/MetroidBuilder.ts` (create) + - `tests/cortex/MetroidBuilder.test.ts` (create) + - `DESIGN.md` (update if architecture changes) + + - type: textarea + id: exit-criteria + attributes: + label: Exit Criteria + description: Checklist of conditions that must be met to close this task. + placeholder: | + - [ ] Implementation complete + - [ ] Unit tests passing + - [ ] Lint and typecheck clean + - [ ] guard:model-derived passes (if numerics changed) + validations: + required: true + + - type: textarea + id: dependencies + attributes: + label: Dependencies + description: List any issues or tasks that must be completed first. Use `#issue_number` references. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..acb20c5 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,19 @@ +## Summary + + + +## Changes + + + +- + +## Checklist + +- [ ] Lint passes (`npm run lint`) +- [ ] Typecheck passes (`npm run build`) +- [ ] Unit tests pass (`npm run test:unit`) +- [ ] `guard:model-derived` passes (if numeric constants changed) +- [ ] DESIGN.md updated (if architecture changed) +- [ ] No hardcoded model-derived numbers β€” all sourced from `core/` +- [ ] No server-side dependencies, cloud calls, or telemetry added diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 094079e..0fb2fc1 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -15,10 +15,20 @@ The engine models three biological brain regions: |---|---| | `README.md` | Product vision and quick start | | `DESIGN.md` | Complete architecture specification and design principles | -| `PLAN.md` | Module-by-module implementation status and development phases | -| `TODO.md` | Prioritized actionable tasks to ship v1.0 | -Keep all documents synchronized with the real code state after every implementation pass. +Keep `DESIGN.md` synchronized with the real code state after every implementation pass. + +## Project Management + +All task tracking, prioritization, and status is managed through **GitHub-native features** β€” not markdown files: + +- **GitHub Issues** β€” Every task, bug, and feature request is a GitHub Issue (use the issue templates in `.github/ISSUE_TEMPLATE/`). +- **GitHub Projects** β€” Use project boards for Kanban-style lifecycle tracking (To Do β†’ In Progress β†’ Done). +- **Milestones** β€” Group issues by release phase (`v0.1`, `v0.5`, `v1.0`). +- **Labels** β€” Auto-applied on PRs by `.github/workflows/auto-label.yml` using path-based rules in `.github/labeler.yml`. Priority labels (`P0`–`P3`) and layer labels (`layer: foundation`, `layer: cortex`, etc.) are used for classification. +- **Issue linking** β€” Reference dependencies with `#issue_number`. Use `Closes #N` in PR descriptions to auto-close issues on merge. + +Agents with `gh` CLI access can create, update, and close issues directly. Do not create markdown files for task tracking. ## Directory Structure @@ -119,8 +129,10 @@ Run `npm run guard:model-derived` after any numeric change to verify compliance. - All CI checks must pass: `lint`, `build` (typecheck), `test:unit`. - `npm run guard:model-derived` must pass for any change that touches numeric constants. -- Keep `README.md`, `CORTEX-DESIGN-PLAN-TODO.md`, and `PROJECT-EXECUTION-PLAN.md` synchronized with any implementation state changes. -- Record blockers with file path, failure symptom, and next action. +- PRs are auto-labeled by layer based on changed files (`.github/labeler.yml`). +- Use the PR template (`.github/PULL_REQUEST_TEMPLATE.md`) β€” it is pre-populated on every PR. +- Reference the issue being addressed with `Closes #N` in the PR description. +- Keep `DESIGN.md` synchronized with any architectural changes. ## What NOT to Do diff --git a/.github/labeler.yml b/.github/labeler.yml new file mode 100644 index 0000000..8bc2e27 --- /dev/null +++ b/.github/labeler.yml @@ -0,0 +1,60 @@ +# Maps file-path globs to labels applied automatically on PRs. +# Used by the auto-label.yml workflow (actions/labeler). + +"layer: foundation": + - changed-files: + - any-glob-to-any-file: "core/**" + +"layer: storage": + - changed-files: + - any-glob-to-any-file: "storage/**" + +"layer: compute": + - changed-files: + - any-glob-to-any-file: + - "VectorBackend.ts" + - "WebGPUVectorBackend.ts" + - "WebGLVectorBackend.ts" + - "WebNNVectorBackend.ts" + - "WasmVectorBackend.ts" + - "CreateVectorBackend.ts" + - "BackendKind.ts" + - "TopK.ts" + - "Vectors.glsl" + - "Vectors.wgsl" + - "Vectors.wat" + +"layer: embeddings": + - changed-files: + - any-glob-to-any-file: "embeddings/**" + +"layer: hippocampus": + - changed-files: + - any-glob-to-any-file: "hippocampus/**" + +"layer: cortex": + - changed-files: + - any-glob-to-any-file: "cortex/**" + +"layer: daydreamer": + - changed-files: + - any-glob-to-any-file: "daydreamer/**" + +"layer: testing": + - changed-files: + - any-glob-to-any-file: "tests/**" + +"layer: ci": + - changed-files: + - any-glob-to-any-file: + - ".github/workflows/**" + - ".github/labeler.yml" + +"layer: documentation": + - changed-files: + - any-glob-to-any-file: + - "*.md" + - "docs/**" + - ".github/copilot-instructions.md" + - ".github/ISSUE_TEMPLATE/**" + - ".github/PULL_REQUEST_TEMPLATE.md" diff --git a/.github/workflows/auto-label.yml b/.github/workflows/auto-label.yml new file mode 100644 index 0000000..85f437d --- /dev/null +++ b/.github/workflows/auto-label.yml @@ -0,0 +1,19 @@ +name: Auto-Label PRs + +on: + pull_request: + types: [opened, synchronize, reopened] + +permissions: + contents: read + pull-requests: write + +jobs: + label: + runs-on: ubuntu-latest + steps: + - name: Label PR by changed files + uses: actions/labeler@v5 + with: + configuration-path: .github/labeler.yml + sync-labels: true diff --git a/.github/workflows/close-legacy-issues.yml b/.github/workflows/close-legacy-issues.yml new file mode 100644 index 0000000..dbab262 --- /dev/null +++ b/.github/workflows/close-legacy-issues.yml @@ -0,0 +1,67 @@ +name: Close Legacy Issues + +# One-shot workflow: closes all pre-existing issues that were created by the +# removed sync-github-project.mjs script. Run manually, then delete this file. + +on: + workflow_dispatch: + +permissions: + issues: write + +jobs: + close-all: + runs-on: ubuntu-latest + steps: + - name: Close all open issues as won't-fix + uses: actions/github-script@v7 + with: + script: | + const owner = context.repo.owner; + const repo = context.repo.repo; + + // Ensure the "wontfix" label exists + try { + await github.rest.issues.getLabel({ owner, repo, name: "wontfix" }); + } catch { + await github.rest.issues.createLabel({ + owner, repo, + name: "wontfix", + color: "ffffff", + description: "Closed during migration to GitHub-native project management" + }); + } + + // Paginate through all open issues + let page = 1; + let closed = 0; + while (true) { + const { data: issues } = await github.rest.issues.listForRepo({ + owner, repo, + state: "open", + per_page: 100, + page, + }); + if (issues.length === 0) break; + + for (const issue of issues) { + // Skip pull requests (they appear in the issues API) + if (issue.pull_request) continue; + + await github.rest.issues.addLabels({ + owner, repo, + issue_number: issue.number, + labels: ["wontfix"], + }); + await github.rest.issues.update({ + owner, repo, + issue_number: issue.number, + state: "closed", + state_reason: "not_planned", + }); + console.log(`Closed #${issue.number}: ${issue.title}`); + closed++; + } + page++; + } + console.log(`\nDone β€” closed ${closed} issues.`); diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml new file mode 100644 index 0000000..0d4760b --- /dev/null +++ b/.github/workflows/stale.yml @@ -0,0 +1,37 @@ +name: Stale Issue Management + +on: + schedule: + - cron: "30 2 * * 1" # Every Monday at 02:30 UTC + workflow_dispatch: + +permissions: + issues: write + pull-requests: write + +jobs: + stale: + runs-on: ubuntu-latest + steps: + - name: Mark and close stale issues and PRs + uses: actions/stale@v9 + with: + stale-issue-message: > + This issue has had no activity for 60 days and will be closed in 14 + days unless there is new activity. If this is still relevant, please + comment or remove the `stale` label. + close-issue-message: > + Closed due to inactivity. Reopen if this is still needed. + stale-pr-message: > + This PR has had no activity for 30 days and will be closed in 14 + days unless there is new activity. + close-pr-message: > + Closed due to inactivity. Reopen if this is still needed. + days-before-stale: 60 + days-before-close: 14 + days-before-pr-stale: 30 + days-before-pr-close: 14 + stale-issue-label: stale + stale-pr-label: stale + exempt-issue-labels: "P0: critical,pinned" + exempt-pr-labels: "pinned" diff --git a/.github/workflows/sync-github-project.yml b/.github/workflows/sync-github-project.yml deleted file mode 100644 index e09aa32..0000000 --- a/.github/workflows/sync-github-project.yml +++ /dev/null @@ -1,37 +0,0 @@ -name: Sync GitHub Project - -on: - push: - branches: - - main - paths: - - "TODO.md" - workflow_dispatch: - -concurrency: - group: sync-github-project - cancel-in-progress: false - -permissions: - issues: write - contents: read - -jobs: - sync: - runs-on: ubuntu-latest - steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Setup Bun - uses: oven-sh/setup-bun@v2 - with: - bun-version: "latest" - - - name: Install dependencies - run: bun install --frozen-lockfile - - - name: Sync GitHub Project (milestones, labels, issues) - env: - GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: bun run sync:github-project diff --git a/DESIGN.md b/DESIGN.md index 2f1dfe9..2501836 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -407,7 +407,7 @@ This keeps subgraph expansion cost sublinear in graph mass at scale while remain ### Policy Source of Truth -All hotpath constants β€” `c`, `Ξ±`, `Ξ²`, `Ξ³`, `q_s`, `q_v`, `q_b`, `q_p` β€” live in `core/HotpathPolicy.ts` as a frozen default policy object. These are **policy-derived constants** (not model-derived) and are kept strictly separate from `core/ModelDefaults.ts`. A companion guard (or an extension to `guard:model-derived`) is planned (see TODO.md P3-E3) to prevent these constants from being hardcoded elsewhere; until that guard is in place, discipline is enforced by convention. +All hotpath constants β€” `c`, `Ξ±`, `Ξ²`, `Ξ³`, `q_s`, `q_v`, `q_b`, `q_p` β€” live in `core/HotpathPolicy.ts` as a frozen default policy object. These are **policy-derived constants** (not model-derived) and are kept strictly separate from `core/ModelDefaults.ts`. A companion guard (or an extension to `guard:model-derived`) is planned to prevent these constants from being hardcoded elsewhere; until that guard is in place, discipline is enforced by convention. --- diff --git a/PLAN.md b/PLAN.md deleted file mode 100644 index 0d8c942..0000000 --- a/PLAN.md +++ /dev/null @@ -1,493 +0,0 @@ -# CORTEX Implementation Plan - -**Version:** 1.2 -**Last Updated:** 2026-03-13 - -This document tracks the implementation status of each major module in CORTEX. It shows what's been built, what's in progress, and what remains. - -## Status Legend - -- βœ… **Complete** β€” Fully implemented with test coverage -- 🟑 **Partial** β€” Core implementation exists but incomplete or lacking tests -- ❌ **Missing** β€” Not yet implemented -- πŸ”„ **In Progress** β€” Currently being developed - ---- - -## Module Status - -### Foundation Layer - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| Core Types | βœ… Complete | `core/types.ts` | All entity interfaces defined; includes `PageActivity`, `HotpathEntry`, `TierQuotas`, and `MetadataStore` hotpath method signatures | -| Model Profiles | βœ… Complete | `core/ModelProfile.ts`, `core/ModelDefaults.ts`, `core/ModelProfileResolver.ts`, `core/BuiltInModelProfiles.ts` | Source-of-truth for model-derived numerics; guard script enforces compliance | -| Numeric Constants | βœ… Complete | `core/NumericConstants.ts` | Runtime constants (byte sizes, workgroup limits) centralized | -| Crypto Helpers | βœ… Complete | `core/crypto/hash.ts`, `core/crypto/sign.ts`, `core/crypto/verify.ts` | SHA-256 hashing; Ed25519 sign/verify; 26 tests passing | -| Hotpath Policy | βœ… Complete | `core/HotpathPolicy.ts` | Williams Bound policy implementation; covered by `tests/HotpathPolicy.test.ts` | -| Salience Engine | βœ… Complete | `core/SalienceEngine.ts` | Per-node salience computation, promotion/eviction lifecycle helpers, community-aware admission logic; covered by `tests/SalienceEngine.test.ts` | - -**Foundation Status:** 6/6 complete (100%) - ---- - -### Storage Layer - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| Vector Store (OPFS) | βœ… Complete | `storage/OPFSVectorStore.ts` | Append-only binary vector file; byte-offset addressing; test coverage via `tests/Persistence.test.ts` | -| Vector Store (Memory) | βœ… Complete | `storage/MemoryVectorStore.ts` | In-memory implementation for testing | -| Metadata Store (IndexedDB) | βœ… Complete | `storage/IndexedDbMetadataStore.ts` | Full CRUD for all entities; reverse indexes; semantic neighbor graph operations (currently misnamed as "Metroid neighbor" β€” see TODO P0-X); dirty-volume flags; includes `hotpath_index` and `page_activity` object stores; hotpath CRUD methods are implemented and covered by `tests/Persistence.test.ts` | - -**Storage Status:** 3/3 complete (100%) - ---- - -### Vector Compute Layer - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| Backend Abstraction | βœ… Complete | `VectorBackend.ts`, `BackendKind.ts`, `CreateVectorBackend.ts` | Unified interface across all backends | -| WebGPU Backend | βœ… Complete | `WebGPUVectorBackend.ts`, `Vectors.wgsl` | Compute shader implementation | -| WebGL Backend | βœ… Complete | `WebGLVectorBackend.ts`, `Vectors.glsl` | Fragment shader implementation | -| WebNN Backend | βœ… Complete | `WebNNVectorBackend.ts` | ML accelerator path | -| WASM Backend | βœ… Complete | `WasmVectorBackend.ts`, `Vectors.wat` | Hand-written WebAssembly; guaranteed fallback | -| TopK Selection | βœ… Complete | `TopK.ts` | Utility for top-K similarity selection | - -**Vector Compute Status:** 6/6 complete (100%) - ---- - -### Embedding Layer - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| Embedding Interface | βœ… Complete | `embeddings/EmbeddingBackend.ts` | Abstract interface for all providers | -| Provider Resolver | βœ… Complete | `embeddings/ProviderResolver.ts` | Capability filtering + benchmark-based winner selection | -| Embedding Runner | βœ… Complete | `embeddings/EmbeddingRunner.ts` | High-level orchestrator with fallback chain | -| Dummy Provider | βœ… Complete | `embeddings/DeterministicDummyEmbeddingBackend.ts` | SHA-256-based deterministic embedder for testing | -| Transformers.js Provider | βœ… Complete | `embeddings/TransformersJsEmbeddingBackend.ts` | Real ONNX inference (`webnn`/`webgpu`/`wasm`); default model: `onnx-community/embeddinggemma-300m-ONNX` | -| ORT WebGL Provider | ❌ Missing | `embeddings/OrtWebglEmbeddingBackend.ts` (planned) | Explicit `webgl` fallback path not yet implemented | - -**Embedding Status:** 5/6 complete (83%) - ---- - -### Hippocampus (Ingest Orchestration) - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| Text Chunking | βœ… Complete | `hippocampus/Chunker.ts` | Token-aware sentence-boundary splitting respecting `ModelProfile.maxChunkTokens`; covered by `tests/hippocampus/Chunker.test.ts` | -| Page Builder | βœ… Complete | `hippocampus/PageBuilder.ts` | Builds signed `Page` entities with `contentHash`, `vectorHash`, `prevPageId`/`nextPageId` linkage; covered by `tests/hippocampus/PageBuilder.test.ts` | -| Ingest Orchestrator | 🟑 Partial | `hippocampus/Ingest.ts` | `ingestText()` implemented: chunk β†’ embed β†’ persist pages + PageActivity β†’ create Book β†’ run hotpath promotion sweep. **Missing:** hierarchy building (Volume/Shelf), semantic neighbor insertion. | -| Hierarchy Builder | ❌ Missing | `hippocampus/HierarchyBuilder.ts` (planned) | Construct/update Books, Volumes, Shelves; attempt tier-quota hotpath admission for each level's medoid/prototype; Williams-derived fanout bounds; trigger split via ClusterStability when bounds exceeded | -| Fast Semantic Neighbor Insert | ❌ Missing | `hippocampus/FastNeighborInsert.ts` (planned) | Cosine-nearest neighbors within Williams-cutoff distance (not fixed K). Degree overflow evicts lowest-cosine-similarity neighbor. Initial edges only at ingest; Daydreamer builds additional edges lazily. `SemanticNeighbor.cosineSimilarity` drives discovery + Bayesian updates; Hebbian weights (separate) drive TSP traversal. See DESIGN.md Β§Graph Structures for the full edge-role invariant. | - -**Hippocampus Status:** 2.5/5 complete (50%) - -**Critical Blocker:** Hierarchy builder and semantic neighbor insertion missing; ingest produces no graph structure beyond a single Book. - ---- - -### Cortex (Retrieval Orchestration) - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| Ranking Pipeline | ❌ Missing | `cortex/Ranking.ts` (planned) | Resident-first scoring cascade: HOT shelves β†’ HOT volumes β†’ HOT books β†’ HOT pages; spill to WARM/COLD only when coverage insufficient | -| MetroidBuilder | ❌ Missing | `cortex/MetroidBuilder.ts` (planned) | Constructs Metroid `{ m1, m2, c }` via Matryoshka dimensional unwinding; antithesis discovery; centroid computation; knowledge gap detection | -| Dialectical Search Pipeline | ❌ Missing | `cortex/DialecticalSearch.ts` (planned) | Orchestrates thesis/antithesis/synthesis zone exploration using a Metroid; prevents confirmation bias | -| Knowledge Gap Detector | ❌ Missing | `cortex/KnowledgeGapDetector.ts` (planned) | Determines when MetroidBuilder cannot find m2; emits curiosity probe | -| Seed Selection | ❌ Missing | `cortex/SeedSelection.ts` (planned) | Threshold-based top-k page selection from ranking output | -| Subgraph Expansion | 🟑 Partial | `storage/IndexedDbMetadataStore.ts` (`getInducedMetroidSubgraph` β€” to be renamed `getInducedNeighborSubgraph`) | BFS expansion implemented in storage layer; needs dynamic Williams bounds; needs orchestration wrapper | -| Open TSP Solver | ❌ Missing | `cortex/OpenTSPSolver.ts` (planned) | Dummy-node open-path heuristic for coherent ordering | -| Query Orchestrator | 🟑 Needs Rework | `cortex/Query.ts` | Flat top-K scoring implemented (hotpath-first β†’ warm/cold spill β†’ PageActivity update β†’ promotion sweep). **Must be substantially reworked** to implement the full dialectical pipeline: replace flat scoring with hierarchical resident-first ranking, add MetroidBuilder, dialectical zone scoring (thesis/antithesis/synthesis), subgraph expansion with dynamic Williams bounds, TSP coherence path, and query cost meter. The existing implementation does not use Hebbian edges or cosine-similarity-bounded subgraph expansion; it is a functional placeholder only. | -| Result DTO | 🟑 Needs Rework | `cortex/QueryResult.ts` | Minimal DTO (`pages`, `scores`, `metadata`). **Must be reworked** to add `coherencePath: Hash[]`, `metroid?: { m1, m2, centroid }`, `knowledgeGap?: KnowledgeGap`, and `provenance: { subgraphSize, hopCount, edgeWeights, vectorOpCost, earlyStop }`. | - -**Cortex Status:** 1.5/9 complete (17%) - -**Critical Blocker:** MetroidBuilder, dialectical search pipeline, and knowledge gap detector entirely absent. Existing `Query.ts` implements flat top-K retrieval only. - ---- - -### Daydreamer (Background Consolidation) - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| Idle Scheduler | ❌ Missing | `daydreamer/IdleScheduler.ts` (planned) | Cooperative background loop; interruptible; respects CPU budget | -| Hebbian Updates | ❌ Missing | `daydreamer/HebbianUpdater.ts` (planned) | LTP (strengthen), LTD (decay), prune below threshold; recompute Οƒ(v) for changed nodes; run promotion/eviction sweep | -| Prototype Recomputation | ❌ Missing | `daydreamer/PrototypeRecomputer.ts` (planned) | Recalculate volume/shelf medoids and centroids; recompute salience for affected entries; run tier-quota promotion/eviction | -| Full Neighbor Graph Recalc | ❌ Missing | `daydreamer/FullNeighborRecalc.ts` (planned) | Rebuild bounded neighbor lists for dirty volumes; batch size bounded by O(√(t log t)) per idle cycle; recompute salience after recalc. **Note:** Currently planned as `FullMetroidRecalc` β€” this is a naming error; see TODO P0-X. | -| Experience Replay | ❌ Missing | `daydreamer/ExperienceReplay.ts` (planned) | Simulate queries to reinforce connections | -| Cluster Stability | ❌ Missing | `daydreamer/ClusterStability.ts` (planned) | Detect/trigger split/merge for unstable clusters; run lightweight label propagation for community detection; store community labels in PageActivity | - -**Daydreamer Status:** 0/6 complete (0%) - -**Note:** Not a v1 blocker β€” system can ship without background consolidation (manual recalc only). Community detection is required before graph-community quota enforcement is active. - ---- - -### Policy & Configuration - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| Routing Policy | βœ… Complete | `Policy.ts` | Derives routing dimensions from ModelProfile; integration tested | -| Hotpath Policy | βœ… Complete | `core/HotpathPolicy.ts` | Williams Bound policy implementation, salience weights, tier quotas, community quotas; separate from model-derived numerics | - -**Policy Status:** 2/2 complete (100%) - ---- - -### Runtime Harness - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| Browser Harness | βœ… Complete | `runtime/harness/index.html`, `scripts/runtime-harness-server.mjs` | Localhost-served HTML harness for browser testing | -| Electron Wrapper | βœ… Complete | `scripts/electron-harness-main.mjs` | Thin Electron launcher for GPU-realism testing | -| Playwright Tests | βœ… Complete | `tests/runtime/browser-harness.spec.mjs`, `tests/runtime/electron-harness.spec.mjs` | Browser lane passes; Electron context-sensitive | -| Docker Debug Lane | βœ… Complete | `docker/electron-debug/*`, `docker-compose.electron-debug.yml` | Sandbox-isolated Electron debugging via VS Code attach | - -**Runtime Status:** 4/4 complete (100%) - ---- - -### Testing & Validation - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| Unit Tests | βœ… Complete | `tests/*.test.ts`, `tests/**/*.test.ts` | 115 tests across 13 files; all passing | -| Persistence Tests | βœ… Complete | `tests/Persistence.test.ts` | Full storage layer coverage (OPFS, IndexedDB, semantic neighbor graph β€” currently tested as "Metroid neighbors", hotpath indexes) | -| Model Tests | βœ… Complete | `tests/model/*.test.ts` | Profile resolution, defaults, routing policy | -| Embedding Tests | βœ… Complete | `tests/embeddings/*.test.ts` | Provider resolver, runner, real/dummy backends | -| Backend Smoke Tests | βœ… Complete | `tests/BackendSmoke.test.ts` | All vector backends instantiate cleanly | -| Runtime Tests | βœ… Complete | `tests/runtime/*.spec.mjs` | Browser harness validated; Electron context-sensitive | -| Integration Tests | βœ… Complete | `tests/integration/IngestQuery.test.ts` | End-to-end: ingest β†’ persist β†’ query β†’ verify results; persistence across sessions | -| Hotpath Policy Tests | βœ… Complete | `tests/HotpathPolicy.test.ts` | H(t) sublinearity and monotonicity; tier quota sums; community quota minimums; salience determinism | -| Salience Engine Tests | βœ… Complete | `tests/SalienceEngine.test.ts` | Bootstrap fills to H(t); steady-state eviction; community/tier quota enforcement; determinism | -| Scaling Benchmarks | ❌ Missing | `tests/benchmarks/HotpathScaling.bench.ts` (planned) | Synthetic graphs at 1K/10K/100K/1M; assert resident count ≀ H(t); query cost sublinear | -| Benchmarks | 🟑 Partial | `tests/benchmarks/DummyEmbedderHotpath.bench.ts` | Baseline dummy embedder benchmark; real-provider and hotpath scaling benchmarks needed | - -**Testing Status:** 9/12 complete (75%) - ---- - -### Build & CI - -| Module | Status | Files | Notes | -|--------|--------|-------|-------| -| TypeScript Config | βœ… Complete | `tsconfig.json` | Strict mode, ES2022 target, ESNext modules | -| Build Script | βœ… Complete | `package.json` (`build` script) | Type-check via `tsc --noEmit` | -| Lint Config | βœ… Complete | `eslint.config.mjs` | TypeScript-ESLint rules | -| Model-Derived Guard | βœ… Complete | `scripts/guard-model-derived.mjs` | Scans for hardcoded model numerics; enforces source-of-truth | -| Test Runner | βœ… Complete | `package.json` (Vitest scripts) | Unit, browser, electron, runtime, benchmark targets | -| CI Pipeline | 🟑 Partial | `.github/workflows/*` (if exists) | Needs verification; not examined in detail | -| GitHub Issue Sync | βœ… Complete | `scripts/sync-github-project.mjs`, `.github/workflows/sync-github-project.yml` | Syncs TODO.md β†’ GitHub issues/milestones; smoke test via TODO task | - -**Build Status:** 5/6 complete (83%) - ---- - -## Overall Progress Summary - -| Layer | Completion | Critical Gap | -|-------|-----------|--------------| -| Foundation | 100% | β€” | -| Storage | 100% | β€” | -| Vector Compute | 100% | β€” | -| Embedding | 83% | WebGL provider (low priority) | -| Hippocampus | 50% | Chunker + PageBuilder + minimal Ingest done; hierarchy builder and semantic neighbor insertion missing | -| Cortex | 17% | Minimal Query + QueryResult done; MetroidBuilder, dialectical search, knowledge gap detection all missing | -| Daydreamer | 0% | Not v1 blocker | -| Policy | 100% | β€” | -| Runtime | 100% | β€” | -| Testing | 67% | Integration tests, scaling benchmarks | -| Build/CI | 83% | β€” | - -**System-Wide Completion:** ~75% (core infrastructure, policy foundation, chunking, page building, and minimal ingest/query implemented; hierarchy builder, MetroidBuilder, and graph coherence remain.) - ---- - -## What Works Today - -- βœ… Store/retrieve vectors and metadata -- βœ… Vector similarity operations on all backends -- βœ… Generate real embeddings via Transformers.js -- βœ… Resolve model profiles and derive routing policies (including `matryoshkaProtectedDim` for Matryoshka models) -- βœ… Run browser/Electron runtime harness -- βœ… Pass 115 unit tests -- βœ… Hash text/binary content (SHA-256) and sign/verify Ed25519 signatures -- βœ… Chunk text and build signed `Page` entities -- βœ… Ingest text (minimal): chunk β†’ embed β†’ persist pages + PageActivity β†’ create Book β†’ hotpath promotion - -## What Doesn't Work Today - -- ❌ **No hierarchy beyond single Book** β€” Volume/Shelf hierarchy builder not yet implemented -- ❌ **No semantic neighbor graph** β€” `FastNeighborInsert` not yet implemented; subgraph expansion has no edges -- ❌ **No dialectical retrieval** β€” `MetroidBuilder`, `KnowledgeGapDetector`, and dialectical pipeline not yet implemented; current `Query.ts` is flat top-K retrieval only -- ❌ **No coherent path ordering** β€” No TSP solver; results are ranked list, not narrative chain -- ❌ **Cannot consolidate** β€” No Daydreamer loop -- ❌ **Cannot share discovery updates safely** β€” No P2P curiosity broadcasting or privacy-filtered exchange - ---- - -## Recommended Implementation Order - -### Phase 1: Unblock Basic Functionality (Ship v0.1) - -**Goal:** Enable ingest and retrieval for a single user session, with Williams Bound policy foundation in place. - -1. **Crypto Helpers** (`core/crypto/*`) βœ… **Complete** - - SHA-256 hashing for text and binary - - Ed25519 signing/verification - - 26 tests passing - -2. **Williams Bound Policy Foundation** βœ… **Complete** - - `core/HotpathPolicy.ts`, `core/SalienceEngine.ts`, `core/types.ts` extensions, `storage/IndexedDbMetadataStore.ts` hotpath stores - -3. **Text Chunking** (`hippocampus/Chunker.ts`) βœ… **Complete** - - Token-aware sentence-boundary splitting; tests passing - -4. **Page Builder** (`hippocampus/PageBuilder.ts`) βœ… **Complete** - - Signed Page entities with hash linkage; tests passing - -5. **Hippocampus Ingest** (`hippocampus/Ingest.ts`) 🟑 **Partial** - - Minimal `ingestText()` implemented (chunk β†’ embed β†’ persist pages β†’ single Book β†’ hotpath admission) - - **Remaining:** semantic neighbor insertion (deferred to Phase 2) - -6. **Cortex Query** (`cortex/Query.ts`) 🟑 **Partial** - - Minimal `query()` implemented (hotpath-first flat scoring; warm/cold spill) - - **Remaining:** MetroidBuilder, dialectical pipeline (deferred to Phase 2) - -7. **Integration Test** (`tests/integration/IngestQuery.test.ts`) βœ… **Complete** - - Ingest text β†’ Retrieve by query β†’ Validate results; persistence across sessions - -**Exit Criteria:** User can ingest text and retrieve relevant pages by query; Williams Bound policy is in place. - ---- - -### Phase 2: Add Hierarchy, Dialectical Search & Resident-First Routing (Ship v0.5) - -**Goal:** Hierarchical routing, MetroidBuilder, dialectical search pipeline, coherent path ordering, and fully resident-first query path. - -1. **Hierarchy Builder** (`hippocampus/HierarchyBuilder.ts`) - - Cluster pages into Books (medoid selection) - - Cluster books into Volumes (prototype computation) - - Build Shelves for coarse routing - - Attempt tier-quota hotpath admission for each level's medoid/prototype via `SalienceEngine` - - Williams-derived fanout bounds; trigger split via `ClusterStability` when exceeded - -2. **MetroidBuilder** (`cortex/MetroidBuilder.ts`) - - Select m1 (topic medoid) for a given query embedding - - Freeze protected Matryoshka dimensions - - Search for m2 (antithesis medoid) within unfrozen dimensions - - Compute centroid `c = (m1 + m2) / 2` - - Unwind Matryoshka layers progressively, repeating antithesis search - - Return `Metroid { m1, m2, c }` or signal knowledge gap - -3. **Knowledge Gap Detector** (`cortex/KnowledgeGapDetector.ts`) - - Evaluate MetroidBuilder result - - Emit `KnowledgeGap` DTO with dimensional boundary info - - Trigger P2P curiosity probe emission - -4. **Ranking Pipeline** (`cortex/Ranking.ts`) - - Resident-first cascade: HOT shelves β†’ HOT volumes β†’ HOT books β†’ HOT pages - - Spill to WARM/COLD only when resident coverage insufficient - -5. **Open TSP Solver** (`cortex/OpenTSPSolver.ts`) - - Dummy-node open-path heuristic - - Test on synthetic graphs - -6. **Full Query Orchestrator** (`cortex/Query.ts` β€” upgrade) - - Embed query β†’ select m1 β†’ build Metroid β†’ dialectical scoring cascade - - Dynamic subgraph expansion bounds from `HotpathPolicy` - - Query cost meter; early-stop on budget exceeded - - Coherent path via TSP - - Rich result DTO with provenance and knowledge gap flag - -**Exit Criteria:** User gets epistemically balanced context chains via MetroidBuilder and dialectical search; knowledge gaps are detected; query latency controlled by H(t). - ---- - -### Phase 3: Background Consolidation, Community Quotas & Smart Sharing (Ship v1.0) - -**Goal:** Idle maintenance keeps memory healthy, community-aware hotpath coverage stays diverse, and privacy-safe interest sharing is available. - -1. **Idle Scheduler** (`daydreamer/IdleScheduler.ts`) - - Cooperative, interruptible loop - - CPU budget awareness - -2. **Hebbian Updater** (`daydreamer/HebbianUpdater.ts`) - - LTP/LTD rules; edge pruning - - Recompute Οƒ(v) for changed nodes; run promotion/eviction sweep - -3. **Full Neighbor Graph Recalc** (`daydreamer/FullNeighborRecalc.ts`) - - Rebuild neighbor lists for dirty volumes - - O(√(t log t)) batch size per idle cycle - -4. **Prototype Recomputer** (`daydreamer/PrototypeRecomputer.ts`) - - Update volume/shelf prototypes - - Tier-quota promotion/eviction after recomputation - -5. **Community Detection** (`daydreamer/ClusterStability.ts` β€” extend) - - Label propagation on semantic neighbor graph - - Store community labels in `PageActivity.communityId` - - Wire community IDs into `SalienceEngine` promotion/eviction - -6. **Smart Interest Sharing** (`sharing/*` planned) - - `sharing/EligibilityClassifier.ts` β€” classify candidate nodes for share eligibility; block identity/PII-bearing nodes - - `sharing/SubgraphExporter.ts` β€” export signed, topic-scoped graph slices from eligible nodes only - - `sharing/SubgraphImporter.ts` β€” verify signatures/provenance and merge imported slices into local discovery index - - `sharing/PeerExchange.ts` β€” opt-in peer transport for exchanging eligible graph slices - -**Exit Criteria:** System self-maintains over extended use; community-aware hotpath quotas enforced; privacy-safe smart sharing works end-to-end. - ---- - -### Phase 4: Polish & Release Prep (Ship v1.0 Final) - -**Goal:** Production-ready quality. - -1. **Integration Test Suite** (`tests/integration/*`) - - Full vertical slice coverage - - Edge cases and error paths - - Performance regression tests - -2. **Scaling Benchmark Suite** (`tests/benchmarks/HotpathScaling.bench.ts`) - - Synthetic graphs at 1K, 10K, 100K, 1M nodes+edges - - Assert: resident count never exceeds H(t); query cost scales sublinearly - - Record baselines in `benchmarks/BASELINES.md` - -3. **Documentation** (`docs/*`) - - API reference - - Integration guide - - Troubleshooting - -4. **CI Hardening** - - Electron runtime gate policy - - Guard scripts in merge checks (model-derived + hotpath policy) - - Benchmark baselines - -5. **Product Surface UX Contract** - - Define standalone browser-extension UX baseline: - - Passive capture of visited pages into the local ingest queue - - Search-first recall UI over pages the user has actually seen - - Lightweight metrics panel that supports, but does not dominate, retrieval UX - - Define model-mode UX contract for the standalone app: - - Nomic mode = multimodal retrieval (text + images in shared latent space) - - Gemma mode = high-precision text retrieval (no image embedding) - - Capability messaging in UI so users understand image-recall availability by mode - - Define library boundary contract: - - Keep extension shell concerns outside core ingest/query APIs - - Keep library docs and examples headless/integration-first - - Add acceptance checks for rabbit-hole recall UX: - - Vague text recollection recovers previously visited pages - - Vague visual recollection recovers previously seen images when multimodal mode is enabled - -**Exit Criteria:** All tests pass; benchmarks recorded; docs complete; product-surface UX contract documented; ready for public use. - ---- - -## Known Blockers & Risks - -### Blocker 1: No Hierarchy Builder or Semantic Neighbor Graph -**Impact:** Ingest produces only a single flat Book; no Volume/Shelf structure; subgraph expansion has no edges to traverse. -**Mitigation:** Phase 2 priority; `HierarchyBuilder` and `FastNeighborInsert` must be implemented before dialectical retrieval is possible. - -### Blocker 2: No MetroidBuilder or Dialectical Pipeline -**Impact:** Queries return flat top-K results only; no epistemic balance, no knowledge gap detection, no P2P curiosity. -**Mitigation:** Phase 2 priority; depends on semantic neighbor graph (Blocker 1) and hierarchy builder. - -### Blocker 3: No Privacy-Safe Sharing or Curiosity Broadcasting Pipeline -**Impact:** Core discovery-sharing value proposition is missing; knowledge gaps cannot be resolved via P2P. -**Mitigation:** Phase 3 required track; implement eligibility classifier + curiosity broadcaster + signed subgraph exchange as v1 scope. CuriosityProbe must include `mimeType` and `modelUrn` to prevent incommensurable graph merges. - -### Blocker 4: Naming Drift (P0-X) -**Impact:** The term "Metroid" is currently used for the proximity graph in all code. MetroidBuilder cannot be introduced without a rename collision. -**Mitigation:** P0-X tasks (rename `MetroidNeighbor` β†’ `SemanticNeighbor`, etc.) must be completed before MetroidBuilder is implemented. - -### Risk 1: TSP Complexity -Open TSP is NP-hard; heuristic may be slow on large subgraphs. -**Mitigation:** Dynamic Williams-derived subgraph bounds shrink the problem as graph grows; defer to Phase 2; use deterministic greedy heuristic. - -### Risk 2: Electron Runtime Stability -Host-shell Electron can SIGSEGV in constrained contexts. -**Mitigation:** Docker attach lane validated for debugging; document GPU requirements for production. - -### Risk 3: WebGL Provider Gap -Transformers.js doesn't expose `webgl` device directly. -**Mitigation:** Low priority; `webgpu` and `wasm` sufficient for most users; explicit ORT adapter deferred to Phase 4. - -### Risk 4: Empirical Calibration of `c` -The Williams Bound scaling constant `c` is not theorem-given; wrong value causes either hotpath over-allocation (wastes RAM) or under-allocation (defeats purpose). -**Mitigation:** Default `c = 0.5` is conservative; scaling benchmarks in Phase 4 will validate and tune. Keep `c` in `core/HotpathPolicy.ts` as an overrideable policy constant. - ---- - -## Development Workflow - -### Command Reference - -```bash -# Install dependencies -npm ci - -# Type-check -npm run build - -# Lint -npm run lint - -# Run all unit tests -npm run test:unit - -# Run specific test file -npm run test:unit -- tests/model/ModelProfileResolver.test.ts - -# Model-derived numeric guard -npm run guard:model-derived - -# Run benchmarks -npm run benchmark - -# Start dev harness -npm run dev:harness - -# Run browser runtime tests -npm run test:browser - -# Run Electron tests (context-sensitive) -npm run test:electron - -# Docker Electron debug lane -npm run docker:electron:up -npm run docker:electron:down - -# Run all tests -npm run test:all -``` - -### Pre-Commit Checklist - -1. βœ… `npm run build` passes -2. βœ… `npm run lint` passes -3. βœ… `npm run test:unit` passes -4. βœ… `npm run guard:model-derived` passes -5. βœ… No hardcoded model-derived numerics outside `core/BuiltInModelProfiles.ts` -6. βœ… Tests added/updated for new functionality - -### Documentation Sync Protocol - -After every implementation pass: -1. Update `PLAN.md` module status -2. Update `TODO.md` to reflect completed items -3. Update `README.md` if user-facing changes - ---- - -## Notes - -- **Metroid vs medoid vs semantic neighbor graph:** These are three distinct concepts. `Metroid` refers only to the dialectical search probe `{ m1, m2, c }` constructed by `MetroidBuilder` at query time. `medoid` refers to a cluster representative node. The sparse proximity/neighbor graph (used for BFS subgraph expansion) is the **semantic neighbor graph** β€” it is currently misnamed `MetroidNeighbor`/`MetroidSubgraph` in code (see TODO P0-X for the rename task). -- **Model-derived numerics:** Never hardcode; always source from `core/` model profile modules. -- **Policy-derived constants:** Never hardcode; always source from `core/HotpathPolicy.ts`. -- **Test philosophy:** TDD (Red β†’ Green β†’ Refactor) for all new slices. -- **Runtime realism:** Browser and Electron lanes are required merge gates. -- **Williams Bound invariant:** The resident hotpath count must never exceed H(t). Enforce in tests and assert in benchmarks. diff --git a/README.md b/README.md index 810472c..0d33e20 100644 --- a/README.md +++ b/README.md @@ -114,8 +114,16 @@ bun run dev:harness # start the browser runtime harness at http://127.0.0.1:4173 | Document | Purpose | |---|---| | [`DESIGN.md`](DESIGN.md) | Architecture specification and core design principles | -| [`PLAN.md`](PLAN.md) | Module-by-module implementation status and development phases | -| [`TODO.md`](TODO.md) | Prioritized actionable tasks to ship v1.0 | | [`ARCHITECTURE-REVIEW.md`](ARCHITECTURE-REVIEW.md) | Repository-wide architectural drift report and correction tasks | | [`docs/api.md`](docs/api.md) | API reference for developers integrating with CORTEX | | [`docs/development.md`](docs/development.md) | Build, test, debug, and Docker workflow | + +### Project Management + +Task tracking, prioritization, and sprint planning use **GitHub-native features**: + +- **[Issues](../../issues)** β€” Every task, bug, and feature request. Use the structured templates. +- **[Projects](../../projects)** β€” Kanban boards for lifecycle tracking. +- **Milestones** β€” Group issues by release phase (`v0.1`, `v0.5`, `v1.0`). +- **Labels** β€” Auto-applied on PRs based on changed files. Priority (`P0`–`P3`) and layer labels for classification. +- **`gh` CLI** β€” Agents create, update, and close issues directly via `gh issue create`, `gh issue close`, etc. diff --git a/TODO.md b/TODO.md deleted file mode 100644 index bc21997..0000000 --- a/TODO.md +++ /dev/null @@ -1,892 +0,0 @@ -# CORTEX TODO β€” Path to v1.0 - -**Last Updated:** 2026-03-13 - -This document contains a prioritized, actionable list of specific tasks required to ship CORTEX v1.0. Items are ordered by dependency: highest-priority items are those blocking other work. - ---- - -## 🚨 Critical Path β€” Ship v0.1 (Minimal Viable) - -These items **must** be completed to have a usable system. Without them, users cannot ingest or query memories. - -### P0-A: Crypto Helpers (BLOCKS: all signed entity creation) βœ… COMPLETE - -**Why:** Pages require `contentHash`, `vectorHash`, and `signature`. Cannot create valid pages without crypto. - -- [x] **P0-A1:** Implement `core/crypto/hash.ts` - - SHA-256 for text content - - SHA-256 for binary vector data - - Test with known vectors - -- [x] **P0-A2:** Implement `core/crypto/sign.ts` - - Ed25519 key pair generation - - Sign canonical page representation - - Test with example pages - -- [x] **P0-A3:** Implement `core/crypto/verify.ts` - - Verify signature against public key - - Test reject invalid signatures - -- [x] **P0-A4:** Add crypto test coverage - - `tests/crypto/hash.test.ts` - - `tests/crypto/sign.test.ts` - - `tests/crypto/verify.test.ts` - -**Exit Criteria:** Can hash content, sign pages, verify signatures. βœ… Met β€” 26 tests passing. - ---- - -### P0-F: Williams Bound Policy Foundation (BLOCKS: all hotpath-aware modules) βœ… COMPLETE - -**Why:** The HotpathPolicy and SalienceEngine are the central source of truth for the Williams Bound architecture. Every subsequent module (ingest, query, hierarchy, Daydreamer) depends on them. Implementing these first ensures the bound is enforced from day one rather than retrofitted. - -- [x] **P0-F1:** Implement `core/HotpathPolicy.ts` - - `computeCapacity(graphMass: number): number` β€” H(t) = ⌈c Β· √(t Β· logβ‚‚(1+t))βŒ‰ - - `computeSalience(hebbianIn: number, recency: number, queryHits: number, weights?: SalienceWeights): number` β€” Οƒ = Ξ±Β·H_in + Ξ²Β·R + Ξ³Β·Q - - `deriveTierQuotas(capacity: number, quotaRatios?: TierQuotaRatios): TierQuotas` β€” allocate H(t) across shelf/volume/book/page tiers - - `deriveCommunityQuotas(tierBudget: number, communitySizes: number[]): number[]` β€” proportional with min(1) guarantee - - Export a frozen `DEFAULT_HOTPATH_POLICY` object containing all constants: `c = 0.5`, `Ξ± = 0.5`, `Ξ² = 0.3`, `Ξ³ = 0.2`, `q_s = 0.10`, `q_v = 0.20`, `q_b = 0.20`, `q_p = 0.50` - - Keep strictly separate from `core/ModelDefaults.ts` (policy-derived β‰  model-derived) - -- [x] **P0-F2:** Add HotpathPolicy test coverage (`tests/HotpathPolicy.test.ts`) - - H(t) grows sublinearly: verify `H(10_000) / 10_000 < H(1_000) / 1_000` - - H(t) is monotonically non-decreasing over a representative range: verify `H(t+1) >= H(t)` for each `t` in `[0, 1, 2, 10, 100, 1_000, 10_000, 100_000]` - - H(t) is a finite integer β‰₯ 1 for edge inputs: `t = 0`, `t = 1`, `t = Number.MAX_SAFE_INTEGER`; result must never be `NaN`, `Infinity`, or `< 1` - - Derived tier-quota *counts* sum exactly to capacity: `deriveTierQuotas(cap).shelf + .volume + .book + .page === cap` for `cap` in `[1, 10, 100, 1_000]` - - Community quota counts sum exactly to `tier_budget`: `sum(deriveCommunityQuotas(budget, sizes)) === budget` for representative `(budget, sizes)` inputs including edge cases (`budget = 0`, empty `sizes` array, `budget < sizes.length`) - - Community quotas never produce `NaN`, `Infinity`, or negative values for any valid input, including `sizes` with a single community or all equal sizes - - Salience is deterministic for same inputs - - Salience clamps output to a finite number: never `NaN` or `Infinity` for extreme weight or hit-count values - -- [x] **P0-F3:** Extend `core/types.ts` - - Add `PageActivity` interface: `{ pageId: Hash; queryHitCount: number; lastQueryAt: string; communityId?: string }` - - Add `HotpathEntry` interface: `{ entityId: Hash; tier: 'shelf' | 'volume' | 'book' | 'page'; salience: number; communityId?: string }` - - Add `TierQuotas` type: `{ shelf: number; volume: number; book: number; page: number }` - - Add hotpath method signatures to `MetadataStore` interface: - - `putHotpathEntry(entry: HotpathEntry): Promise` - - `getHotpathEntries(tier?: HotpathEntry['tier']): Promise` - - `evictWeakest(tier: HotpathEntry['tier'], communityId?: string): Promise` - - `getResidentCount(): Promise` - - `putPageActivity(activity: PageActivity): Promise` - - `getPageActivity(pageId: Hash): Promise` - -- [x] **P0-F4:** Extend `storage/IndexedDbMetadataStore.ts` - - Add `hotpath_index` object store keyed by `entityId`; secondary index by `tier` - - Add `page_activity` object store keyed by `pageId` - - Implement all six new `MetadataStore` hotpath methods - - Extend `tests/Persistence.test.ts` with hotpath store tests: - - put/get/evict cycle for `HotpathEntry` - - put/get for `PageActivity` - - `getResidentCount` returns correct value after multiple puts - -**Exit Criteria:** `HotpathPolicy` module passes all tests; `types.ts` has hotpath interfaces; IndexedDB hotpath stores are implemented and tested. βœ… Met β€” tests passing. - ---- - -### P0-G: Salience Engine (BLOCKS: hotpath promotion in ingest and Daydreamer) βœ… COMPLETE - -**Why:** The SalienceEngine is the decision-making layer for hotpath admission. It is needed by ingest (new page admission), query (hit-count update), and Daydreamer (post-LTP/LTD sweeps). Implementing it before ingest ensures promotion logic is correct from the first page written. - -- [x] **P0-G1:** Implement `core/SalienceEngine.ts` - - `computeNodeSalience(pageId: Hash, metadataStore: MetadataStore): Promise` β€” fetch PageActivity and incident Hebbian edges; apply Οƒ formula via HotpathPolicy - - `batchComputeSalience(pageIds: Hash[], metadataStore: MetadataStore): Promise>` β€” efficient batch version - - `shouldPromote(candidateSalience: number, weakestResidentSalience: number, capacityRemaining: number): boolean` β€” admission gating - - `selectEvictionTarget(tier: HotpathEntry['tier'], communityId: string | undefined, metadataStore: MetadataStore): Promise` β€” find weakest resident in tier/community bucket - -- [x] **P0-G2:** Implement promotion/eviction lifecycle helpers in `core/SalienceEngine.ts` - - `bootstrapHotpath(metadataStore: MetadataStore, policy: HotpathPolicy): Promise` β€” fill hotpath greedily by salience while resident count < H(t) - - `runPromotionSweep(candidateIds: Hash[], metadataStore: MetadataStore, policy: HotpathPolicy): Promise` β€” steady-state: promote if salience > weakest in same tier/community bucket; evict weakest on promotion - -- [x] **P0-G3:** Add SalienceEngine test coverage (`tests/SalienceEngine.test.ts`) - - Bootstrap fills hotpath to exactly H(t) given enough candidates - - Steady-state promotes only when candidate beats the weakest resident - - Steady-state evicts exactly the weakest resident (not a random entry) - - Community quotas prevent a single community from consuming all page-tier slots - - Tier quotas prevent one hierarchy level from dominating - - Eviction is deterministic under the same state - -**Exit Criteria:** `SalienceEngine` module passes all tests; promotion/eviction lifecycle is correct and deterministic. βœ… Met β€” tests passing. - ---- - -### P0-B: Text Chunking (BLOCKS: ingest orchestration) - -**Why:** Must split text into page-sized chunks respecting ModelProfile token limits. - -- [x] **P0-B1:** Implement `hippocampus/Chunker.ts` - - Token-aware splitting (use ModelProfile `maxContextLength`) - - Respect sentence boundaries where possible - - Handle edge cases (empty input, single-token input, huge paragraphs) - -- [x] **P0-B2:** Add chunker test coverage - - `tests/hippocampus/Chunker.test.ts` - - Test various text lengths (short, medium, long, huge) - - Test boundary conditions - -**Exit Criteria:** Can reliably split arbitrary text into page chunks. - ---- - -### P0-C: Hippocampus Ingest (Minimal) (BLOCKS: user workflow) - -**Why:** Users need to add memories to the system. - -- [x] **P0-C1:** Implement `hippocampus/PageBuilder.ts` - - Create `Page` entities from text chunks - - Generate `pageId`, `contentHash`, `vectorHash` - - Sign with provided key pair - - Link pages via `prevPageId`/`nextPageId` - - Initialise `PageActivity` records with zero counts - -- [x] **P0-C2:** Implement `hippocampus/Ingest.ts` (minimal version) - - Entry point: `ingestText(text, { modelProfile, embeddingRunner, vectorStore, metadataStore, keyPair, ... })` - - Chunk text via `Chunker` - - Batch embed chunks via `EmbeddingRunner` - - Persist vectors to `VectorStore` - - Build pages via `PageBuilder`; persist pages and `PageActivity` to `MetadataStore` - - Build single `Book` containing all pages (medoid = first page for now) - - After persisting pages, check each new page for hotpath admission via `SalienceEngine.runPromotionSweep` - - **Defer:** Volume/Shelf hierarchy, fast neighbor insert - -- [ ] **P0-C3:** Add ingest test coverage - - `tests/hippocampus/Ingest.test.ts` - - Test happy path (text β†’ pages β†’ book) - - Test persistence (can retrieve pages after ingest) - - Test that new pages are considered for hotpath admission after ingest - -**Exit Criteria:** User can call `ingestText(...)` and pages are persisted; PageActivity records exist; hotpath admission runs. - ---- - -### P0-D: Cortex Query (Minimal) (BLOCKS: user workflow) - -**Why:** Users need to retrieve relevant memories. - -- [x] **P0-D1:** Implement `cortex/Query.ts` (minimal version) - - Entry point: `query(queryText, modelProfile, vectorStore, metadataStore, topK)` - - Embed query via `EmbeddingRunner` - - Score resident hotpath entries first (HOT pages); fall back to full scan for WARM/COLD - - Compute similarities via `VectorBackend` - - Select top-K pages; increment `queryHitCount` in `PageActivity`; recompute salience; run promotion sweep - - Return `QueryResult` with page IDs and scores - - **Defer:** Full hierarchical ranking, subgraph expansion, TSP coherence, query cost meter - -- [x] **P0-D2:** Implement `cortex/QueryResult.ts` - - DTO with `pages: Page[]`, `scores: number[]`, `metadata: object` - -- [x] **P0-D3:** Add query test coverage - - `tests/cortex/Query.test.ts` - - Test happy path (query β†’ top-K pages) - - Test empty corpus (no results) - - Test relevance (query for known content returns expected pages) - - Test `PageActivity.queryHitCount` incremented after query hit - -**Exit Criteria:** User can call `query(...)` and get ranked pages; query hits update PageActivity and trigger salience recomputation. - ---- - -### P0-E: End-to-End Integration (VALIDATES: v0.1 completeness) - -**Why:** Prove the system works soup-to-nuts. - -- [x] **P0-E1:** Implement `tests/integration/IngestQuery.test.ts` - - Ingest sample text corpus (e.g., Wikipedia articles) - - Query for specific topics - - Verify expected pages returned - - Verify persistence (restart session, query again) - -- [x] **P0-E2:** Run integration test in browser harness - - Ensure real IndexedDB and OPFS work correctly - - Verify WebGPU/WebGL/WASM backends function - -**Exit Criteria:** Integration test passes; system is minimally usable. - ---- - -### P0-X: Fix Architectural Naming Drift (BLOCKS: correct design implementation) - -**Why:** The codebase uses the term "Metroid" to name the sparse proximity/neighbor graph (`MetroidNeighbor`, `MetroidSubgraph`, `metroid_neighbors`, `getInducedMetroidSubgraph`, `FastMetroidInsert`, `FullMetroidRecalc`). This is architecturally incorrect. In CORTEX, a **Metroid** is a structured dialectical search probe `{ m1, m2, c }` β€” a concept that does not yet exist in the codebase at all. The proximity graph has nothing to do with Metroids. This naming collision will cause permanent confusion and make the MetroidBuilder impossible to implement cleanly without a rename. - -- [ ] **P0-X1:** Rename `MetroidNeighbor` β†’ `SemanticNeighbor` in `core/types.ts` - - Update all references in `storage/IndexedDbMetadataStore.ts` - - Update all references in test files - - Update JSDoc and inline comments - -- [ ] **P0-X2:** Rename `MetroidSubgraph` β†’ `SemanticNeighborSubgraph` in `core/types.ts` - - Update all references in `storage/IndexedDbMetadataStore.ts` - - Update all references in `cortex/Query.ts` - - Update JSDoc and inline comments - -- [ ] **P0-X3:** Rename `MetadataStore` proximity graph methods: - - `putMetroidNeighbors` β†’ `putSemanticNeighbors` - - `getMetroidNeighbors` β†’ `getSemanticNeighbors` - - `getInducedMetroidSubgraph` β†’ `getInducedNeighborSubgraph` - - `needsMetroidRecalc` β†’ `needsNeighborRecalc` - - `flagVolumeForMetroidRecalc` β†’ `flagVolumeForNeighborRecalc` - - `clearMetroidRecalcFlag` β†’ `clearNeighborRecalcFlag` - - Update all callers in `storage/IndexedDbMetadataStore.ts`, `cortex/Query.ts`, and test files - -- [ ] **P0-X4:** Rename planned Hippocampus file `hippocampus/FastMetroidInsert.ts` β†’ `hippocampus/FastNeighborInsert.ts` - - Rename class/function to `FastNeighborInsert`/`insertSemanticNeighbors` - -- [ ] **P0-X5:** Rename planned Daydreamer file `daydreamer/FullMetroidRecalc.ts` β†’ `daydreamer/FullNeighborRecalc.ts` - - Rename class/function to `FullNeighborRecalc`/`runNeighborRecalc` - -- [ ] **P0-X6:** Rename IndexedDB object store from `metroid_neighbors` β†’ `neighbor_graph` - - Increment `DB_VERSION` in `storage/IndexedDbMetadataStore.ts` - - Add migration in `applyUpgrade` to copy data from old store to new store - -- [ ] **P0-X7:** Update all documentation strings and JSDoc that use "Metroid neighbor" to use "semantic neighbor" - -**Exit Criteria:** No source file uses "Metroid" to refer to the proximity graph. The term "Metroid" is reserved exclusively for the `{ m1, m2, c }` dialectical probe type implemented in `cortex/MetroidBuilder.ts`. - ---- - -## 🟑 High Priority β€” Ship v0.5 (Hierarchical + Coherent) - -These items add hierarchical routing and coherent path ordering. They transform CORTEX from a flat vector search into a biologically-inspired memory system. - -### P1-A: Hierarchy Builder (UNBLOCKS: hierarchical routing) - -**Why:** Need Volume and Shelf structures for efficient coarse-to-fine routing. Tier-quota hotpath admission must be integrated so hierarchy prototypes enter the resident index from the moment they are created. - -- [ ] **P1-A1:** Implement `hippocampus/HierarchyBuilder.ts` - - Cluster pages into Books (K-means or similar; select medoid) - - Cluster books into Volumes (compute prototype vectors) - - Cluster volumes into Shelves (coarse routing prototypes) - - Persist prototypes to `VectorStore`; update metadata in `MetadataStore` - - After each level: attempt hotpath admission via `SalienceEngine.runPromotionSweep`: - - Book medoid β†’ page-tier quota - - Volume prototypes β†’ volume-tier quota - - Shelf routing prototypes β†’ shelf-tier quota - - Enforce Williams-derived fanout bounds (see `HotpathPolicy`); when exceeded, trigger split via `ClusterStability` - -- [ ] **P1-A2:** Upgrade `hippocampus/Ingest.ts` - - After persisting pages, call `HierarchyBuilder` - - Maintain hierarchy incrementally (append to existing structures) - -- [ ] **P1-A3:** Add hierarchy test coverage - - `tests/hippocampus/HierarchyBuilder.test.ts` - - Test clustering produces valid Books/Volumes/Shelves - - Test prototypes are valid vectors - - Test that hierarchy medoids/prototypes are admitted to correct tier quota - - Test fanout bounds respected; split triggered when exceeded - -**Exit Criteria:** Ingestion produces full Page β†’ Book β†’ Volume β†’ Shelf hierarchy with tier-quota hotpath admission at every level. - ---- - -### P1-B: Ranking Pipeline (UNBLOCKS: efficient queries) - -**Why:** Hierarchical ranking avoids scanning all pages; reduces query latency. The resident hotpath is the primary lookup target β€” WARM/COLD spill happens only when the hot set provides insufficient coverage. - -- [ ] **P1-B1:** Implement `cortex/Ranking.ts` - - `rankShelves(queryEmbedding, residentShelves, topK)` β€” score HOT shelf prototypes first - - `rankVolumes(queryEmbedding, residentVolumes, topK)` β€” score HOT volume prototypes within top shelves - - `rankBooks(queryEmbedding, residentBooks, topK)` β€” score HOT book medoids within top volumes - - `rankPages(queryEmbedding, residentPages, topK)` β€” score HOT page representatives within top books - - `spillToWarm(tier, queryEmbedding, metadataStore, topK)` β€” spill to IndexedDB lookup when resident set insufficient - - Each step narrows the search space; H(t) is the primary latency lever - -- [ ] **P1-B2:** Upgrade `cortex/Query.ts` - - Replace flat search with resident-first hierarchical ranking cascade - - HOT shelves β†’ HOT volumes β†’ HOT books β†’ HOT pages β†’ WARM/COLD spill - -- [ ] **P1-B3:** Add ranking test coverage - - `tests/cortex/Ranking.test.ts` - - Test each ranking function independently - - Test full cascade produces correct top pages - - Test that resident entries are scored before non-resident entries - -**Exit Criteria:** Queries use resident-first hierarchical routing; latency scales with H(t), not corpus size. - ---- - -### P1-C: Fast Semantic Neighbor Insert (UNBLOCKS: graph coherence) - -**Why:** Need a sparse semantic neighbor graph for coherent path tracing. This graph connects pages with high cosine similarity and is used for BFS subgraph expansion during retrieval. Degree must be bounded by `HotpathPolicy` to prevent unbounded graph mass growth. **This is not related to Metroid construction** β€” the semantic neighbor graph is a proximity concept, not a dialectical probe concept. - -- [ ] **P1-C1:** Implement `hippocampus/FastNeighborInsert.ts` - - For each new page, find cosine-nearest neighbors within Williams-cutoff **distance** (not a fixed K); derive the cutoff radius from `HotpathPolicy` rather than a hardcoded constant - - Insert forward edges (page β†’ neighbors) as `SemanticNeighbor` records, respecting max degree - - Insert reverse edges (neighbors β†’ page), respecting max degree per direction - - If a page is already at max degree, evict the neighbor with the lowest cosine similarity - - Insert only initial edges at ingest time; do not attempt full cross-edge reconnection β€” Daydreamer walks the graph during idle passes to build additional edges (avoids full graph recalc on every insert) - - **Edge role invariant:** `SemanticNeighbor.cosineSimilarity` is used for neighbor discovery and Bayesian belief updates. Hebbian edge weights (in `edges_hebbian`) are used for TSP tour traversal. These are separate edge types with separate roles; do not mix them. - - Mark affected volumes as dirty for full Daydreamer recalc - - After insertion, check new page for hotpath admission via `SalienceEngine` - -- [ ] **P1-C2:** Upgrade `hippocampus/Ingest.ts` - - After persisting pages, call `FastNeighborInsert` - -- [ ] **P1-C3:** Add semantic neighbor insert test coverage - - `tests/hippocampus/FastNeighborInsert.test.ts` - - Test neighbor lists are bounded by Williams-cutoff distance (not a fixed K) - - Test symmetry (if Aβ†’B, then Bβ†’A) - - Test that degree overflow evicts lowest-cosine-similarity neighbor, not a random one - - Test that new page is considered for hotpath admission after insertion - - Test that `edges_hebbian` records are NOT created by FastNeighborInsert (Hebbian is Daydreamer's concern) - -**Exit Criteria:** Semantic neighbor graph is maintained during ingest with policy-bounded degree. - ---- - -### P1-D: Open TSP Solver (UNBLOCKS: coherent path ordering) - -**Why:** Need to trace coherent path through induced subgraph, not just ranked list. - -- [ ] **P1-D1:** Implement `cortex/OpenTSPSolver.ts` - - Dummy-node open-path heuristic (greedy nearest-neighbor) - - Input: `SemanticNeighborSubgraph` (nodes + edges with distances; after P0-X2 rename) - - Output: ordered path through all nodes - - Deterministic for same input - -- [ ] **P1-D2:** Add TSP solver test coverage - - `tests/cortex/OpenTSPSolver.test.ts` - - Test on synthetic small graphs (3-10 nodes) - - Test determinism (same input β†’ same output) - - Test path validity (all nodes visited exactly once) - -**Exit Criteria:** Can compute coherent open path through subgraph. - ---- - -### P1-M: MetroidBuilder (DELIVERS: dialectical epistemology) - -**Why:** MetroidBuilder is the core of what makes CORTEX an _epistemic_ system rather than a vector search engine. Without it, the system merely returns nearest neighbors and cannot explore opposing perspectives, detect knowledge gaps, or trigger P2P curiosity requests. The Metroid loop converts conceptual opposition into navigable exploration steps. - -- [ ] **P1-M1:** Implement `cortex/MetroidBuilder.ts` - - Accept a query embedding `q` and a list of resident medoids (shelf/volume/book representatives) - - **Thesis (select m1):** Find `m1` via medoid search β€” the medoid minimizing distance to `q`. A - medoid (not a centroid) is always an existing memory node; it ensures the search anchor is an - actual data point rather than an averaged phantom position. This keeps the search on the - correct conceptual road. - - Read `matryoshkaProtectedDim` from `ModelProfile` (e.g. 128 for embeddinggemma-300m, 64 for - nomic-embed-text-v1.5). If `undefined` on the current model (non-Matryoshka), return - `{ m1, m2: null, c: null, knowledgeGap: true }` immediately. - - **Freeze:** Lock all dimensions with index < `matryoshkaProtectedDim`. - - **Antithesis (find m2):** In the unfrozen upper dimensions (index >= `matryoshkaProtectedDim`): - 1. Score every candidate medoid as `-cosine_similarity(candidate_free_dims, m1_free_dims)`. - The highest-scoring candidates are farthest from m1 in the free dimensions β€” maximal - conceptual divergence. - 2. Find the **medoid of that cosine-opposite set** (the top-scoring candidates). This is `m2`. - 3. `m2` must be an existing memory node (not a computed position). The medoid operation - ensures this. This is distinct from simply finding the node with the lowest cosine - similarity to m1. - - **Synthesis (freeze centroid):** Compute `c` once and freeze it: - - Protected dims (< `matryoshkaProtectedDim`): copy from m1 (domain invariant). - - Free dims (>= `matryoshkaProtectedDim`): `c[i] = (m1[i] + m2[i]) / 2`. - - This frozen `c` is never recalculated. All future candidates in the Matryoshka unwind are - evaluated relative to this frozen platform. - - Return `Metroid { m1, m2, c }`; if no valid m2 found, return - `{ m1, m2: null, c: null, knowledgeGap: true }` - -- [ ] **P1-M2:** Implement Matryoshka dimensional unwinding in `cortex/MetroidBuilder.ts` - - After the initial Metroid construction, progressively expand the antithesis search into deeper - embedding layers by shifting the protected dimension boundary outward one Matryoshka tier at a - time. - - At each new tier, find a new `m2` candidate via cosine-opposite medoid search in the expanded - free dimensions. - - Evaluate each candidate against the **frozen** `c` (not a recomputed centroid). If close - enough to `c`, accept and freeze this step; take the next conceptual leap. If not, - continue unwinding. - - Stop when the protected dimension floor is reached or a satisfactory `m2` is accepted. - - If no satisfactory `m2` is found at any layer, return `knowledgeGap: true`. - -- [ ] **P1-M3:** Add MetroidBuilder test coverage - - `tests/cortex/MetroidBuilder.test.ts` - - Test m1 selection: the medoid minimising distance to q is chosen (not the centroid) - - Test m2 selection: medoid of cosine-opposite set β€” not merely nearest semantically-opposing node - - Test centroid computation: protected dims copied from m1; free dims averaged element-wise - - Test centroid is frozen: subsequent unwinding steps do not recompute c - - Test dimensional unwinding: search expands progressively through Matryoshka layers - - Test knowledge gap: when no valid m2 exists in any layer, returns `knowledgeGap: true` - - Test protected dimensions are never searched for antithesis - - Test determinism: same inputs always produce same Metroid - -**Exit Criteria:** MetroidBuilder constructs valid Metroids (m1 via medoid search, m2 via -cosine-opposite medoid of the top-scoring candidates, c computed once and never recomputed during -Matryoshka unwinding) and correctly detects knowledge gaps. - ---- - -### P1-N: Knowledge Gap Detection & Curiosity Probe (DELIVERS: epistemic honesty) - -**Why:** When MetroidBuilder cannot find m2, the system must acknowledge its knowledge boundary rather than hallucinating. The curiosity probe mechanism enables distributed learning by broadcasting the gap to peers. - -- [ ] **P1-N1:** Implement `cortex/KnowledgeGapDetector.ts` - - Accept MetroidBuilder result; if `knowledgeGap: true`, emit a `KnowledgeGap` DTO - - `KnowledgeGap { topicMedoidId: Hash, queryEmbedding: Float32Array, dimensionalBoundary: number, timestamp: string }` - - This DTO is returned to the caller as part of `QueryResult` - -- [ ] **P1-N2:** Implement curiosity probe construction in `cortex/KnowledgeGapDetector.ts` - - Build `CuriosityProbe { m1, partialMetroid, queryContext, knowledgeBoundary, mimeType, modelUrn }` - - `mimeType`: MIME type of embedded content (e.g. `text/plain`). Enables receiving peers to validate content-type compatibility before comparing graph sections. - - `modelUrn`: URN of the embedding model (e.g. `urn:model:onnx-community/embeddinggemma-300m-ONNX:v1`) sourced from the active `ModelProfile.modelId`. Peers **must** reject probes whose `modelUrn` does not match a model they support β€” accepting fragments from a different embedding model would produce incommensurable similarity scores at Matryoshka layer boundaries. - - Store probe locally for broadcast via P2P layer (see P2-G) - - Do not broadcast immediately β€” queue for the P2P sharing layer - -- [ ] **P1-N3:** Upgrade `cortex/QueryResult.ts` - - Add `knowledgeGap?: KnowledgeGap` field β€” present when MetroidBuilder failed to find m2 - - Document that callers must check this field before treating results as epistemically complete - -- [ ] **P1-N4:** Add knowledge gap test coverage - - `tests/cortex/KnowledgeGapDetector.test.ts` - - Test that a KnowledgeGap DTO is produced when MetroidBuilder returns `knowledgeGap: true` - - Test that a CuriosityProbe is constructed with correct fields including `mimeType` and `modelUrn` - - Test that `modelUrn` is derived from `ModelProfile.modelId` (not hardcoded) - - Test that QueryResult includes the KnowledgeGap when present - - Test that queries against a rich corpus do NOT produce false-positive knowledge gaps - -**Exit Criteria:** System correctly signals knowledge boundaries; callers can distinguish epistemically complete from incomplete results. - ---- - -### P1-E: Full Query Orchestrator (DELIVERS: dialectical retrieval) - -**Why:** This is the "aha" moment β€” return memories in natural narrative order through the resident hotpath via dialectical Metroid exploration, with dynamic, sublinear expansion bounds. - -> **Note on scope:** The existing `cortex/Query.ts` is a flat top-K scorer that does not use MetroidBuilder, Hebbian edge traversal, or cosine-similarity-bounded subgraph expansion. It must be **substantially reworked** β€” not merely extended β€” to implement the dialectical pipeline described below. The same applies to `cortex/QueryResult.ts`. Do not attempt to preserve the flat-scoring code path; it is superseded entirely. - -- [ ] **P1-E1:** Rewrite `cortex/Query.ts` (full dialectical version) - - Use resident-first hierarchical ranking to select topic medoid (m1) - - Call `MetroidBuilder` to construct `{ m1, m2, c }` - - If knowledge gap detected, include in result and continue with partial Metroid (m1 only) - - Use centroid `c` as the primary scoring anchor for page selection - - Derive dynamic subgraph bounds from `HotpathPolicy` (`maxSubgraphSize`, `maxHops`, `perHopBranching`) - - Call `MetadataStore.getInducedNeighborSubgraph(seedPages, maxHops)` using dynamic `maxHops`; traverse edges using Hebbian weights for tour distance (not cosine similarity) - - Call `OpenTSPSolver.solve(subgraph)` - - Return ordered page list via coherent path - - **Query cost meter:** count vector operations; early-stop and return best-so-far if cost exceeds Williams-derived budget - - Include provenance metadata (hop count, edge weights, subgraph size, cost, Metroid details) - -- [ ] **P1-E2:** Rewrite `cortex/QueryResult.ts` - - Add `coherencePath: Hash[]` (ordered page IDs) - - Add `metroid?: { m1: Hash; m2: Hash | null; centroid: Float32Array | null }` (Metroid used for this query) - - Add `knowledgeGap?: KnowledgeGap` (if antithesis discovery failed) - - Add `provenance: { subgraphSize: number; hopCount: number; edgeWeights: number[]; vectorOpCost: number; earlyStop: boolean }` - -- [ ] **P1-E3:** Add full query test coverage - - `tests/cortex/Query.test.ts` (upgrade) - - Test subgraph expansion stays within `maxSubgraphSize` - - Test TSP ordering - - Test Metroid is built and included in provenance - - Test knowledge gap is returned when antithesis not found - - Test provenance metadata - - Test early-stop fires when cost budget exceeded - -**Exit Criteria:** Queries return dialectically balanced, coherent context chains through the resident hotpath; MetroidBuilder active; knowledge gaps surfaced. - ---- - -### P1-F: Integration Test (Hierarchical + Dialectical) - -**Why:** Validate v0.5 completeness including resident-first routing, MetroidBuilder, and dialectical subgraph bounds. - -- [ ] **P1-F1:** Upgrade `tests/integration/IngestQuery.test.ts` - - Verify hierarchical structures exist after ingest - - Verify hotpath entries exist for hierarchy prototypes after ingest - - Verify queries build a valid Metroid `{ m1, m2, c }` - - Verify queries return coherent paths through resident hotpath - - Verify dynamic subgraph bounds honoured (no expansion beyond `maxSubgraphSize`) - - Verify knowledge gap is correctly signalled when corpus is sparse - - Compare dialectical retrieval vs flat ranking (show epistemic breadth improvement) - -**Exit Criteria:** Integration test demonstrates dialectically balanced retrieval with resident-first routing and knowledge gap detection. - ---- - -## 🟒 Medium Priority β€” Ship v1.0 (Background Consolidation + Smart Sharing) - -These items add idle background maintenance and privacy-safe interest sharing. Together they deliver v1.0's discovery value. - -### P2-A: Idle Scheduler (UNBLOCKS: Daydreamer operations) - -**Why:** Need cooperative background loop that doesn't block foreground. - -- [ ] **P2-A1:** Implement `daydreamer/IdleScheduler.ts` - - Loop via `requestIdleCallback` (browser) or `setImmediate` (Node) - - Interruptible (yield after N ms of work) - - CPU budget awareness (pause if main thread busy) - - Task queue (prioritize high-value work) - -- [ ] **P2-A2:** Add scheduler test coverage - - `tests/daydreamer/IdleScheduler.test.ts` - - Test cooperative yielding - - Test interruption doesn't corrupt state - -**Exit Criteria:** Background loop runs without blocking UI. - ---- - -### P2-B: Hebbian Updater (DELIVERS: connection plasticity) - -**Why:** Strengthen useful connections, decay unused ones. Edge changes alter Οƒ(v) values and can trigger hotpath promotions or evictions. - -- [ ] **P2-B1:** Implement `daydreamer/HebbianUpdater.ts` - - LTP: strengthen edges traversed during successful queries - - LTD: decay all edges by small factor each pass - - Prune: remove edges below threshold; keep Metroid degree within `HotpathPolicy`-derived bounds - - After LTP/LTD: recompute Οƒ(v) for all nodes whose incident edges changed (via `SalienceEngine.batchComputeSalience`) - - Run promotion/eviction sweep for changed nodes via `SalienceEngine.runPromotionSweep` - - Update `MetadataStore.putEdges` - -- [ ] **P2-B2:** Add Hebbian test coverage - - `tests/daydreamer/HebbianUpdater.test.ts` - - Test strengthen increases weight - - Test decay decreases weight - - Test pruning removes weak edges and keeps degree within bounds - - Test that salience is recomputed for changed nodes - - Test that promotion sweep runs after LTP increases salience above weakest resident - -**Exit Criteria:** Edge weights adapt based on usage; salience and hotpath updated accordingly. - ---- - -### P2-C: Full Neighbor Graph Recalc (DELIVERS: graph maintenance) - -**Why:** Incremental fast semantic neighbor insert is approximate; need periodic full recalc. Recalc batch size must be bounded by H(t)-derived maintenance budget to avoid blocking the idle loop. - -- [ ] **P2-C1:** Implement `daydreamer/FullNeighborRecalc.ts` - - Query `MetadataStore.needsNeighborRecalc(volumeId)` for dirty volumes; prioritise dirtiest first - - Load all pages in volume; compute pairwise similarities - - Bound batch: process at most `HotpathPolicy.computeCapacity(graphMass)` pairwise comparisons per idle cycle (O(√(t log t))) - - Select policy-derived max neighbors for each page; update `MetadataStore.putSemanticNeighbors` - - Clear dirty flag via `MetadataStore.clearNeighborRecalcFlag` - - Recompute Οƒ(v) for affected nodes via `SalienceEngine.batchComputeSalience`; run promotion sweep - -- [ ] **P2-C2:** Add neighbor graph recalc test coverage - - `tests/daydreamer/FullNeighborRecalc.test.ts` - - Test dirty flag cleared after recalc - - Test neighbor quality improved vs fast insert - - Test batch size respects O(√(t log t)) limit per cycle - - Test salience recomputed and promotion sweep runs after recalc - -**Exit Criteria:** Dirty volumes are recalculated in background within bounded compute budget; salience updated. - ---- - -### P2-D: Prototype Recomputer (DELIVERS: prototype quality) - -**Why:** Keep volume/shelf prototypes accurate as pages/books change. Prototype updates change which entries should occupy the volume and shelf tier quotas. - -- [ ] **P2-D1:** Implement `daydreamer/PrototypeRecomputer.ts` - - Recompute volume medoids (select medoid page per volume) - - Recompute volume centroids (average of book embeddings) - - Recompute shelf routing prototypes - - Update vectors in `VectorStore` (append new, update offsets) - - After recomputing each level: recompute salience for affected representative entries via `SalienceEngine`; run tier-quota promotion/eviction for that tier - -- [ ] **P2-D2:** Add prototype recomputer test coverage - - `tests/daydreamer/PrototypeRecomputer.test.ts` - - Test medoid selection algorithm - - Test centroid computation - - Test that tier-quota hotpath entries are updated after prototype recomputation - -**Exit Criteria:** Prototypes stay accurate over time; tier quota entries reflect current prototypes. - ---- - -### P2-E: Integration Test (Background Consolidation) - -**Why:** Validate Daydreamer improves system health and hotpath stays consistent. - -- [ ] **P2-E1:** Implement `tests/integration/Daydreamer.test.ts` - - Ingest corpus - - Run queries (generate edge traversals and PageActivity updates) - - Run Daydreamer for N passes - - Verify edge weights updated - - Verify dirty volumes recalculated - - Verify prototypes updated - - Verify resident count never exceeds H(t) after any Daydreamer pass - -**Exit Criteria:** Daydreamer demonstrably maintains system health; Williams Bound invariant holds. - ---- - -### P2-F: Community Detection & Graph Coverage Quotas (DELIVERS: topic-diverse hotpath) - -**Why:** Without community detection, a single dense topic can fill the entire page-tier quota, crowding out unrelated memories. Community quotas ensure the hotpath is both hot (high salience) and diverse (topic-representative). - -- [ ] **P2-F1:** Add community detection to `daydreamer/ClusterStability.ts` - - Implement lightweight label propagation on the semantic neighbor graph - - Run during idle passes when dirty-volume flags indicate meaningful structural change - - Store community labels in `PageActivity.communityId` via `MetadataStore.putPageActivity` - - Rerun when graph topology changes significantly (post-split, post-merge, post-full-recalc) - -- [ ] **P2-F2:** Wire community labels into `SalienceEngine` promotion/eviction - - `selectEvictionTarget` uses `communityId` to find weakest resident in the community bucket - - Promotion checks community quota remaining before admitting - - If community quota is full: candidate must beat weakest resident in that community - - If community is unknown (`communityId` not yet set): place node in temporary pending pool borrowing from page-tier budget - - Empty communities release their slots back to the page-tier budget - -- [ ] **P2-F3:** Add community-aware eviction tests - - `tests/daydreamer/ClusterStability.test.ts` - - Test that a single dense community cannot consume all page-tier hotpath slots - - Test that a new community (previously unknown) receives at least one slot - - Test that an empty community releases its slots correctly - - Test that label propagation converges and produces stable community assignments - -**Exit Criteria:** Community-aware hotpath quotas active; topic diversity enforced; label propagation stable. - ---- - -### P2-G: Curiosity Broadcasting & Smart Interest Sharing (DELIVERS: distributed learning without hallucination) - -**Why:** When knowledge gaps are detected, CORTEX must be able to broadcast the incomplete Metroid as a curiosity probe to connected peers. Peers respond with relevant fragments, enabling collaborative learning. Additionally, interest sharing is a core product value for both app and library surfaces. v1 must share public-interest graph sections while preventing personal data leakage. - -- [ ] **P2-G0:** Implement `sharing/CuriosityBroadcaster.ts` - - Consume pending `CuriosityProbe` objects queued by `KnowledgeGapDetector` - - Serialize and broadcast to connected peers via P2P transport - - Handle responses: deserialize incoming graph fragments; pass to `SubgraphImporter` for integration - - Rate-limit broadcasts to prevent spam - - Include `knowledgeBoundary` field in probe so peers can target search precisely - -- [ ] **P2-G1:** Implement `sharing/EligibilityClassifier.ts` - - Classify candidate nodes as share-eligible vs blocked before export - - Detect identity/PII-bearing content (person-specific identifiers, credentials, financial/health traces) - - Emit deterministic eligibility decisions with reason codes for auditability - -- [ ] **P2-G2:** Implement `sharing/SubgraphExporter.ts` - - Build topic-scoped graph slices from eligible nodes only - - For curiosity responses: select graph fragment relevant to the received probe's `knowledgeBoundary` - - Preserve node/edge signatures and provenance - - Strip or coarsen personal metadata fields that are not needed for discovery - -- [ ] **P2-G3:** Implement `sharing/PeerExchange.ts` and `sharing/SubgraphImporter.ts` - - Opt-in peer exchange over P2P transport - - Verify signatures and schema on import; reject invalid or tampered payloads - - Merge imported slices into discovery pathways without exposing sender identity metadata - - After import, retry MetroidBuilder for any pending knowledge gaps that may be resolved by new data - -- [ ] **P2-G4:** Add sharing safety and discovery tests - - `tests/sharing/EligibilityClassifier.test.ts` - - `tests/sharing/CuriosityBroadcaster.test.ts` - - `tests/sharing/SubgraphExchange.test.ts` - - Assert blocked nodes are never exported; assert imported fragments are discoverable via query - - Assert that after receiving a response to a curiosity probe, MetroidBuilder can now construct m2 for the previously-gapped topic - -**Exit Criteria:** v1 can broadcast curiosity probes for knowledge gaps, receive graph fragments from peers, retry MetroidBuilder with new data, and exchange signed public-interest slices with PII blocking. - ---- - -### P3-H: GitHub sync smoke test (VALIDATES: automation pipeline) - -**Why:** Ensure the `TODO.md` β†’ GitHub issue sync path works end-to-end (milestones/labels/issues created by `scripts/sync-github-project.mjs`). - -- [ ] **P3-H1:** Push this change to `main` so the sync workflow runs -- [ ] **P3-H2:** Verify a new issue is created automatically from this task group -- [ ] **P3-H3:** Mark this task complete once the issue exists and the workflow succeeded - -**Exit Criteria:** A GitHub issue was created from this task group after pushing to `main`, and the sync workflow succeeded. - ---- - -## πŸ”΅ Lower Priority β€” Polish & Ship - -These items improve quality, performance, and developer experience. Not blockers for v1.0 launch. - -### P3-A: WebGL Embedding Provider - -**Why:** Explicit `webgl` fallback for systems without WebGPU/WebNN. - -- [ ] **P3-A1:** Implement `embeddings/OrtWebglEmbeddingBackend.ts` - - Use ONNX Runtime Web with explicit `webgl` execution provider - - Implement `EmbeddingBackend` interface - - Add to `ProviderResolver` candidate sets - -- [ ] **P3-A2:** Add WebGL provider test coverage - - `tests/embeddings/OrtWebglEmbeddingBackend.test.ts` - -**Exit Criteria:** `webgl` backend available for systems without WebGPU. - ---- - -### P3-B: Experience Replay - -**Why:** Simulate queries during idle time to reinforce connection patterns. - -- [ ] **P3-B1:** Implement `daydreamer/ExperienceReplay.ts` - - Sample random or recent queries - - Execute query (triggers edge traversals) - - Mark traversed edges for LTP strengthening - -- [ ] **P3-B2:** Add experience replay test coverage - - `tests/daydreamer/ExperienceReplay.test.ts` - -**Exit Criteria:** Daydreamer reinforces memory patterns. - ---- - -### P3-C: Cluster Stability (full implementation) - -**Why:** Detect and fix unstable clusters (split oversized, merge undersized). The community detection added in P2-F is a subset of this module; here we add the full split/merge machinery. - -- [ ] **P3-C1:** Complete `daydreamer/ClusterStability.ts` - - Detect high-variance volumes (unstable) - - Trigger split (K-means with K=2) - - Detect low-count volumes - - Trigger merge with nearest neighbor volume - - Re-run community detection and update PageActivity after split/merge - -- [ ] **P3-C2:** Add cluster stability test coverage - - `tests/daydreamer/ClusterStability.test.ts` (extend from P2-F) - - Test split produces two balanced volumes - - Test merge produces one combined volume - - Test community labels updated after structural change - -**Exit Criteria:** Clusters stay balanced over time; community labels stay current. - ---- - -### P3-D: Benchmark Suite - -**Why:** Measure performance, validate Williams Bound invariants, and track regressions. - -- [ ] **P3-D1:** Implement real-provider benchmarks - - `tests/benchmarks/TransformersJsEmbedding.bench.ts` - - Throughput (embeddings/sec) for various batch sizes - -- [ ] **P3-D2:** Implement query latency benchmarks - - `tests/benchmarks/QueryLatency.bench.ts` - - Latency vs corpus size (100 pages, 1K pages, 10K pages) - -- [ ] **P3-D3:** Implement storage overhead benchmarks - - `tests/benchmarks/StorageOverhead.bench.ts` - - Disk usage vs page count - -- [ ] **P3-D4:** Implement hotpath scaling benchmarks - - `tests/benchmarks/HotpathScaling.bench.ts` - - Synthetic graphs at 1K, 10K, 100K, 1M nodes+edges - - Measure: resident set size vs H(t), query latency vs corpus size, promotion/eviction throughput - - **Assert:** resident count never exceeds H(t); query cost scales sublinearly with corpus size - - Assert: H(t) values match expected sublinear curve at each scale point - -- [ ] **P3-D5:** Record baseline measurements - - Add `benchmarks/BASELINES.md` with results from all benchmarks - - Include H(t) curve data at 1K/10K/100K/1M - -**Exit Criteria:** Benchmark suite exists; baselines recorded; Williams Bound invariants asserted. - ---- - -### P3-E: CI Hardening - -**Why:** Ensure tests run reliably in CI; enforce both model-derived and policy-derived numeric guards. - -- [ ] **P3-E1:** Add GitHub Actions workflow - - `.github/workflows/ci.yml` - - Run `npm run build`, `npm run lint`, `npm run test:unit`, `npm run guard:model-derived` - -- [ ] **P3-E2:** Define Electron runtime gate policy - - Document GPU/graphics requirements - - Decide CI runner capabilities (software vs hardware rendering) - - Update `scripts/run-electron-runtime-tests.mjs` gate logic - -- [ ] **P3-E3:** Add hotpath policy constants guard - - Extend `scripts/guard-model-derived.mjs` or add `scripts/guard-hotpath-policy.mjs` - - Scan for numeric literals assigned to hotpath policy fields outside `core/HotpathPolicy.ts` - - Add as required CI gate alongside `guard:model-derived` - - Add `npm run guard:hotpath-policy` script to `package.json` - -**Exit Criteria:** CI runs on every PR; merge blocked if tests or guards fail; both model-derived and policy-derived constants are enforced. - ---- - -### P3-F: Documentation - -**Why:** Users need to know how to integrate CORTEX. - -- [ ] **P3-F1:** Update `docs/api.md` - - Document `ingestText(...)` API - - Document `query(...)` API - - Document `QueryResult` structure - -- [ ] **P3-F2:** Update `docs/development.md` - - Add troubleshooting section - - Add performance tuning guide - -- [ ] **P3-F3:** Add architecture diagrams - - Data flow: ingest path - - Data flow: query path - - Module dependency graph - -**Exit Criteria:** API docs complete; developer guide useful. - ---- - -### P3-G: Product Surface UX Contract - -**Why:** v1.0 needs an explicit UX contract for the standalone app while keeping the library surface headless and integration-first. - -- [ ] **P3-G1:** Add `docs/product-surfaces.md` - - Define app-vs-library scope, boundaries, and non-goals - - Define standalone extension user journey: passive capture -> search -> revisit - - Define what remains local-only and private in the app shell - -- [ ] **P3-G2:** Add standalone search UX checklist to `docs/product-surfaces.md` - - Search-first information architecture (query bar, results, lightweight metrics) - - Result-card contract (title, URL, snippet/thumbnail, visit recency, relevance signal) - - UX states: empty index, no matches, loading/indexing, error recovery - -- [ ] **P3-G3:** Add model-mode UX contract to `docs/product-surfaces.md` - - Nomic mode: multimodal recall (text + images in shared latent space) - - Gemma mode: fine-grained text recall (no image embedding) - - UI copy rules that make image-recall availability explicit by mode - -- [ ] **P3-G4:** Add rabbit-hole recall acceptance checklist - - Vague text recollection scenario recovers a previously visited page path - - Vague visual recollection scenario recovers a previously seen image when Nomic mode is enabled - - Add manual validation steps for model toggle behavior and capability messaging - -**Exit Criteria:** Standalone app UX contract and model-mode behavior are documented; library boundary remains explicit and implementation-ready. - ---- - -## πŸ“‹ Summary by Phase - -| Phase | Items | Status | Blocking | -|-------|-------|--------|----------| -| v0.1 (Minimal Viable) | 30 tasks (P0-A through P0-G + P0-E + P0-X) | 🟑 In Progress (P0-A, P0-F, P0-G complete; P0-X architectural rename pending) | User cannot use system correctly; P0-X blocks MetroidBuilder | -| v0.5 (Hierarchical + Dialectical) | 20 tasks (P1-A through P1-F + P1-M + P1-N) | ❌ Not started | Blocked by v0.1 | -| v1.0 (Background Consolidation + Smart Sharing) | 20 tasks (P2-A through P2-G) | ❌ Not started | Blocked by v0.5 | -| Polish & Ship | 21 tasks (P3-A through P3-G) | ❌ Not started | Not blocking v1.0 | - -**Total:** ~91 actionable tasks - ---- - -## Quick Reference: Next Tasks to Unblock Everything - -If you're reading this and want to know "what do I work on right now?", here's the answer: - -**Immediate (unblock MetroidBuilder):** -1. **P0-X1–X7:** Fix architectural naming drift (`MetroidNeighbor` β†’ `SemanticNeighbor` and related renames) - -**After P0-X (complete v0.1):** -2. **P0-B1:** Implement `hippocampus/Chunker.ts` -3. **P0-C1/C2:** Implement `hippocampus/PageBuilder.ts` and `hippocampus/Ingest.ts` -4. **P0-D1:** Implement `cortex/Query.ts` (minimal) - -**After v0.1 (start v0.5):** -5. **P1-A1:** Implement `hippocampus/HierarchyBuilder.ts` -6. **P1-C1:** Implement `hippocampus/FastNeighborInsert.ts` -7. **P1-M1/M2:** Implement `cortex/MetroidBuilder.ts` with Matryoshka unwinding -8. **P1-N1/N2:** Implement `cortex/KnowledgeGapDetector.ts` -9. **P1-D1:** Implement `cortex/OpenTSPSolver.ts` -10. **P1-E1:** Rewrite `cortex/Query.ts` to full dialectical orchestrator (substantial rework; not backward-compatible with flat top-K version) - ---- - -## Notes - -- **Dependencies:** Items are ordered so that completing tasks in sequence minimises blocked work. P0-X (naming drift fix) must precede MetroidBuilder. P0-F and P0-G (Williams Bound foundation) must precede all hotpath-aware modules. -- **Estimates:** Each P0/P1/P2 task is roughly 1-4 hours for an experienced developer familiar with the codebase. -- **Testing:** Every implementation task should be accompanied by test coverage (explicitly called out). -- **TDD Approach:** Write failing tests first, then implement to green. -- **Documentation Sync:** Update PLAN.md module status as tasks complete. -- **Williams Bound Invariant:** The resident count must never exceed H(t). Every test that touches the hotpath should assert this. -- **Policy constants:** Never hardcode hotpath constants outside `core/HotpathPolicy.ts`. P3-E3 will add a guard to enforce this automatically; until then, enforce by convention. -- **Metroid vs medoid vs semantic neighbor graph:** These are three distinct concepts. `Metroid` = dialectical probe `{ m1, m2, c }` (ephemeral, query-time). `medoid` = cluster representative node. Semantic neighbor graph = sparse proximity edges used for BFS subgraph expansion. Do not conflate them. See P0-X for the code rename tasks that fix the current conflation. diff --git a/docs/development.md b/docs/development.md index c729d5f..e2a60e4 100644 --- a/docs/development.md +++ b/docs/development.md @@ -170,9 +170,9 @@ This command will fail the build if it detects numeric literals that are likely At the end of every implementation pass, update documents in this order: -1. **`PROJECT-EXECUTION-PLAN.md`** β€” append pass status delta and exact commands executed. -2. **`CORTEX-DESIGN-PLAN-TODO.md`** β€” update the design-to-code status matrix. -3. **`README.md`** β€” confirm the project description still reflects reality. -4. **`docs/api.md`** β€” update if new public APIs are added or existing ones change. +1. **`DESIGN.md`** β€” update if architecture changes. +2. **`README.md`** β€” confirm the project description still reflects reality. +3. **`docs/api.md`** β€” update if new public APIs are added or existing ones change. +4. **GitHub Issues** β€” close completed tasks, create new ones as needed via `gh` CLI or the web UI. > Numeric examples in design docs are illustrative unless explicitly sourced from model metadata. diff --git a/package.json b/package.json index d9855a1..81fc6bb 100644 --- a/package.json +++ b/package.json @@ -23,8 +23,7 @@ "docker:electron:build": "docker compose -f docker-compose.electron-debug.yml build electron-debug", "docker:electron:up": "docker compose -f docker-compose.electron-debug.yml up --build --force-recreate electron-debug", "docker:electron:down": "docker compose -f docker-compose.electron-debug.yml down --remove-orphans", - "docker:electron:logs": "docker compose -f docker-compose.electron-debug.yml logs -f electron-debug", - "sync:github-project": "bun scripts/sync-github-project.mjs" + "docker:electron:logs": "docker compose -f docker-compose.electron-debug.yml logs -f electron-debug" }, "devDependencies": { "@eslint/js": "latest", diff --git a/scripts/sync-github-project.mjs b/scripts/sync-github-project.mjs deleted file mode 100644 index 2a2ccc9..0000000 --- a/scripts/sync-github-project.mjs +++ /dev/null @@ -1,495 +0,0 @@ -#!/usr/bin/env bun - -/** - * sync-github-project.mjs - * - * Reads PLAN.md and TODO.md and creates the corresponding GitHub structure: - * - Milestones (one per release phase, sourced from PLAN.md) - * - Labels (priority P0–P3 and layer labels) - * - Issues (one per ### task-group in TODO.md) - * - * Re-run safe: existing milestones, labels, and issues with matching titles - * are detected and skipped. Completed groups (βœ… COMPLETE) are created and - * immediately closed. - * - * Usage: - * bun scripts/sync-github-project.mjs [--dry-run] - * - * --dry-run Print every action that would be taken; make no API calls. - * - * Prerequisites: - * gh CLI installed and authenticated (run: gh auth status) - */ - -import { readFile } from "node:fs/promises"; -import { execFile } from "node:child_process"; -import { promisify } from "node:util"; -import path from "node:path"; -import process from "node:process"; - -const execFileAsync = promisify(execFile); -const ROOT = process.cwd(); -const DRY_RUN = process.argv.includes("--dry-run"); - -// --------------------------------------------------------------------------- -// Milestone definitions β€” derived from the four phases in PLAN.md -// --------------------------------------------------------------------------- - -const MILESTONES = [ - { - phaseEmoji: "🚨", - title: "v0.1 β€” Minimal Viable", - description: - "Critical path: enable ingest and retrieval for a single user session. " + - "Exit criteria: user can call ingestText() and query() end-to-end.", - }, - { - phaseEmoji: "🟑", - title: "v0.5 β€” Hierarchical + Coherent", - description: - "Add hierarchical routing and coherent path ordering. " + - "Exit criteria: queries return ordered context chains, not just ranked pages.", - }, - { - phaseEmoji: "🟒", - title: "v1.0 β€” Background Consolidation", - description: - "Idle background maintenance keeps memory healthy. " + - "Exit criteria: system self-maintains over extended use without manual intervention.", - }, - { - phaseEmoji: "πŸ”΅", - title: "v1.0 β€” Polish & Ship", - description: - "Improve quality, performance, and developer experience. " + - "Exit criteria: all tests pass; benchmarks recorded; docs complete; ready for public use.", - }, -]; - -// --------------------------------------------------------------------------- -// Label definitions -// --------------------------------------------------------------------------- - -const PRIORITY_LABELS = [ - { name: "P0: critical", color: "D73A4A", description: "Critical path β€” blocks all dependent work" }, - { name: "P1: high", color: "E4E669", description: "High priority β€” targets v0.5" }, - { name: "P2: medium", color: "0075CA", description: "Medium priority β€” targets v1.0" }, - { name: "P3: low", color: "CFE2F3", description: "Lower priority β€” polish and release prep" }, -]; - -const LAYER_LABELS = [ - { name: "layer: foundation", color: "F9D0C4", description: "Core types, model profiles, crypto" }, - { name: "layer: storage", color: "FEF2C0", description: "OPFS vector store and IndexedDB metadata store" }, - { name: "layer: compute", color: "C2E0C6", description: "WebGPU / WebGL / WebNN / WASM vector backends" }, - { name: "layer: embeddings", color: "FBCA04", description: "Embedding providers and resolver" }, - { name: "layer: hippocampus", color: "5319E7", description: "Ingest orchestration (chunk β†’ embed β†’ persist)" }, - { name: "layer: cortex", color: "1D76DB", description: "Retrieval orchestration (rank β†’ expand β†’ order)" }, - { name: "layer: daydreamer", color: "0E8A16", description: "Background consolidation (LTP/LTD, recalc)" }, - { name: "layer: testing", color: "C5DEF5", description: "Test coverage and integration tests" }, - { name: "layer: ci", color: "BFD4F2", description: "CI/CD pipeline and build tooling" }, - { name: "layer: documentation",color: "BFDADC", description: "API docs, developer guide, architecture diagrams" }, -]; - -const ALL_LABELS = [...PRIORITY_LABELS, ...LAYER_LABELS]; - -// --------------------------------------------------------------------------- -// TODO.md parser -// --------------------------------------------------------------------------- - -/** - * @typedef {{ header: string, bodyLines: string[], isComplete: boolean }} IssueGroup - * @typedef {{ header: string, phaseEmoji: string, groups: IssueGroup[] }} Phase - */ - -/** - * Parse TODO.md into phases and task groups. - * @param {string} content - * @returns {Phase[]} - */ -function parseTodoMd(content) { - const lines = content.split(/\r?\n/); - /** @type {Phase[]} */ - const phases = []; - /** @type {Phase | null} */ - let currentPhase = null; - /** @type {IssueGroup | null} */ - let currentGroup = null; - - for (const line of lines) { - if (line.startsWith("## ")) { - const header = line.slice(3).trim(); - const emojiMatch = header.match(/^(\p{Emoji})/u); - currentPhase = { - header, - phaseEmoji: emojiMatch ? emojiMatch[1] : "", - groups: [], - }; - phases.push(currentPhase); - currentGroup = null; - } else if (line.startsWith("### ") && currentPhase) { - const header = line.slice(4).trim(); - currentGroup = { - header, - bodyLines: [], - isComplete: header.includes("βœ… COMPLETE"), - }; - currentPhase.groups.push(currentGroup); - } else if (line === "---") { - currentGroup = null; - } else if (currentGroup) { - currentGroup.bodyLines.push(line); - } - } - - return phases; -} - -// --------------------------------------------------------------------------- -// Label inference helpers -// --------------------------------------------------------------------------- - -const LAYER_KEYWORD_MAP = /** @type {[string, string[]][]} */ ([ - ["layer: hippocampus", ["hippocampus", "Hippocampus", "Chunker", "Ingest", "PageBuilder", "FastMetroid", "HierarchyBuilder"]], - ["layer: cortex", ["cortex/", "Ranking.ts", "Query.ts", "OpenTSPSolver", "QueryResult", "SeedSelection"]], - ["layer: daydreamer", ["daydreamer", "Daydreamer", "IdleScheduler", "HebbianUpdater", "FullMetroidRecalc", "PrototypeRecomputer", "ExperienceReplay", "ClusterStability"]], - ["layer: embeddings", ["embeddings/", "EmbeddingBackend", "OrtWebgl", "TransformersJs", "ProviderResolver"]], - ["layer: testing", ["tests/integration", "tests/benchmarks", "Integration", "Benchmark", "bench.ts"]], - ["layer: ci", ["CI", "GitHub Actions", ".github/workflows", "guard-model-derived", "ci.yml"]], - ["layer: documentation",["docs/", "documentation", "API reference", "architecture diagram"]], -]); - -/** - * Infer layer labels from a group's header and body. - * @param {string} header - * @param {string[]} bodyLines - * @returns {string[]} - */ -function inferLayerLabels(header, bodyLines) { - const haystack = [header, ...bodyLines].join("\n"); - const found = []; - for (const [label, keywords] of LAYER_KEYWORD_MAP) { - if (keywords.some((kw) => haystack.includes(kw))) { - found.push(label); - } - } - return found; -} - -/** - * Infer priority label from a group's header prefix (P0-X, P1-X …). - * @param {string} header - * @returns {string | null} - */ -function inferPriorityLabel(header) { - const m = header.match(/^P([0-3])-/); - if (!m) { - return null; - } - const map = { 0: "P0: critical", 1: "P1: high", 2: "P2: medium", 3: "P3: low" }; - return map[m[1]] ?? null; -} - -/** - * Map a phase's emoji to the matching milestone title. - * @param {string} emoji - * @returns {string | null} - */ -function phaseEmojiToMilestone(emoji) { - const m = MILESTONES.find((ms) => ms.phaseEmoji === emoji); - return m ? m.title : null; -} - -// --------------------------------------------------------------------------- -// GitHub API helpers (uses `gh auth token` for auth, native fetch for calls) -// --------------------------------------------------------------------------- - -/** @returns {Promise} */ -async function getGitHubToken() { - const { stdout } = await execFileAsync("gh", ["auth", "token"]); - return stdout.trim(); -} - -/** - * @returns {Promise<{ owner: string, repo: string }>} - */ -async function getRepoIdentity() { - const { stdout } = await execFileAsync("gh", [ - "repo", "view", "--json", "name,owner", - ]); - const { name, owner } = JSON.parse(stdout); - return { owner: owner.login, repo: name }; -} - -/** - * Fetch all pages of a GitHub API endpoint (link-header pagination). - * @param {string} token - * @param {string} url - * @returns {Promise} - */ -async function fetchAllPages(token, url) { - const results = []; - let nextUrl = url; - - while (nextUrl) { - const res = await fetch(nextUrl, { - headers: { - Authorization: `Bearer ${token}`, - Accept: "application/vnd.github+json", - "X-GitHub-Api-Version": "2022-11-28", - }, - }); - - if (!res.ok) { - const body = await res.text(); - throw new Error(`GET ${nextUrl} β†’ ${res.status}: ${body}`); - } - - const page = await res.json(); - results.push(...page); - - const link = res.headers.get("link") ?? ""; - const match = link.match(/<([^>]+)>;\s*rel="next"/); - // null signals the end of pagination - nextUrl = match ? match[1] : null; - } - - return results; -} - -/** - * Call GitHub REST API with a JSON body. - * @param {string} token - * @param {"POST"|"PATCH"|"DELETE"} method - * @param {string} url - * @param {object} [body] - * @returns {Promise} - */ -async function githubApi(token, method, url, body) { - const res = await fetch(url, { - method, - headers: { - Authorization: `Bearer ${token}`, - Accept: "application/vnd.github+json", - "X-GitHub-Api-Version": "2022-11-28", - "Content-Type": "application/json", - }, - body: body ? JSON.stringify(body) : undefined, - }); - - const text = await res.text(); - if (!res.ok) { - throw new Error(`${method} ${url} β†’ ${res.status}: ${text}`); - } - return text ? JSON.parse(text) : null; -} - -// --------------------------------------------------------------------------- -// Sync helpers -// --------------------------------------------------------------------------- - -/** - * Create any milestones from MILESTONES that do not yet exist. - * Returns a map of milestone title β†’ milestone number. - * @param {string} token - * @param {string} owner - * @param {string} repo - * @returns {Promise>} - */ -async function syncMilestones(token, owner, repo) { - const base = `https://api.github.com/repos/${owner}/${repo}`; - - /** @type {Array<{ title: string, number: number }>} */ - const existing = /** @type {any[]} */ ( - await fetchAllPages(token, `${base}/milestones?state=all&per_page=100`) - ); - - const existingTitles = new Map(existing.map((m) => [m.title, m.number])); - const milestoneMap = new Map(existingTitles); - - for (const ms of MILESTONES) { - if (existingTitles.has(ms.title)) { - globalThis.console.log(` ⏭ Milestone already exists: "${ms.title}"`); - continue; - } - - globalThis.console.log(` βž• Creating milestone: "${ms.title}"`); - if (!DRY_RUN) { - const created = /** @type {{ number: number }} */ ( - await githubApi(token, "POST", `${base}/milestones`, { - title: ms.title, - description: ms.description, - state: "open", - }) - ); - milestoneMap.set(ms.title, created.number); - } else { - milestoneMap.set(ms.title, -1); - } - } - - return milestoneMap; -} - -/** - * Create any labels in ALL_LABELS that do not yet exist. - * @param {string} token - * @param {string} owner - * @param {string} repo - */ -async function syncLabels(token, owner, repo) { - const base = `https://api.github.com/repos/${owner}/${repo}`; - - /** @type {Array<{ name: string }>} */ - const existing = /** @type {any[]} */ ( - await fetchAllPages(token, `${base}/labels?per_page=100`) - ); - - const existingNames = new Set(existing.map((l) => l.name)); - - for (const label of ALL_LABELS) { - if (existingNames.has(label.name)) { - globalThis.console.log(` ⏭ Label already exists: "${label.name}"`); - continue; - } - - globalThis.console.log(` βž• Creating label: "${label.name}"`); - if (!DRY_RUN) { - await githubApi(token, "POST", `${base}/labels`, { - name: label.name, - color: label.color, - description: label.description, - }); - } - } -} - -/** - * Create GitHub issues for every task group in the parsed phases. - * Already-existing issues (same title) are skipped. Completed groups are - * created then immediately closed. - * - * @param {string} token - * @param {string} owner - * @param {string} repo - * @param {Phase[]} phases - * @param {Map} milestoneMap - */ -async function syncIssues(token, owner, repo, phases, milestoneMap) { - const base = `https://api.github.com/repos/${owner}/${repo}`; - - /** @type {Array<{ title: string, number: number, state: string }>} */ - const existingIssues = /** @type {any[]} */ ( - await fetchAllPages(token, `${base}/issues?state=all&per_page=100`) - ); - - const existingTitles = new Set(existingIssues.map((i) => i.title)); - - for (const phase of phases) { - if (phase.groups.length === 0) { - continue; - } - - const milestoneName = phaseEmojiToMilestone(phase.phaseEmoji); - const milestoneNumber = milestoneName ? milestoneMap.get(milestoneName) : null; - - for (const group of phase.groups) { - const title = group.header.replace(/\s*βœ…\s*COMPLETE\s*/g, "").trim(); - - if (existingTitles.has(title)) { - globalThis.console.log(` ⏭ Issue already exists: "${title}"`); - continue; - } - - // Build issue body: preserve the markdown content as-is - const body = group.bodyLines.join("\n").trim(); - const priorityLabel = inferPriorityLabel(title); - const layerLabels = inferLayerLabels(title, group.bodyLines); - const labels = [priorityLabel, ...layerLabels].filter(Boolean); - - globalThis.console.log(` βž• Creating issue: "${title}"`); - if (labels.length > 0) { - globalThis.console.log(` Labels: ${labels.join(", ")}`); - } - if (milestoneName) { - globalThis.console.log(` Milestone: ${milestoneName}`); - } - - if (!DRY_RUN) { - /** @type {{ number: number }} */ - const created = /** @type {any} */ ( - await githubApi(token, "POST", `${base}/issues`, { - title, - body, - labels, - ...(milestoneNumber != null && milestoneNumber !== -1 - ? { milestone: milestoneNumber } - : {}), - }) - ); - - if (group.isComplete) { - await githubApi(token, "PATCH", `${base}/issues/${created.number}`, { - state: "closed", - state_reason: "completed", - }); - globalThis.console.log(` Closed (already complete)`); - } - } - } - } -} - -// --------------------------------------------------------------------------- -// Main -// --------------------------------------------------------------------------- - -async function main() { - if (DRY_RUN) { - globalThis.console.log("πŸ” Dry-run mode β€” no changes will be made.\n"); - } - - // Verify gh CLI is available - try { - await execFileAsync("gh", ["auth", "status"]); - } catch { - globalThis.console.error( - "❌ gh CLI is not authenticated. Run: gh auth login", - ); - process.exit(1); - } - - const token = await getGitHubToken(); - const { owner, repo } = await getRepoIdentity(); - globalThis.console.log(`\nRepository: ${owner}/${repo}\n`); - - // Parse TODO.md - const todoPath = path.join(ROOT, "TODO.md"); - const todoContent = await readFile(todoPath, "utf8"); - const phases = parseTodoMd(todoContent); - - const totalGroups = phases.reduce((n, p) => n + p.groups.length, 0); - globalThis.console.log(`Parsed TODO.md: ${phases.length} phases, ${totalGroups} task groups\n`); - - // 1. Milestones - globalThis.console.log("── Milestones ──────────────────────────────────────────────"); - const milestoneMap = await syncMilestones(token, owner, repo); - - // 2. Labels - globalThis.console.log("\n── Labels ──────────────────────────────────────────────────"); - await syncLabels(token, owner, repo); - - // 3. Issues - globalThis.console.log("\n── Issues ──────────────────────────────────────────────────"); - await syncIssues(token, owner, repo, phases, milestoneMap); - - globalThis.console.log( - DRY_RUN - ? "\nβœ… Dry-run complete. Re-run without --dry-run to apply changes." - : "\nβœ… Sync complete.", - ); -} - -main().catch((error) => { - globalThis.console.error("sync-github-project crashed:", error); - process.exit(1); -}); From 49ea77151c0e3f7725da0a66b2647235b34f35c2 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Fri, 13 Mar 2026 21:47:18 +0000 Subject: [PATCH 3/3] Keep TODO.md and PLAN.md, scope close-legacy-issues to explicit allowlist - Restore TODO.md and PLAN.md (keep existing docs while GitHub-native project management is set up) - Rewrite close-legacy-issues.yml to only close the 29 specific issues created by the old sync script, not all future issues - Restore PLAN.md/TODO.md rows in README.md and copilot-instructions.md Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com> --- .github/copilot-instructions.md | 2 + .github/workflows/close-legacy-issues.yml | 50 +- PLAN.md | 493 ++++++++++++ README.md | 2 + TODO.md | 892 ++++++++++++++++++++++ 5 files changed, 1415 insertions(+), 24 deletions(-) create mode 100644 PLAN.md create mode 100644 TODO.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 0fb2fc1..9f08291 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -15,6 +15,8 @@ The engine models three biological brain regions: |---|---| | `README.md` | Product vision and quick start | | `DESIGN.md` | Complete architecture specification and design principles | +| `PLAN.md` | Module-by-module implementation status and development phases | +| `TODO.md` | Prioritized actionable tasks to ship v1.0 | Keep `DESIGN.md` synchronized with the real code state after every implementation pass. diff --git a/.github/workflows/close-legacy-issues.yml b/.github/workflows/close-legacy-issues.yml index dbab262..86c922b 100644 --- a/.github/workflows/close-legacy-issues.yml +++ b/.github/workflows/close-legacy-issues.yml @@ -1,7 +1,7 @@ name: Close Legacy Issues -# One-shot workflow: closes all pre-existing issues that were created by the -# removed sync-github-project.mjs script. Run manually, then delete this file. +# One-shot workflow: closes the 29 issues created by the removed +# sync-github-project.mjs script. Run manually once, then delete this file. on: workflow_dispatch: @@ -10,16 +10,22 @@ permissions: issues: write jobs: - close-all: + close-legacy: runs-on: ubuntu-latest steps: - - name: Close all open issues as won't-fix + - name: Close legacy sync-generated issues as won't-fix uses: actions/github-script@v7 with: script: | const owner = context.repo.owner; const repo = context.repo.repo; + // Exact issue numbers created by the old sync-github-project.mjs script. + const LEGACY_ISSUES = [ + 21, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, + 36, 37, 38, 39, 40, 41, 42, 56, 57, 58, 59, 60, 61, 62, 63 + ]; + // Ensure the "wontfix" label exists try { await github.rest.issues.getLabel({ owner, repo, name: "wontfix" }); @@ -32,36 +38,32 @@ jobs: }); } - // Paginate through all open issues - let page = 1; let closed = 0; - while (true) { - const { data: issues } = await github.rest.issues.listForRepo({ - owner, repo, - state: "open", - per_page: 100, - page, - }); - if (issues.length === 0) break; - - for (const issue of issues) { - // Skip pull requests (they appear in the issues API) - if (issue.pull_request) continue; - + for (const num of LEGACY_ISSUES) { + try { + const { data: issue } = await github.rest.issues.get({ + owner, repo, + issue_number: num, + }); + if (issue.state === "closed") { + console.log(`#${num} already closed β€” skipping`); + continue; + } await github.rest.issues.addLabels({ owner, repo, - issue_number: issue.number, + issue_number: num, labels: ["wontfix"], }); await github.rest.issues.update({ owner, repo, - issue_number: issue.number, + issue_number: num, state: "closed", state_reason: "not_planned", }); - console.log(`Closed #${issue.number}: ${issue.title}`); + console.log(`Closed #${num}: ${issue.title}`); closed++; + } catch (err) { + console.log(`#${num}: skipped (${err.message})`); } - page++; } - console.log(`\nDone β€” closed ${closed} issues.`); + console.log(`\nDone β€” closed ${closed} legacy issues.`); diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..0d8c942 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,493 @@ +# CORTEX Implementation Plan + +**Version:** 1.2 +**Last Updated:** 2026-03-13 + +This document tracks the implementation status of each major module in CORTEX. It shows what's been built, what's in progress, and what remains. + +## Status Legend + +- βœ… **Complete** β€” Fully implemented with test coverage +- 🟑 **Partial** β€” Core implementation exists but incomplete or lacking tests +- ❌ **Missing** β€” Not yet implemented +- πŸ”„ **In Progress** β€” Currently being developed + +--- + +## Module Status + +### Foundation Layer + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| Core Types | βœ… Complete | `core/types.ts` | All entity interfaces defined; includes `PageActivity`, `HotpathEntry`, `TierQuotas`, and `MetadataStore` hotpath method signatures | +| Model Profiles | βœ… Complete | `core/ModelProfile.ts`, `core/ModelDefaults.ts`, `core/ModelProfileResolver.ts`, `core/BuiltInModelProfiles.ts` | Source-of-truth for model-derived numerics; guard script enforces compliance | +| Numeric Constants | βœ… Complete | `core/NumericConstants.ts` | Runtime constants (byte sizes, workgroup limits) centralized | +| Crypto Helpers | βœ… Complete | `core/crypto/hash.ts`, `core/crypto/sign.ts`, `core/crypto/verify.ts` | SHA-256 hashing; Ed25519 sign/verify; 26 tests passing | +| Hotpath Policy | βœ… Complete | `core/HotpathPolicy.ts` | Williams Bound policy implementation; covered by `tests/HotpathPolicy.test.ts` | +| Salience Engine | βœ… Complete | `core/SalienceEngine.ts` | Per-node salience computation, promotion/eviction lifecycle helpers, community-aware admission logic; covered by `tests/SalienceEngine.test.ts` | + +**Foundation Status:** 6/6 complete (100%) + +--- + +### Storage Layer + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| Vector Store (OPFS) | βœ… Complete | `storage/OPFSVectorStore.ts` | Append-only binary vector file; byte-offset addressing; test coverage via `tests/Persistence.test.ts` | +| Vector Store (Memory) | βœ… Complete | `storage/MemoryVectorStore.ts` | In-memory implementation for testing | +| Metadata Store (IndexedDB) | βœ… Complete | `storage/IndexedDbMetadataStore.ts` | Full CRUD for all entities; reverse indexes; semantic neighbor graph operations (currently misnamed as "Metroid neighbor" β€” see TODO P0-X); dirty-volume flags; includes `hotpath_index` and `page_activity` object stores; hotpath CRUD methods are implemented and covered by `tests/Persistence.test.ts` | + +**Storage Status:** 3/3 complete (100%) + +--- + +### Vector Compute Layer + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| Backend Abstraction | βœ… Complete | `VectorBackend.ts`, `BackendKind.ts`, `CreateVectorBackend.ts` | Unified interface across all backends | +| WebGPU Backend | βœ… Complete | `WebGPUVectorBackend.ts`, `Vectors.wgsl` | Compute shader implementation | +| WebGL Backend | βœ… Complete | `WebGLVectorBackend.ts`, `Vectors.glsl` | Fragment shader implementation | +| WebNN Backend | βœ… Complete | `WebNNVectorBackend.ts` | ML accelerator path | +| WASM Backend | βœ… Complete | `WasmVectorBackend.ts`, `Vectors.wat` | Hand-written WebAssembly; guaranteed fallback | +| TopK Selection | βœ… Complete | `TopK.ts` | Utility for top-K similarity selection | + +**Vector Compute Status:** 6/6 complete (100%) + +--- + +### Embedding Layer + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| Embedding Interface | βœ… Complete | `embeddings/EmbeddingBackend.ts` | Abstract interface for all providers | +| Provider Resolver | βœ… Complete | `embeddings/ProviderResolver.ts` | Capability filtering + benchmark-based winner selection | +| Embedding Runner | βœ… Complete | `embeddings/EmbeddingRunner.ts` | High-level orchestrator with fallback chain | +| Dummy Provider | βœ… Complete | `embeddings/DeterministicDummyEmbeddingBackend.ts` | SHA-256-based deterministic embedder for testing | +| Transformers.js Provider | βœ… Complete | `embeddings/TransformersJsEmbeddingBackend.ts` | Real ONNX inference (`webnn`/`webgpu`/`wasm`); default model: `onnx-community/embeddinggemma-300m-ONNX` | +| ORT WebGL Provider | ❌ Missing | `embeddings/OrtWebglEmbeddingBackend.ts` (planned) | Explicit `webgl` fallback path not yet implemented | + +**Embedding Status:** 5/6 complete (83%) + +--- + +### Hippocampus (Ingest Orchestration) + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| Text Chunking | βœ… Complete | `hippocampus/Chunker.ts` | Token-aware sentence-boundary splitting respecting `ModelProfile.maxChunkTokens`; covered by `tests/hippocampus/Chunker.test.ts` | +| Page Builder | βœ… Complete | `hippocampus/PageBuilder.ts` | Builds signed `Page` entities with `contentHash`, `vectorHash`, `prevPageId`/`nextPageId` linkage; covered by `tests/hippocampus/PageBuilder.test.ts` | +| Ingest Orchestrator | 🟑 Partial | `hippocampus/Ingest.ts` | `ingestText()` implemented: chunk β†’ embed β†’ persist pages + PageActivity β†’ create Book β†’ run hotpath promotion sweep. **Missing:** hierarchy building (Volume/Shelf), semantic neighbor insertion. | +| Hierarchy Builder | ❌ Missing | `hippocampus/HierarchyBuilder.ts` (planned) | Construct/update Books, Volumes, Shelves; attempt tier-quota hotpath admission for each level's medoid/prototype; Williams-derived fanout bounds; trigger split via ClusterStability when bounds exceeded | +| Fast Semantic Neighbor Insert | ❌ Missing | `hippocampus/FastNeighborInsert.ts` (planned) | Cosine-nearest neighbors within Williams-cutoff distance (not fixed K). Degree overflow evicts lowest-cosine-similarity neighbor. Initial edges only at ingest; Daydreamer builds additional edges lazily. `SemanticNeighbor.cosineSimilarity` drives discovery + Bayesian updates; Hebbian weights (separate) drive TSP traversal. See DESIGN.md Β§Graph Structures for the full edge-role invariant. | + +**Hippocampus Status:** 2.5/5 complete (50%) + +**Critical Blocker:** Hierarchy builder and semantic neighbor insertion missing; ingest produces no graph structure beyond a single Book. + +--- + +### Cortex (Retrieval Orchestration) + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| Ranking Pipeline | ❌ Missing | `cortex/Ranking.ts` (planned) | Resident-first scoring cascade: HOT shelves β†’ HOT volumes β†’ HOT books β†’ HOT pages; spill to WARM/COLD only when coverage insufficient | +| MetroidBuilder | ❌ Missing | `cortex/MetroidBuilder.ts` (planned) | Constructs Metroid `{ m1, m2, c }` via Matryoshka dimensional unwinding; antithesis discovery; centroid computation; knowledge gap detection | +| Dialectical Search Pipeline | ❌ Missing | `cortex/DialecticalSearch.ts` (planned) | Orchestrates thesis/antithesis/synthesis zone exploration using a Metroid; prevents confirmation bias | +| Knowledge Gap Detector | ❌ Missing | `cortex/KnowledgeGapDetector.ts` (planned) | Determines when MetroidBuilder cannot find m2; emits curiosity probe | +| Seed Selection | ❌ Missing | `cortex/SeedSelection.ts` (planned) | Threshold-based top-k page selection from ranking output | +| Subgraph Expansion | 🟑 Partial | `storage/IndexedDbMetadataStore.ts` (`getInducedMetroidSubgraph` β€” to be renamed `getInducedNeighborSubgraph`) | BFS expansion implemented in storage layer; needs dynamic Williams bounds; needs orchestration wrapper | +| Open TSP Solver | ❌ Missing | `cortex/OpenTSPSolver.ts` (planned) | Dummy-node open-path heuristic for coherent ordering | +| Query Orchestrator | 🟑 Needs Rework | `cortex/Query.ts` | Flat top-K scoring implemented (hotpath-first β†’ warm/cold spill β†’ PageActivity update β†’ promotion sweep). **Must be substantially reworked** to implement the full dialectical pipeline: replace flat scoring with hierarchical resident-first ranking, add MetroidBuilder, dialectical zone scoring (thesis/antithesis/synthesis), subgraph expansion with dynamic Williams bounds, TSP coherence path, and query cost meter. The existing implementation does not use Hebbian edges or cosine-similarity-bounded subgraph expansion; it is a functional placeholder only. | +| Result DTO | 🟑 Needs Rework | `cortex/QueryResult.ts` | Minimal DTO (`pages`, `scores`, `metadata`). **Must be reworked** to add `coherencePath: Hash[]`, `metroid?: { m1, m2, centroid }`, `knowledgeGap?: KnowledgeGap`, and `provenance: { subgraphSize, hopCount, edgeWeights, vectorOpCost, earlyStop }`. | + +**Cortex Status:** 1.5/9 complete (17%) + +**Critical Blocker:** MetroidBuilder, dialectical search pipeline, and knowledge gap detector entirely absent. Existing `Query.ts` implements flat top-K retrieval only. + +--- + +### Daydreamer (Background Consolidation) + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| Idle Scheduler | ❌ Missing | `daydreamer/IdleScheduler.ts` (planned) | Cooperative background loop; interruptible; respects CPU budget | +| Hebbian Updates | ❌ Missing | `daydreamer/HebbianUpdater.ts` (planned) | LTP (strengthen), LTD (decay), prune below threshold; recompute Οƒ(v) for changed nodes; run promotion/eviction sweep | +| Prototype Recomputation | ❌ Missing | `daydreamer/PrototypeRecomputer.ts` (planned) | Recalculate volume/shelf medoids and centroids; recompute salience for affected entries; run tier-quota promotion/eviction | +| Full Neighbor Graph Recalc | ❌ Missing | `daydreamer/FullNeighborRecalc.ts` (planned) | Rebuild bounded neighbor lists for dirty volumes; batch size bounded by O(√(t log t)) per idle cycle; recompute salience after recalc. **Note:** Currently planned as `FullMetroidRecalc` β€” this is a naming error; see TODO P0-X. | +| Experience Replay | ❌ Missing | `daydreamer/ExperienceReplay.ts` (planned) | Simulate queries to reinforce connections | +| Cluster Stability | ❌ Missing | `daydreamer/ClusterStability.ts` (planned) | Detect/trigger split/merge for unstable clusters; run lightweight label propagation for community detection; store community labels in PageActivity | + +**Daydreamer Status:** 0/6 complete (0%) + +**Note:** Not a v1 blocker β€” system can ship without background consolidation (manual recalc only). Community detection is required before graph-community quota enforcement is active. + +--- + +### Policy & Configuration + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| Routing Policy | βœ… Complete | `Policy.ts` | Derives routing dimensions from ModelProfile; integration tested | +| Hotpath Policy | βœ… Complete | `core/HotpathPolicy.ts` | Williams Bound policy implementation, salience weights, tier quotas, community quotas; separate from model-derived numerics | + +**Policy Status:** 2/2 complete (100%) + +--- + +### Runtime Harness + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| Browser Harness | βœ… Complete | `runtime/harness/index.html`, `scripts/runtime-harness-server.mjs` | Localhost-served HTML harness for browser testing | +| Electron Wrapper | βœ… Complete | `scripts/electron-harness-main.mjs` | Thin Electron launcher for GPU-realism testing | +| Playwright Tests | βœ… Complete | `tests/runtime/browser-harness.spec.mjs`, `tests/runtime/electron-harness.spec.mjs` | Browser lane passes; Electron context-sensitive | +| Docker Debug Lane | βœ… Complete | `docker/electron-debug/*`, `docker-compose.electron-debug.yml` | Sandbox-isolated Electron debugging via VS Code attach | + +**Runtime Status:** 4/4 complete (100%) + +--- + +### Testing & Validation + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| Unit Tests | βœ… Complete | `tests/*.test.ts`, `tests/**/*.test.ts` | 115 tests across 13 files; all passing | +| Persistence Tests | βœ… Complete | `tests/Persistence.test.ts` | Full storage layer coverage (OPFS, IndexedDB, semantic neighbor graph β€” currently tested as "Metroid neighbors", hotpath indexes) | +| Model Tests | βœ… Complete | `tests/model/*.test.ts` | Profile resolution, defaults, routing policy | +| Embedding Tests | βœ… Complete | `tests/embeddings/*.test.ts` | Provider resolver, runner, real/dummy backends | +| Backend Smoke Tests | βœ… Complete | `tests/BackendSmoke.test.ts` | All vector backends instantiate cleanly | +| Runtime Tests | βœ… Complete | `tests/runtime/*.spec.mjs` | Browser harness validated; Electron context-sensitive | +| Integration Tests | βœ… Complete | `tests/integration/IngestQuery.test.ts` | End-to-end: ingest β†’ persist β†’ query β†’ verify results; persistence across sessions | +| Hotpath Policy Tests | βœ… Complete | `tests/HotpathPolicy.test.ts` | H(t) sublinearity and monotonicity; tier quota sums; community quota minimums; salience determinism | +| Salience Engine Tests | βœ… Complete | `tests/SalienceEngine.test.ts` | Bootstrap fills to H(t); steady-state eviction; community/tier quota enforcement; determinism | +| Scaling Benchmarks | ❌ Missing | `tests/benchmarks/HotpathScaling.bench.ts` (planned) | Synthetic graphs at 1K/10K/100K/1M; assert resident count ≀ H(t); query cost sublinear | +| Benchmarks | 🟑 Partial | `tests/benchmarks/DummyEmbedderHotpath.bench.ts` | Baseline dummy embedder benchmark; real-provider and hotpath scaling benchmarks needed | + +**Testing Status:** 9/12 complete (75%) + +--- + +### Build & CI + +| Module | Status | Files | Notes | +|--------|--------|-------|-------| +| TypeScript Config | βœ… Complete | `tsconfig.json` | Strict mode, ES2022 target, ESNext modules | +| Build Script | βœ… Complete | `package.json` (`build` script) | Type-check via `tsc --noEmit` | +| Lint Config | βœ… Complete | `eslint.config.mjs` | TypeScript-ESLint rules | +| Model-Derived Guard | βœ… Complete | `scripts/guard-model-derived.mjs` | Scans for hardcoded model numerics; enforces source-of-truth | +| Test Runner | βœ… Complete | `package.json` (Vitest scripts) | Unit, browser, electron, runtime, benchmark targets | +| CI Pipeline | 🟑 Partial | `.github/workflows/*` (if exists) | Needs verification; not examined in detail | +| GitHub Issue Sync | βœ… Complete | `scripts/sync-github-project.mjs`, `.github/workflows/sync-github-project.yml` | Syncs TODO.md β†’ GitHub issues/milestones; smoke test via TODO task | + +**Build Status:** 5/6 complete (83%) + +--- + +## Overall Progress Summary + +| Layer | Completion | Critical Gap | +|-------|-----------|--------------| +| Foundation | 100% | β€” | +| Storage | 100% | β€” | +| Vector Compute | 100% | β€” | +| Embedding | 83% | WebGL provider (low priority) | +| Hippocampus | 50% | Chunker + PageBuilder + minimal Ingest done; hierarchy builder and semantic neighbor insertion missing | +| Cortex | 17% | Minimal Query + QueryResult done; MetroidBuilder, dialectical search, knowledge gap detection all missing | +| Daydreamer | 0% | Not v1 blocker | +| Policy | 100% | β€” | +| Runtime | 100% | β€” | +| Testing | 67% | Integration tests, scaling benchmarks | +| Build/CI | 83% | β€” | + +**System-Wide Completion:** ~75% (core infrastructure, policy foundation, chunking, page building, and minimal ingest/query implemented; hierarchy builder, MetroidBuilder, and graph coherence remain.) + +--- + +## What Works Today + +- βœ… Store/retrieve vectors and metadata +- βœ… Vector similarity operations on all backends +- βœ… Generate real embeddings via Transformers.js +- βœ… Resolve model profiles and derive routing policies (including `matryoshkaProtectedDim` for Matryoshka models) +- βœ… Run browser/Electron runtime harness +- βœ… Pass 115 unit tests +- βœ… Hash text/binary content (SHA-256) and sign/verify Ed25519 signatures +- βœ… Chunk text and build signed `Page` entities +- βœ… Ingest text (minimal): chunk β†’ embed β†’ persist pages + PageActivity β†’ create Book β†’ hotpath promotion + +## What Doesn't Work Today + +- ❌ **No hierarchy beyond single Book** β€” Volume/Shelf hierarchy builder not yet implemented +- ❌ **No semantic neighbor graph** β€” `FastNeighborInsert` not yet implemented; subgraph expansion has no edges +- ❌ **No dialectical retrieval** β€” `MetroidBuilder`, `KnowledgeGapDetector`, and dialectical pipeline not yet implemented; current `Query.ts` is flat top-K retrieval only +- ❌ **No coherent path ordering** β€” No TSP solver; results are ranked list, not narrative chain +- ❌ **Cannot consolidate** β€” No Daydreamer loop +- ❌ **Cannot share discovery updates safely** β€” No P2P curiosity broadcasting or privacy-filtered exchange + +--- + +## Recommended Implementation Order + +### Phase 1: Unblock Basic Functionality (Ship v0.1) + +**Goal:** Enable ingest and retrieval for a single user session, with Williams Bound policy foundation in place. + +1. **Crypto Helpers** (`core/crypto/*`) βœ… **Complete** + - SHA-256 hashing for text and binary + - Ed25519 signing/verification + - 26 tests passing + +2. **Williams Bound Policy Foundation** βœ… **Complete** + - `core/HotpathPolicy.ts`, `core/SalienceEngine.ts`, `core/types.ts` extensions, `storage/IndexedDbMetadataStore.ts` hotpath stores + +3. **Text Chunking** (`hippocampus/Chunker.ts`) βœ… **Complete** + - Token-aware sentence-boundary splitting; tests passing + +4. **Page Builder** (`hippocampus/PageBuilder.ts`) βœ… **Complete** + - Signed Page entities with hash linkage; tests passing + +5. **Hippocampus Ingest** (`hippocampus/Ingest.ts`) 🟑 **Partial** + - Minimal `ingestText()` implemented (chunk β†’ embed β†’ persist pages β†’ single Book β†’ hotpath admission) + - **Remaining:** semantic neighbor insertion (deferred to Phase 2) + +6. **Cortex Query** (`cortex/Query.ts`) 🟑 **Partial** + - Minimal `query()` implemented (hotpath-first flat scoring; warm/cold spill) + - **Remaining:** MetroidBuilder, dialectical pipeline (deferred to Phase 2) + +7. **Integration Test** (`tests/integration/IngestQuery.test.ts`) βœ… **Complete** + - Ingest text β†’ Retrieve by query β†’ Validate results; persistence across sessions + +**Exit Criteria:** User can ingest text and retrieve relevant pages by query; Williams Bound policy is in place. + +--- + +### Phase 2: Add Hierarchy, Dialectical Search & Resident-First Routing (Ship v0.5) + +**Goal:** Hierarchical routing, MetroidBuilder, dialectical search pipeline, coherent path ordering, and fully resident-first query path. + +1. **Hierarchy Builder** (`hippocampus/HierarchyBuilder.ts`) + - Cluster pages into Books (medoid selection) + - Cluster books into Volumes (prototype computation) + - Build Shelves for coarse routing + - Attempt tier-quota hotpath admission for each level's medoid/prototype via `SalienceEngine` + - Williams-derived fanout bounds; trigger split via `ClusterStability` when exceeded + +2. **MetroidBuilder** (`cortex/MetroidBuilder.ts`) + - Select m1 (topic medoid) for a given query embedding + - Freeze protected Matryoshka dimensions + - Search for m2 (antithesis medoid) within unfrozen dimensions + - Compute centroid `c = (m1 + m2) / 2` + - Unwind Matryoshka layers progressively, repeating antithesis search + - Return `Metroid { m1, m2, c }` or signal knowledge gap + +3. **Knowledge Gap Detector** (`cortex/KnowledgeGapDetector.ts`) + - Evaluate MetroidBuilder result + - Emit `KnowledgeGap` DTO with dimensional boundary info + - Trigger P2P curiosity probe emission + +4. **Ranking Pipeline** (`cortex/Ranking.ts`) + - Resident-first cascade: HOT shelves β†’ HOT volumes β†’ HOT books β†’ HOT pages + - Spill to WARM/COLD only when resident coverage insufficient + +5. **Open TSP Solver** (`cortex/OpenTSPSolver.ts`) + - Dummy-node open-path heuristic + - Test on synthetic graphs + +6. **Full Query Orchestrator** (`cortex/Query.ts` β€” upgrade) + - Embed query β†’ select m1 β†’ build Metroid β†’ dialectical scoring cascade + - Dynamic subgraph expansion bounds from `HotpathPolicy` + - Query cost meter; early-stop on budget exceeded + - Coherent path via TSP + - Rich result DTO with provenance and knowledge gap flag + +**Exit Criteria:** User gets epistemically balanced context chains via MetroidBuilder and dialectical search; knowledge gaps are detected; query latency controlled by H(t). + +--- + +### Phase 3: Background Consolidation, Community Quotas & Smart Sharing (Ship v1.0) + +**Goal:** Idle maintenance keeps memory healthy, community-aware hotpath coverage stays diverse, and privacy-safe interest sharing is available. + +1. **Idle Scheduler** (`daydreamer/IdleScheduler.ts`) + - Cooperative, interruptible loop + - CPU budget awareness + +2. **Hebbian Updater** (`daydreamer/HebbianUpdater.ts`) + - LTP/LTD rules; edge pruning + - Recompute Οƒ(v) for changed nodes; run promotion/eviction sweep + +3. **Full Neighbor Graph Recalc** (`daydreamer/FullNeighborRecalc.ts`) + - Rebuild neighbor lists for dirty volumes + - O(√(t log t)) batch size per idle cycle + +4. **Prototype Recomputer** (`daydreamer/PrototypeRecomputer.ts`) + - Update volume/shelf prototypes + - Tier-quota promotion/eviction after recomputation + +5. **Community Detection** (`daydreamer/ClusterStability.ts` β€” extend) + - Label propagation on semantic neighbor graph + - Store community labels in `PageActivity.communityId` + - Wire community IDs into `SalienceEngine` promotion/eviction + +6. **Smart Interest Sharing** (`sharing/*` planned) + - `sharing/EligibilityClassifier.ts` β€” classify candidate nodes for share eligibility; block identity/PII-bearing nodes + - `sharing/SubgraphExporter.ts` β€” export signed, topic-scoped graph slices from eligible nodes only + - `sharing/SubgraphImporter.ts` β€” verify signatures/provenance and merge imported slices into local discovery index + - `sharing/PeerExchange.ts` β€” opt-in peer transport for exchanging eligible graph slices + +**Exit Criteria:** System self-maintains over extended use; community-aware hotpath quotas enforced; privacy-safe smart sharing works end-to-end. + +--- + +### Phase 4: Polish & Release Prep (Ship v1.0 Final) + +**Goal:** Production-ready quality. + +1. **Integration Test Suite** (`tests/integration/*`) + - Full vertical slice coverage + - Edge cases and error paths + - Performance regression tests + +2. **Scaling Benchmark Suite** (`tests/benchmarks/HotpathScaling.bench.ts`) + - Synthetic graphs at 1K, 10K, 100K, 1M nodes+edges + - Assert: resident count never exceeds H(t); query cost scales sublinearly + - Record baselines in `benchmarks/BASELINES.md` + +3. **Documentation** (`docs/*`) + - API reference + - Integration guide + - Troubleshooting + +4. **CI Hardening** + - Electron runtime gate policy + - Guard scripts in merge checks (model-derived + hotpath policy) + - Benchmark baselines + +5. **Product Surface UX Contract** + - Define standalone browser-extension UX baseline: + - Passive capture of visited pages into the local ingest queue + - Search-first recall UI over pages the user has actually seen + - Lightweight metrics panel that supports, but does not dominate, retrieval UX + - Define model-mode UX contract for the standalone app: + - Nomic mode = multimodal retrieval (text + images in shared latent space) + - Gemma mode = high-precision text retrieval (no image embedding) + - Capability messaging in UI so users understand image-recall availability by mode + - Define library boundary contract: + - Keep extension shell concerns outside core ingest/query APIs + - Keep library docs and examples headless/integration-first + - Add acceptance checks for rabbit-hole recall UX: + - Vague text recollection recovers previously visited pages + - Vague visual recollection recovers previously seen images when multimodal mode is enabled + +**Exit Criteria:** All tests pass; benchmarks recorded; docs complete; product-surface UX contract documented; ready for public use. + +--- + +## Known Blockers & Risks + +### Blocker 1: No Hierarchy Builder or Semantic Neighbor Graph +**Impact:** Ingest produces only a single flat Book; no Volume/Shelf structure; subgraph expansion has no edges to traverse. +**Mitigation:** Phase 2 priority; `HierarchyBuilder` and `FastNeighborInsert` must be implemented before dialectical retrieval is possible. + +### Blocker 2: No MetroidBuilder or Dialectical Pipeline +**Impact:** Queries return flat top-K results only; no epistemic balance, no knowledge gap detection, no P2P curiosity. +**Mitigation:** Phase 2 priority; depends on semantic neighbor graph (Blocker 1) and hierarchy builder. + +### Blocker 3: No Privacy-Safe Sharing or Curiosity Broadcasting Pipeline +**Impact:** Core discovery-sharing value proposition is missing; knowledge gaps cannot be resolved via P2P. +**Mitigation:** Phase 3 required track; implement eligibility classifier + curiosity broadcaster + signed subgraph exchange as v1 scope. CuriosityProbe must include `mimeType` and `modelUrn` to prevent incommensurable graph merges. + +### Blocker 4: Naming Drift (P0-X) +**Impact:** The term "Metroid" is currently used for the proximity graph in all code. MetroidBuilder cannot be introduced without a rename collision. +**Mitigation:** P0-X tasks (rename `MetroidNeighbor` β†’ `SemanticNeighbor`, etc.) must be completed before MetroidBuilder is implemented. + +### Risk 1: TSP Complexity +Open TSP is NP-hard; heuristic may be slow on large subgraphs. +**Mitigation:** Dynamic Williams-derived subgraph bounds shrink the problem as graph grows; defer to Phase 2; use deterministic greedy heuristic. + +### Risk 2: Electron Runtime Stability +Host-shell Electron can SIGSEGV in constrained contexts. +**Mitigation:** Docker attach lane validated for debugging; document GPU requirements for production. + +### Risk 3: WebGL Provider Gap +Transformers.js doesn't expose `webgl` device directly. +**Mitigation:** Low priority; `webgpu` and `wasm` sufficient for most users; explicit ORT adapter deferred to Phase 4. + +### Risk 4: Empirical Calibration of `c` +The Williams Bound scaling constant `c` is not theorem-given; wrong value causes either hotpath over-allocation (wastes RAM) or under-allocation (defeats purpose). +**Mitigation:** Default `c = 0.5` is conservative; scaling benchmarks in Phase 4 will validate and tune. Keep `c` in `core/HotpathPolicy.ts` as an overrideable policy constant. + +--- + +## Development Workflow + +### Command Reference + +```bash +# Install dependencies +npm ci + +# Type-check +npm run build + +# Lint +npm run lint + +# Run all unit tests +npm run test:unit + +# Run specific test file +npm run test:unit -- tests/model/ModelProfileResolver.test.ts + +# Model-derived numeric guard +npm run guard:model-derived + +# Run benchmarks +npm run benchmark + +# Start dev harness +npm run dev:harness + +# Run browser runtime tests +npm run test:browser + +# Run Electron tests (context-sensitive) +npm run test:electron + +# Docker Electron debug lane +npm run docker:electron:up +npm run docker:electron:down + +# Run all tests +npm run test:all +``` + +### Pre-Commit Checklist + +1. βœ… `npm run build` passes +2. βœ… `npm run lint` passes +3. βœ… `npm run test:unit` passes +4. βœ… `npm run guard:model-derived` passes +5. βœ… No hardcoded model-derived numerics outside `core/BuiltInModelProfiles.ts` +6. βœ… Tests added/updated for new functionality + +### Documentation Sync Protocol + +After every implementation pass: +1. Update `PLAN.md` module status +2. Update `TODO.md` to reflect completed items +3. Update `README.md` if user-facing changes + +--- + +## Notes + +- **Metroid vs medoid vs semantic neighbor graph:** These are three distinct concepts. `Metroid` refers only to the dialectical search probe `{ m1, m2, c }` constructed by `MetroidBuilder` at query time. `medoid` refers to a cluster representative node. The sparse proximity/neighbor graph (used for BFS subgraph expansion) is the **semantic neighbor graph** β€” it is currently misnamed `MetroidNeighbor`/`MetroidSubgraph` in code (see TODO P0-X for the rename task). +- **Model-derived numerics:** Never hardcode; always source from `core/` model profile modules. +- **Policy-derived constants:** Never hardcode; always source from `core/HotpathPolicy.ts`. +- **Test philosophy:** TDD (Red β†’ Green β†’ Refactor) for all new slices. +- **Runtime realism:** Browser and Electron lanes are required merge gates. +- **Williams Bound invariant:** The resident hotpath count must never exceed H(t). Enforce in tests and assert in benchmarks. diff --git a/README.md b/README.md index 0d33e20..cf92341 100644 --- a/README.md +++ b/README.md @@ -114,6 +114,8 @@ bun run dev:harness # start the browser runtime harness at http://127.0.0.1:4173 | Document | Purpose | |---|---| | [`DESIGN.md`](DESIGN.md) | Architecture specification and core design principles | +| [`PLAN.md`](PLAN.md) | Module-by-module implementation status and development phases | +| [`TODO.md`](TODO.md) | Prioritized actionable tasks to ship v1.0 | | [`ARCHITECTURE-REVIEW.md`](ARCHITECTURE-REVIEW.md) | Repository-wide architectural drift report and correction tasks | | [`docs/api.md`](docs/api.md) | API reference for developers integrating with CORTEX | | [`docs/development.md`](docs/development.md) | Build, test, debug, and Docker workflow | diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..bc21997 --- /dev/null +++ b/TODO.md @@ -0,0 +1,892 @@ +# CORTEX TODO β€” Path to v1.0 + +**Last Updated:** 2026-03-13 + +This document contains a prioritized, actionable list of specific tasks required to ship CORTEX v1.0. Items are ordered by dependency: highest-priority items are those blocking other work. + +--- + +## 🚨 Critical Path β€” Ship v0.1 (Minimal Viable) + +These items **must** be completed to have a usable system. Without them, users cannot ingest or query memories. + +### P0-A: Crypto Helpers (BLOCKS: all signed entity creation) βœ… COMPLETE + +**Why:** Pages require `contentHash`, `vectorHash`, and `signature`. Cannot create valid pages without crypto. + +- [x] **P0-A1:** Implement `core/crypto/hash.ts` + - SHA-256 for text content + - SHA-256 for binary vector data + - Test with known vectors + +- [x] **P0-A2:** Implement `core/crypto/sign.ts` + - Ed25519 key pair generation + - Sign canonical page representation + - Test with example pages + +- [x] **P0-A3:** Implement `core/crypto/verify.ts` + - Verify signature against public key + - Test reject invalid signatures + +- [x] **P0-A4:** Add crypto test coverage + - `tests/crypto/hash.test.ts` + - `tests/crypto/sign.test.ts` + - `tests/crypto/verify.test.ts` + +**Exit Criteria:** Can hash content, sign pages, verify signatures. βœ… Met β€” 26 tests passing. + +--- + +### P0-F: Williams Bound Policy Foundation (BLOCKS: all hotpath-aware modules) βœ… COMPLETE + +**Why:** The HotpathPolicy and SalienceEngine are the central source of truth for the Williams Bound architecture. Every subsequent module (ingest, query, hierarchy, Daydreamer) depends on them. Implementing these first ensures the bound is enforced from day one rather than retrofitted. + +- [x] **P0-F1:** Implement `core/HotpathPolicy.ts` + - `computeCapacity(graphMass: number): number` β€” H(t) = ⌈c Β· √(t Β· logβ‚‚(1+t))βŒ‰ + - `computeSalience(hebbianIn: number, recency: number, queryHits: number, weights?: SalienceWeights): number` β€” Οƒ = Ξ±Β·H_in + Ξ²Β·R + Ξ³Β·Q + - `deriveTierQuotas(capacity: number, quotaRatios?: TierQuotaRatios): TierQuotas` β€” allocate H(t) across shelf/volume/book/page tiers + - `deriveCommunityQuotas(tierBudget: number, communitySizes: number[]): number[]` β€” proportional with min(1) guarantee + - Export a frozen `DEFAULT_HOTPATH_POLICY` object containing all constants: `c = 0.5`, `Ξ± = 0.5`, `Ξ² = 0.3`, `Ξ³ = 0.2`, `q_s = 0.10`, `q_v = 0.20`, `q_b = 0.20`, `q_p = 0.50` + - Keep strictly separate from `core/ModelDefaults.ts` (policy-derived β‰  model-derived) + +- [x] **P0-F2:** Add HotpathPolicy test coverage (`tests/HotpathPolicy.test.ts`) + - H(t) grows sublinearly: verify `H(10_000) / 10_000 < H(1_000) / 1_000` + - H(t) is monotonically non-decreasing over a representative range: verify `H(t+1) >= H(t)` for each `t` in `[0, 1, 2, 10, 100, 1_000, 10_000, 100_000]` + - H(t) is a finite integer β‰₯ 1 for edge inputs: `t = 0`, `t = 1`, `t = Number.MAX_SAFE_INTEGER`; result must never be `NaN`, `Infinity`, or `< 1` + - Derived tier-quota *counts* sum exactly to capacity: `deriveTierQuotas(cap).shelf + .volume + .book + .page === cap` for `cap` in `[1, 10, 100, 1_000]` + - Community quota counts sum exactly to `tier_budget`: `sum(deriveCommunityQuotas(budget, sizes)) === budget` for representative `(budget, sizes)` inputs including edge cases (`budget = 0`, empty `sizes` array, `budget < sizes.length`) + - Community quotas never produce `NaN`, `Infinity`, or negative values for any valid input, including `sizes` with a single community or all equal sizes + - Salience is deterministic for same inputs + - Salience clamps output to a finite number: never `NaN` or `Infinity` for extreme weight or hit-count values + +- [x] **P0-F3:** Extend `core/types.ts` + - Add `PageActivity` interface: `{ pageId: Hash; queryHitCount: number; lastQueryAt: string; communityId?: string }` + - Add `HotpathEntry` interface: `{ entityId: Hash; tier: 'shelf' | 'volume' | 'book' | 'page'; salience: number; communityId?: string }` + - Add `TierQuotas` type: `{ shelf: number; volume: number; book: number; page: number }` + - Add hotpath method signatures to `MetadataStore` interface: + - `putHotpathEntry(entry: HotpathEntry): Promise` + - `getHotpathEntries(tier?: HotpathEntry['tier']): Promise` + - `evictWeakest(tier: HotpathEntry['tier'], communityId?: string): Promise` + - `getResidentCount(): Promise` + - `putPageActivity(activity: PageActivity): Promise` + - `getPageActivity(pageId: Hash): Promise` + +- [x] **P0-F4:** Extend `storage/IndexedDbMetadataStore.ts` + - Add `hotpath_index` object store keyed by `entityId`; secondary index by `tier` + - Add `page_activity` object store keyed by `pageId` + - Implement all six new `MetadataStore` hotpath methods + - Extend `tests/Persistence.test.ts` with hotpath store tests: + - put/get/evict cycle for `HotpathEntry` + - put/get for `PageActivity` + - `getResidentCount` returns correct value after multiple puts + +**Exit Criteria:** `HotpathPolicy` module passes all tests; `types.ts` has hotpath interfaces; IndexedDB hotpath stores are implemented and tested. βœ… Met β€” tests passing. + +--- + +### P0-G: Salience Engine (BLOCKS: hotpath promotion in ingest and Daydreamer) βœ… COMPLETE + +**Why:** The SalienceEngine is the decision-making layer for hotpath admission. It is needed by ingest (new page admission), query (hit-count update), and Daydreamer (post-LTP/LTD sweeps). Implementing it before ingest ensures promotion logic is correct from the first page written. + +- [x] **P0-G1:** Implement `core/SalienceEngine.ts` + - `computeNodeSalience(pageId: Hash, metadataStore: MetadataStore): Promise` β€” fetch PageActivity and incident Hebbian edges; apply Οƒ formula via HotpathPolicy + - `batchComputeSalience(pageIds: Hash[], metadataStore: MetadataStore): Promise>` β€” efficient batch version + - `shouldPromote(candidateSalience: number, weakestResidentSalience: number, capacityRemaining: number): boolean` β€” admission gating + - `selectEvictionTarget(tier: HotpathEntry['tier'], communityId: string | undefined, metadataStore: MetadataStore): Promise` β€” find weakest resident in tier/community bucket + +- [x] **P0-G2:** Implement promotion/eviction lifecycle helpers in `core/SalienceEngine.ts` + - `bootstrapHotpath(metadataStore: MetadataStore, policy: HotpathPolicy): Promise` β€” fill hotpath greedily by salience while resident count < H(t) + - `runPromotionSweep(candidateIds: Hash[], metadataStore: MetadataStore, policy: HotpathPolicy): Promise` β€” steady-state: promote if salience > weakest in same tier/community bucket; evict weakest on promotion + +- [x] **P0-G3:** Add SalienceEngine test coverage (`tests/SalienceEngine.test.ts`) + - Bootstrap fills hotpath to exactly H(t) given enough candidates + - Steady-state promotes only when candidate beats the weakest resident + - Steady-state evicts exactly the weakest resident (not a random entry) + - Community quotas prevent a single community from consuming all page-tier slots + - Tier quotas prevent one hierarchy level from dominating + - Eviction is deterministic under the same state + +**Exit Criteria:** `SalienceEngine` module passes all tests; promotion/eviction lifecycle is correct and deterministic. βœ… Met β€” tests passing. + +--- + +### P0-B: Text Chunking (BLOCKS: ingest orchestration) + +**Why:** Must split text into page-sized chunks respecting ModelProfile token limits. + +- [x] **P0-B1:** Implement `hippocampus/Chunker.ts` + - Token-aware splitting (use ModelProfile `maxContextLength`) + - Respect sentence boundaries where possible + - Handle edge cases (empty input, single-token input, huge paragraphs) + +- [x] **P0-B2:** Add chunker test coverage + - `tests/hippocampus/Chunker.test.ts` + - Test various text lengths (short, medium, long, huge) + - Test boundary conditions + +**Exit Criteria:** Can reliably split arbitrary text into page chunks. + +--- + +### P0-C: Hippocampus Ingest (Minimal) (BLOCKS: user workflow) + +**Why:** Users need to add memories to the system. + +- [x] **P0-C1:** Implement `hippocampus/PageBuilder.ts` + - Create `Page` entities from text chunks + - Generate `pageId`, `contentHash`, `vectorHash` + - Sign with provided key pair + - Link pages via `prevPageId`/`nextPageId` + - Initialise `PageActivity` records with zero counts + +- [x] **P0-C2:** Implement `hippocampus/Ingest.ts` (minimal version) + - Entry point: `ingestText(text, { modelProfile, embeddingRunner, vectorStore, metadataStore, keyPair, ... })` + - Chunk text via `Chunker` + - Batch embed chunks via `EmbeddingRunner` + - Persist vectors to `VectorStore` + - Build pages via `PageBuilder`; persist pages and `PageActivity` to `MetadataStore` + - Build single `Book` containing all pages (medoid = first page for now) + - After persisting pages, check each new page for hotpath admission via `SalienceEngine.runPromotionSweep` + - **Defer:** Volume/Shelf hierarchy, fast neighbor insert + +- [ ] **P0-C3:** Add ingest test coverage + - `tests/hippocampus/Ingest.test.ts` + - Test happy path (text β†’ pages β†’ book) + - Test persistence (can retrieve pages after ingest) + - Test that new pages are considered for hotpath admission after ingest + +**Exit Criteria:** User can call `ingestText(...)` and pages are persisted; PageActivity records exist; hotpath admission runs. + +--- + +### P0-D: Cortex Query (Minimal) (BLOCKS: user workflow) + +**Why:** Users need to retrieve relevant memories. + +- [x] **P0-D1:** Implement `cortex/Query.ts` (minimal version) + - Entry point: `query(queryText, modelProfile, vectorStore, metadataStore, topK)` + - Embed query via `EmbeddingRunner` + - Score resident hotpath entries first (HOT pages); fall back to full scan for WARM/COLD + - Compute similarities via `VectorBackend` + - Select top-K pages; increment `queryHitCount` in `PageActivity`; recompute salience; run promotion sweep + - Return `QueryResult` with page IDs and scores + - **Defer:** Full hierarchical ranking, subgraph expansion, TSP coherence, query cost meter + +- [x] **P0-D2:** Implement `cortex/QueryResult.ts` + - DTO with `pages: Page[]`, `scores: number[]`, `metadata: object` + +- [x] **P0-D3:** Add query test coverage + - `tests/cortex/Query.test.ts` + - Test happy path (query β†’ top-K pages) + - Test empty corpus (no results) + - Test relevance (query for known content returns expected pages) + - Test `PageActivity.queryHitCount` incremented after query hit + +**Exit Criteria:** User can call `query(...)` and get ranked pages; query hits update PageActivity and trigger salience recomputation. + +--- + +### P0-E: End-to-End Integration (VALIDATES: v0.1 completeness) + +**Why:** Prove the system works soup-to-nuts. + +- [x] **P0-E1:** Implement `tests/integration/IngestQuery.test.ts` + - Ingest sample text corpus (e.g., Wikipedia articles) + - Query for specific topics + - Verify expected pages returned + - Verify persistence (restart session, query again) + +- [x] **P0-E2:** Run integration test in browser harness + - Ensure real IndexedDB and OPFS work correctly + - Verify WebGPU/WebGL/WASM backends function + +**Exit Criteria:** Integration test passes; system is minimally usable. + +--- + +### P0-X: Fix Architectural Naming Drift (BLOCKS: correct design implementation) + +**Why:** The codebase uses the term "Metroid" to name the sparse proximity/neighbor graph (`MetroidNeighbor`, `MetroidSubgraph`, `metroid_neighbors`, `getInducedMetroidSubgraph`, `FastMetroidInsert`, `FullMetroidRecalc`). This is architecturally incorrect. In CORTEX, a **Metroid** is a structured dialectical search probe `{ m1, m2, c }` β€” a concept that does not yet exist in the codebase at all. The proximity graph has nothing to do with Metroids. This naming collision will cause permanent confusion and make the MetroidBuilder impossible to implement cleanly without a rename. + +- [ ] **P0-X1:** Rename `MetroidNeighbor` β†’ `SemanticNeighbor` in `core/types.ts` + - Update all references in `storage/IndexedDbMetadataStore.ts` + - Update all references in test files + - Update JSDoc and inline comments + +- [ ] **P0-X2:** Rename `MetroidSubgraph` β†’ `SemanticNeighborSubgraph` in `core/types.ts` + - Update all references in `storage/IndexedDbMetadataStore.ts` + - Update all references in `cortex/Query.ts` + - Update JSDoc and inline comments + +- [ ] **P0-X3:** Rename `MetadataStore` proximity graph methods: + - `putMetroidNeighbors` β†’ `putSemanticNeighbors` + - `getMetroidNeighbors` β†’ `getSemanticNeighbors` + - `getInducedMetroidSubgraph` β†’ `getInducedNeighborSubgraph` + - `needsMetroidRecalc` β†’ `needsNeighborRecalc` + - `flagVolumeForMetroidRecalc` β†’ `flagVolumeForNeighborRecalc` + - `clearMetroidRecalcFlag` β†’ `clearNeighborRecalcFlag` + - Update all callers in `storage/IndexedDbMetadataStore.ts`, `cortex/Query.ts`, and test files + +- [ ] **P0-X4:** Rename planned Hippocampus file `hippocampus/FastMetroidInsert.ts` β†’ `hippocampus/FastNeighborInsert.ts` + - Rename class/function to `FastNeighborInsert`/`insertSemanticNeighbors` + +- [ ] **P0-X5:** Rename planned Daydreamer file `daydreamer/FullMetroidRecalc.ts` β†’ `daydreamer/FullNeighborRecalc.ts` + - Rename class/function to `FullNeighborRecalc`/`runNeighborRecalc` + +- [ ] **P0-X6:** Rename IndexedDB object store from `metroid_neighbors` β†’ `neighbor_graph` + - Increment `DB_VERSION` in `storage/IndexedDbMetadataStore.ts` + - Add migration in `applyUpgrade` to copy data from old store to new store + +- [ ] **P0-X7:** Update all documentation strings and JSDoc that use "Metroid neighbor" to use "semantic neighbor" + +**Exit Criteria:** No source file uses "Metroid" to refer to the proximity graph. The term "Metroid" is reserved exclusively for the `{ m1, m2, c }` dialectical probe type implemented in `cortex/MetroidBuilder.ts`. + +--- + +## 🟑 High Priority β€” Ship v0.5 (Hierarchical + Coherent) + +These items add hierarchical routing and coherent path ordering. They transform CORTEX from a flat vector search into a biologically-inspired memory system. + +### P1-A: Hierarchy Builder (UNBLOCKS: hierarchical routing) + +**Why:** Need Volume and Shelf structures for efficient coarse-to-fine routing. Tier-quota hotpath admission must be integrated so hierarchy prototypes enter the resident index from the moment they are created. + +- [ ] **P1-A1:** Implement `hippocampus/HierarchyBuilder.ts` + - Cluster pages into Books (K-means or similar; select medoid) + - Cluster books into Volumes (compute prototype vectors) + - Cluster volumes into Shelves (coarse routing prototypes) + - Persist prototypes to `VectorStore`; update metadata in `MetadataStore` + - After each level: attempt hotpath admission via `SalienceEngine.runPromotionSweep`: + - Book medoid β†’ page-tier quota + - Volume prototypes β†’ volume-tier quota + - Shelf routing prototypes β†’ shelf-tier quota + - Enforce Williams-derived fanout bounds (see `HotpathPolicy`); when exceeded, trigger split via `ClusterStability` + +- [ ] **P1-A2:** Upgrade `hippocampus/Ingest.ts` + - After persisting pages, call `HierarchyBuilder` + - Maintain hierarchy incrementally (append to existing structures) + +- [ ] **P1-A3:** Add hierarchy test coverage + - `tests/hippocampus/HierarchyBuilder.test.ts` + - Test clustering produces valid Books/Volumes/Shelves + - Test prototypes are valid vectors + - Test that hierarchy medoids/prototypes are admitted to correct tier quota + - Test fanout bounds respected; split triggered when exceeded + +**Exit Criteria:** Ingestion produces full Page β†’ Book β†’ Volume β†’ Shelf hierarchy with tier-quota hotpath admission at every level. + +--- + +### P1-B: Ranking Pipeline (UNBLOCKS: efficient queries) + +**Why:** Hierarchical ranking avoids scanning all pages; reduces query latency. The resident hotpath is the primary lookup target β€” WARM/COLD spill happens only when the hot set provides insufficient coverage. + +- [ ] **P1-B1:** Implement `cortex/Ranking.ts` + - `rankShelves(queryEmbedding, residentShelves, topK)` β€” score HOT shelf prototypes first + - `rankVolumes(queryEmbedding, residentVolumes, topK)` β€” score HOT volume prototypes within top shelves + - `rankBooks(queryEmbedding, residentBooks, topK)` β€” score HOT book medoids within top volumes + - `rankPages(queryEmbedding, residentPages, topK)` β€” score HOT page representatives within top books + - `spillToWarm(tier, queryEmbedding, metadataStore, topK)` β€” spill to IndexedDB lookup when resident set insufficient + - Each step narrows the search space; H(t) is the primary latency lever + +- [ ] **P1-B2:** Upgrade `cortex/Query.ts` + - Replace flat search with resident-first hierarchical ranking cascade + - HOT shelves β†’ HOT volumes β†’ HOT books β†’ HOT pages β†’ WARM/COLD spill + +- [ ] **P1-B3:** Add ranking test coverage + - `tests/cortex/Ranking.test.ts` + - Test each ranking function independently + - Test full cascade produces correct top pages + - Test that resident entries are scored before non-resident entries + +**Exit Criteria:** Queries use resident-first hierarchical routing; latency scales with H(t), not corpus size. + +--- + +### P1-C: Fast Semantic Neighbor Insert (UNBLOCKS: graph coherence) + +**Why:** Need a sparse semantic neighbor graph for coherent path tracing. This graph connects pages with high cosine similarity and is used for BFS subgraph expansion during retrieval. Degree must be bounded by `HotpathPolicy` to prevent unbounded graph mass growth. **This is not related to Metroid construction** β€” the semantic neighbor graph is a proximity concept, not a dialectical probe concept. + +- [ ] **P1-C1:** Implement `hippocampus/FastNeighborInsert.ts` + - For each new page, find cosine-nearest neighbors within Williams-cutoff **distance** (not a fixed K); derive the cutoff radius from `HotpathPolicy` rather than a hardcoded constant + - Insert forward edges (page β†’ neighbors) as `SemanticNeighbor` records, respecting max degree + - Insert reverse edges (neighbors β†’ page), respecting max degree per direction + - If a page is already at max degree, evict the neighbor with the lowest cosine similarity + - Insert only initial edges at ingest time; do not attempt full cross-edge reconnection β€” Daydreamer walks the graph during idle passes to build additional edges (avoids full graph recalc on every insert) + - **Edge role invariant:** `SemanticNeighbor.cosineSimilarity` is used for neighbor discovery and Bayesian belief updates. Hebbian edge weights (in `edges_hebbian`) are used for TSP tour traversal. These are separate edge types with separate roles; do not mix them. + - Mark affected volumes as dirty for full Daydreamer recalc + - After insertion, check new page for hotpath admission via `SalienceEngine` + +- [ ] **P1-C2:** Upgrade `hippocampus/Ingest.ts` + - After persisting pages, call `FastNeighborInsert` + +- [ ] **P1-C3:** Add semantic neighbor insert test coverage + - `tests/hippocampus/FastNeighborInsert.test.ts` + - Test neighbor lists are bounded by Williams-cutoff distance (not a fixed K) + - Test symmetry (if Aβ†’B, then Bβ†’A) + - Test that degree overflow evicts lowest-cosine-similarity neighbor, not a random one + - Test that new page is considered for hotpath admission after insertion + - Test that `edges_hebbian` records are NOT created by FastNeighborInsert (Hebbian is Daydreamer's concern) + +**Exit Criteria:** Semantic neighbor graph is maintained during ingest with policy-bounded degree. + +--- + +### P1-D: Open TSP Solver (UNBLOCKS: coherent path ordering) + +**Why:** Need to trace coherent path through induced subgraph, not just ranked list. + +- [ ] **P1-D1:** Implement `cortex/OpenTSPSolver.ts` + - Dummy-node open-path heuristic (greedy nearest-neighbor) + - Input: `SemanticNeighborSubgraph` (nodes + edges with distances; after P0-X2 rename) + - Output: ordered path through all nodes + - Deterministic for same input + +- [ ] **P1-D2:** Add TSP solver test coverage + - `tests/cortex/OpenTSPSolver.test.ts` + - Test on synthetic small graphs (3-10 nodes) + - Test determinism (same input β†’ same output) + - Test path validity (all nodes visited exactly once) + +**Exit Criteria:** Can compute coherent open path through subgraph. + +--- + +### P1-M: MetroidBuilder (DELIVERS: dialectical epistemology) + +**Why:** MetroidBuilder is the core of what makes CORTEX an _epistemic_ system rather than a vector search engine. Without it, the system merely returns nearest neighbors and cannot explore opposing perspectives, detect knowledge gaps, or trigger P2P curiosity requests. The Metroid loop converts conceptual opposition into navigable exploration steps. + +- [ ] **P1-M1:** Implement `cortex/MetroidBuilder.ts` + - Accept a query embedding `q` and a list of resident medoids (shelf/volume/book representatives) + - **Thesis (select m1):** Find `m1` via medoid search β€” the medoid minimizing distance to `q`. A + medoid (not a centroid) is always an existing memory node; it ensures the search anchor is an + actual data point rather than an averaged phantom position. This keeps the search on the + correct conceptual road. + - Read `matryoshkaProtectedDim` from `ModelProfile` (e.g. 128 for embeddinggemma-300m, 64 for + nomic-embed-text-v1.5). If `undefined` on the current model (non-Matryoshka), return + `{ m1, m2: null, c: null, knowledgeGap: true }` immediately. + - **Freeze:** Lock all dimensions with index < `matryoshkaProtectedDim`. + - **Antithesis (find m2):** In the unfrozen upper dimensions (index >= `matryoshkaProtectedDim`): + 1. Score every candidate medoid as `-cosine_similarity(candidate_free_dims, m1_free_dims)`. + The highest-scoring candidates are farthest from m1 in the free dimensions β€” maximal + conceptual divergence. + 2. Find the **medoid of that cosine-opposite set** (the top-scoring candidates). This is `m2`. + 3. `m2` must be an existing memory node (not a computed position). The medoid operation + ensures this. This is distinct from simply finding the node with the lowest cosine + similarity to m1. + - **Synthesis (freeze centroid):** Compute `c` once and freeze it: + - Protected dims (< `matryoshkaProtectedDim`): copy from m1 (domain invariant). + - Free dims (>= `matryoshkaProtectedDim`): `c[i] = (m1[i] + m2[i]) / 2`. + - This frozen `c` is never recalculated. All future candidates in the Matryoshka unwind are + evaluated relative to this frozen platform. + - Return `Metroid { m1, m2, c }`; if no valid m2 found, return + `{ m1, m2: null, c: null, knowledgeGap: true }` + +- [ ] **P1-M2:** Implement Matryoshka dimensional unwinding in `cortex/MetroidBuilder.ts` + - After the initial Metroid construction, progressively expand the antithesis search into deeper + embedding layers by shifting the protected dimension boundary outward one Matryoshka tier at a + time. + - At each new tier, find a new `m2` candidate via cosine-opposite medoid search in the expanded + free dimensions. + - Evaluate each candidate against the **frozen** `c` (not a recomputed centroid). If close + enough to `c`, accept and freeze this step; take the next conceptual leap. If not, + continue unwinding. + - Stop when the protected dimension floor is reached or a satisfactory `m2` is accepted. + - If no satisfactory `m2` is found at any layer, return `knowledgeGap: true`. + +- [ ] **P1-M3:** Add MetroidBuilder test coverage + - `tests/cortex/MetroidBuilder.test.ts` + - Test m1 selection: the medoid minimising distance to q is chosen (not the centroid) + - Test m2 selection: medoid of cosine-opposite set β€” not merely nearest semantically-opposing node + - Test centroid computation: protected dims copied from m1; free dims averaged element-wise + - Test centroid is frozen: subsequent unwinding steps do not recompute c + - Test dimensional unwinding: search expands progressively through Matryoshka layers + - Test knowledge gap: when no valid m2 exists in any layer, returns `knowledgeGap: true` + - Test protected dimensions are never searched for antithesis + - Test determinism: same inputs always produce same Metroid + +**Exit Criteria:** MetroidBuilder constructs valid Metroids (m1 via medoid search, m2 via +cosine-opposite medoid of the top-scoring candidates, c computed once and never recomputed during +Matryoshka unwinding) and correctly detects knowledge gaps. + +--- + +### P1-N: Knowledge Gap Detection & Curiosity Probe (DELIVERS: epistemic honesty) + +**Why:** When MetroidBuilder cannot find m2, the system must acknowledge its knowledge boundary rather than hallucinating. The curiosity probe mechanism enables distributed learning by broadcasting the gap to peers. + +- [ ] **P1-N1:** Implement `cortex/KnowledgeGapDetector.ts` + - Accept MetroidBuilder result; if `knowledgeGap: true`, emit a `KnowledgeGap` DTO + - `KnowledgeGap { topicMedoidId: Hash, queryEmbedding: Float32Array, dimensionalBoundary: number, timestamp: string }` + - This DTO is returned to the caller as part of `QueryResult` + +- [ ] **P1-N2:** Implement curiosity probe construction in `cortex/KnowledgeGapDetector.ts` + - Build `CuriosityProbe { m1, partialMetroid, queryContext, knowledgeBoundary, mimeType, modelUrn }` + - `mimeType`: MIME type of embedded content (e.g. `text/plain`). Enables receiving peers to validate content-type compatibility before comparing graph sections. + - `modelUrn`: URN of the embedding model (e.g. `urn:model:onnx-community/embeddinggemma-300m-ONNX:v1`) sourced from the active `ModelProfile.modelId`. Peers **must** reject probes whose `modelUrn` does not match a model they support β€” accepting fragments from a different embedding model would produce incommensurable similarity scores at Matryoshka layer boundaries. + - Store probe locally for broadcast via P2P layer (see P2-G) + - Do not broadcast immediately β€” queue for the P2P sharing layer + +- [ ] **P1-N3:** Upgrade `cortex/QueryResult.ts` + - Add `knowledgeGap?: KnowledgeGap` field β€” present when MetroidBuilder failed to find m2 + - Document that callers must check this field before treating results as epistemically complete + +- [ ] **P1-N4:** Add knowledge gap test coverage + - `tests/cortex/KnowledgeGapDetector.test.ts` + - Test that a KnowledgeGap DTO is produced when MetroidBuilder returns `knowledgeGap: true` + - Test that a CuriosityProbe is constructed with correct fields including `mimeType` and `modelUrn` + - Test that `modelUrn` is derived from `ModelProfile.modelId` (not hardcoded) + - Test that QueryResult includes the KnowledgeGap when present + - Test that queries against a rich corpus do NOT produce false-positive knowledge gaps + +**Exit Criteria:** System correctly signals knowledge boundaries; callers can distinguish epistemically complete from incomplete results. + +--- + +### P1-E: Full Query Orchestrator (DELIVERS: dialectical retrieval) + +**Why:** This is the "aha" moment β€” return memories in natural narrative order through the resident hotpath via dialectical Metroid exploration, with dynamic, sublinear expansion bounds. + +> **Note on scope:** The existing `cortex/Query.ts` is a flat top-K scorer that does not use MetroidBuilder, Hebbian edge traversal, or cosine-similarity-bounded subgraph expansion. It must be **substantially reworked** β€” not merely extended β€” to implement the dialectical pipeline described below. The same applies to `cortex/QueryResult.ts`. Do not attempt to preserve the flat-scoring code path; it is superseded entirely. + +- [ ] **P1-E1:** Rewrite `cortex/Query.ts` (full dialectical version) + - Use resident-first hierarchical ranking to select topic medoid (m1) + - Call `MetroidBuilder` to construct `{ m1, m2, c }` + - If knowledge gap detected, include in result and continue with partial Metroid (m1 only) + - Use centroid `c` as the primary scoring anchor for page selection + - Derive dynamic subgraph bounds from `HotpathPolicy` (`maxSubgraphSize`, `maxHops`, `perHopBranching`) + - Call `MetadataStore.getInducedNeighborSubgraph(seedPages, maxHops)` using dynamic `maxHops`; traverse edges using Hebbian weights for tour distance (not cosine similarity) + - Call `OpenTSPSolver.solve(subgraph)` + - Return ordered page list via coherent path + - **Query cost meter:** count vector operations; early-stop and return best-so-far if cost exceeds Williams-derived budget + - Include provenance metadata (hop count, edge weights, subgraph size, cost, Metroid details) + +- [ ] **P1-E2:** Rewrite `cortex/QueryResult.ts` + - Add `coherencePath: Hash[]` (ordered page IDs) + - Add `metroid?: { m1: Hash; m2: Hash | null; centroid: Float32Array | null }` (Metroid used for this query) + - Add `knowledgeGap?: KnowledgeGap` (if antithesis discovery failed) + - Add `provenance: { subgraphSize: number; hopCount: number; edgeWeights: number[]; vectorOpCost: number; earlyStop: boolean }` + +- [ ] **P1-E3:** Add full query test coverage + - `tests/cortex/Query.test.ts` (upgrade) + - Test subgraph expansion stays within `maxSubgraphSize` + - Test TSP ordering + - Test Metroid is built and included in provenance + - Test knowledge gap is returned when antithesis not found + - Test provenance metadata + - Test early-stop fires when cost budget exceeded + +**Exit Criteria:** Queries return dialectically balanced, coherent context chains through the resident hotpath; MetroidBuilder active; knowledge gaps surfaced. + +--- + +### P1-F: Integration Test (Hierarchical + Dialectical) + +**Why:** Validate v0.5 completeness including resident-first routing, MetroidBuilder, and dialectical subgraph bounds. + +- [ ] **P1-F1:** Upgrade `tests/integration/IngestQuery.test.ts` + - Verify hierarchical structures exist after ingest + - Verify hotpath entries exist for hierarchy prototypes after ingest + - Verify queries build a valid Metroid `{ m1, m2, c }` + - Verify queries return coherent paths through resident hotpath + - Verify dynamic subgraph bounds honoured (no expansion beyond `maxSubgraphSize`) + - Verify knowledge gap is correctly signalled when corpus is sparse + - Compare dialectical retrieval vs flat ranking (show epistemic breadth improvement) + +**Exit Criteria:** Integration test demonstrates dialectically balanced retrieval with resident-first routing and knowledge gap detection. + +--- + +## 🟒 Medium Priority β€” Ship v1.0 (Background Consolidation + Smart Sharing) + +These items add idle background maintenance and privacy-safe interest sharing. Together they deliver v1.0's discovery value. + +### P2-A: Idle Scheduler (UNBLOCKS: Daydreamer operations) + +**Why:** Need cooperative background loop that doesn't block foreground. + +- [ ] **P2-A1:** Implement `daydreamer/IdleScheduler.ts` + - Loop via `requestIdleCallback` (browser) or `setImmediate` (Node) + - Interruptible (yield after N ms of work) + - CPU budget awareness (pause if main thread busy) + - Task queue (prioritize high-value work) + +- [ ] **P2-A2:** Add scheduler test coverage + - `tests/daydreamer/IdleScheduler.test.ts` + - Test cooperative yielding + - Test interruption doesn't corrupt state + +**Exit Criteria:** Background loop runs without blocking UI. + +--- + +### P2-B: Hebbian Updater (DELIVERS: connection plasticity) + +**Why:** Strengthen useful connections, decay unused ones. Edge changes alter Οƒ(v) values and can trigger hotpath promotions or evictions. + +- [ ] **P2-B1:** Implement `daydreamer/HebbianUpdater.ts` + - LTP: strengthen edges traversed during successful queries + - LTD: decay all edges by small factor each pass + - Prune: remove edges below threshold; keep Metroid degree within `HotpathPolicy`-derived bounds + - After LTP/LTD: recompute Οƒ(v) for all nodes whose incident edges changed (via `SalienceEngine.batchComputeSalience`) + - Run promotion/eviction sweep for changed nodes via `SalienceEngine.runPromotionSweep` + - Update `MetadataStore.putEdges` + +- [ ] **P2-B2:** Add Hebbian test coverage + - `tests/daydreamer/HebbianUpdater.test.ts` + - Test strengthen increases weight + - Test decay decreases weight + - Test pruning removes weak edges and keeps degree within bounds + - Test that salience is recomputed for changed nodes + - Test that promotion sweep runs after LTP increases salience above weakest resident + +**Exit Criteria:** Edge weights adapt based on usage; salience and hotpath updated accordingly. + +--- + +### P2-C: Full Neighbor Graph Recalc (DELIVERS: graph maintenance) + +**Why:** Incremental fast semantic neighbor insert is approximate; need periodic full recalc. Recalc batch size must be bounded by H(t)-derived maintenance budget to avoid blocking the idle loop. + +- [ ] **P2-C1:** Implement `daydreamer/FullNeighborRecalc.ts` + - Query `MetadataStore.needsNeighborRecalc(volumeId)` for dirty volumes; prioritise dirtiest first + - Load all pages in volume; compute pairwise similarities + - Bound batch: process at most `HotpathPolicy.computeCapacity(graphMass)` pairwise comparisons per idle cycle (O(√(t log t))) + - Select policy-derived max neighbors for each page; update `MetadataStore.putSemanticNeighbors` + - Clear dirty flag via `MetadataStore.clearNeighborRecalcFlag` + - Recompute Οƒ(v) for affected nodes via `SalienceEngine.batchComputeSalience`; run promotion sweep + +- [ ] **P2-C2:** Add neighbor graph recalc test coverage + - `tests/daydreamer/FullNeighborRecalc.test.ts` + - Test dirty flag cleared after recalc + - Test neighbor quality improved vs fast insert + - Test batch size respects O(√(t log t)) limit per cycle + - Test salience recomputed and promotion sweep runs after recalc + +**Exit Criteria:** Dirty volumes are recalculated in background within bounded compute budget; salience updated. + +--- + +### P2-D: Prototype Recomputer (DELIVERS: prototype quality) + +**Why:** Keep volume/shelf prototypes accurate as pages/books change. Prototype updates change which entries should occupy the volume and shelf tier quotas. + +- [ ] **P2-D1:** Implement `daydreamer/PrototypeRecomputer.ts` + - Recompute volume medoids (select medoid page per volume) + - Recompute volume centroids (average of book embeddings) + - Recompute shelf routing prototypes + - Update vectors in `VectorStore` (append new, update offsets) + - After recomputing each level: recompute salience for affected representative entries via `SalienceEngine`; run tier-quota promotion/eviction for that tier + +- [ ] **P2-D2:** Add prototype recomputer test coverage + - `tests/daydreamer/PrototypeRecomputer.test.ts` + - Test medoid selection algorithm + - Test centroid computation + - Test that tier-quota hotpath entries are updated after prototype recomputation + +**Exit Criteria:** Prototypes stay accurate over time; tier quota entries reflect current prototypes. + +--- + +### P2-E: Integration Test (Background Consolidation) + +**Why:** Validate Daydreamer improves system health and hotpath stays consistent. + +- [ ] **P2-E1:** Implement `tests/integration/Daydreamer.test.ts` + - Ingest corpus + - Run queries (generate edge traversals and PageActivity updates) + - Run Daydreamer for N passes + - Verify edge weights updated + - Verify dirty volumes recalculated + - Verify prototypes updated + - Verify resident count never exceeds H(t) after any Daydreamer pass + +**Exit Criteria:** Daydreamer demonstrably maintains system health; Williams Bound invariant holds. + +--- + +### P2-F: Community Detection & Graph Coverage Quotas (DELIVERS: topic-diverse hotpath) + +**Why:** Without community detection, a single dense topic can fill the entire page-tier quota, crowding out unrelated memories. Community quotas ensure the hotpath is both hot (high salience) and diverse (topic-representative). + +- [ ] **P2-F1:** Add community detection to `daydreamer/ClusterStability.ts` + - Implement lightweight label propagation on the semantic neighbor graph + - Run during idle passes when dirty-volume flags indicate meaningful structural change + - Store community labels in `PageActivity.communityId` via `MetadataStore.putPageActivity` + - Rerun when graph topology changes significantly (post-split, post-merge, post-full-recalc) + +- [ ] **P2-F2:** Wire community labels into `SalienceEngine` promotion/eviction + - `selectEvictionTarget` uses `communityId` to find weakest resident in the community bucket + - Promotion checks community quota remaining before admitting + - If community quota is full: candidate must beat weakest resident in that community + - If community is unknown (`communityId` not yet set): place node in temporary pending pool borrowing from page-tier budget + - Empty communities release their slots back to the page-tier budget + +- [ ] **P2-F3:** Add community-aware eviction tests + - `tests/daydreamer/ClusterStability.test.ts` + - Test that a single dense community cannot consume all page-tier hotpath slots + - Test that a new community (previously unknown) receives at least one slot + - Test that an empty community releases its slots correctly + - Test that label propagation converges and produces stable community assignments + +**Exit Criteria:** Community-aware hotpath quotas active; topic diversity enforced; label propagation stable. + +--- + +### P2-G: Curiosity Broadcasting & Smart Interest Sharing (DELIVERS: distributed learning without hallucination) + +**Why:** When knowledge gaps are detected, CORTEX must be able to broadcast the incomplete Metroid as a curiosity probe to connected peers. Peers respond with relevant fragments, enabling collaborative learning. Additionally, interest sharing is a core product value for both app and library surfaces. v1 must share public-interest graph sections while preventing personal data leakage. + +- [ ] **P2-G0:** Implement `sharing/CuriosityBroadcaster.ts` + - Consume pending `CuriosityProbe` objects queued by `KnowledgeGapDetector` + - Serialize and broadcast to connected peers via P2P transport + - Handle responses: deserialize incoming graph fragments; pass to `SubgraphImporter` for integration + - Rate-limit broadcasts to prevent spam + - Include `knowledgeBoundary` field in probe so peers can target search precisely + +- [ ] **P2-G1:** Implement `sharing/EligibilityClassifier.ts` + - Classify candidate nodes as share-eligible vs blocked before export + - Detect identity/PII-bearing content (person-specific identifiers, credentials, financial/health traces) + - Emit deterministic eligibility decisions with reason codes for auditability + +- [ ] **P2-G2:** Implement `sharing/SubgraphExporter.ts` + - Build topic-scoped graph slices from eligible nodes only + - For curiosity responses: select graph fragment relevant to the received probe's `knowledgeBoundary` + - Preserve node/edge signatures and provenance + - Strip or coarsen personal metadata fields that are not needed for discovery + +- [ ] **P2-G3:** Implement `sharing/PeerExchange.ts` and `sharing/SubgraphImporter.ts` + - Opt-in peer exchange over P2P transport + - Verify signatures and schema on import; reject invalid or tampered payloads + - Merge imported slices into discovery pathways without exposing sender identity metadata + - After import, retry MetroidBuilder for any pending knowledge gaps that may be resolved by new data + +- [ ] **P2-G4:** Add sharing safety and discovery tests + - `tests/sharing/EligibilityClassifier.test.ts` + - `tests/sharing/CuriosityBroadcaster.test.ts` + - `tests/sharing/SubgraphExchange.test.ts` + - Assert blocked nodes are never exported; assert imported fragments are discoverable via query + - Assert that after receiving a response to a curiosity probe, MetroidBuilder can now construct m2 for the previously-gapped topic + +**Exit Criteria:** v1 can broadcast curiosity probes for knowledge gaps, receive graph fragments from peers, retry MetroidBuilder with new data, and exchange signed public-interest slices with PII blocking. + +--- + +### P3-H: GitHub sync smoke test (VALIDATES: automation pipeline) + +**Why:** Ensure the `TODO.md` β†’ GitHub issue sync path works end-to-end (milestones/labels/issues created by `scripts/sync-github-project.mjs`). + +- [ ] **P3-H1:** Push this change to `main` so the sync workflow runs +- [ ] **P3-H2:** Verify a new issue is created automatically from this task group +- [ ] **P3-H3:** Mark this task complete once the issue exists and the workflow succeeded + +**Exit Criteria:** A GitHub issue was created from this task group after pushing to `main`, and the sync workflow succeeded. + +--- + +## πŸ”΅ Lower Priority β€” Polish & Ship + +These items improve quality, performance, and developer experience. Not blockers for v1.0 launch. + +### P3-A: WebGL Embedding Provider + +**Why:** Explicit `webgl` fallback for systems without WebGPU/WebNN. + +- [ ] **P3-A1:** Implement `embeddings/OrtWebglEmbeddingBackend.ts` + - Use ONNX Runtime Web with explicit `webgl` execution provider + - Implement `EmbeddingBackend` interface + - Add to `ProviderResolver` candidate sets + +- [ ] **P3-A2:** Add WebGL provider test coverage + - `tests/embeddings/OrtWebglEmbeddingBackend.test.ts` + +**Exit Criteria:** `webgl` backend available for systems without WebGPU. + +--- + +### P3-B: Experience Replay + +**Why:** Simulate queries during idle time to reinforce connection patterns. + +- [ ] **P3-B1:** Implement `daydreamer/ExperienceReplay.ts` + - Sample random or recent queries + - Execute query (triggers edge traversals) + - Mark traversed edges for LTP strengthening + +- [ ] **P3-B2:** Add experience replay test coverage + - `tests/daydreamer/ExperienceReplay.test.ts` + +**Exit Criteria:** Daydreamer reinforces memory patterns. + +--- + +### P3-C: Cluster Stability (full implementation) + +**Why:** Detect and fix unstable clusters (split oversized, merge undersized). The community detection added in P2-F is a subset of this module; here we add the full split/merge machinery. + +- [ ] **P3-C1:** Complete `daydreamer/ClusterStability.ts` + - Detect high-variance volumes (unstable) + - Trigger split (K-means with K=2) + - Detect low-count volumes + - Trigger merge with nearest neighbor volume + - Re-run community detection and update PageActivity after split/merge + +- [ ] **P3-C2:** Add cluster stability test coverage + - `tests/daydreamer/ClusterStability.test.ts` (extend from P2-F) + - Test split produces two balanced volumes + - Test merge produces one combined volume + - Test community labels updated after structural change + +**Exit Criteria:** Clusters stay balanced over time; community labels stay current. + +--- + +### P3-D: Benchmark Suite + +**Why:** Measure performance, validate Williams Bound invariants, and track regressions. + +- [ ] **P3-D1:** Implement real-provider benchmarks + - `tests/benchmarks/TransformersJsEmbedding.bench.ts` + - Throughput (embeddings/sec) for various batch sizes + +- [ ] **P3-D2:** Implement query latency benchmarks + - `tests/benchmarks/QueryLatency.bench.ts` + - Latency vs corpus size (100 pages, 1K pages, 10K pages) + +- [ ] **P3-D3:** Implement storage overhead benchmarks + - `tests/benchmarks/StorageOverhead.bench.ts` + - Disk usage vs page count + +- [ ] **P3-D4:** Implement hotpath scaling benchmarks + - `tests/benchmarks/HotpathScaling.bench.ts` + - Synthetic graphs at 1K, 10K, 100K, 1M nodes+edges + - Measure: resident set size vs H(t), query latency vs corpus size, promotion/eviction throughput + - **Assert:** resident count never exceeds H(t); query cost scales sublinearly with corpus size + - Assert: H(t) values match expected sublinear curve at each scale point + +- [ ] **P3-D5:** Record baseline measurements + - Add `benchmarks/BASELINES.md` with results from all benchmarks + - Include H(t) curve data at 1K/10K/100K/1M + +**Exit Criteria:** Benchmark suite exists; baselines recorded; Williams Bound invariants asserted. + +--- + +### P3-E: CI Hardening + +**Why:** Ensure tests run reliably in CI; enforce both model-derived and policy-derived numeric guards. + +- [ ] **P3-E1:** Add GitHub Actions workflow + - `.github/workflows/ci.yml` + - Run `npm run build`, `npm run lint`, `npm run test:unit`, `npm run guard:model-derived` + +- [ ] **P3-E2:** Define Electron runtime gate policy + - Document GPU/graphics requirements + - Decide CI runner capabilities (software vs hardware rendering) + - Update `scripts/run-electron-runtime-tests.mjs` gate logic + +- [ ] **P3-E3:** Add hotpath policy constants guard + - Extend `scripts/guard-model-derived.mjs` or add `scripts/guard-hotpath-policy.mjs` + - Scan for numeric literals assigned to hotpath policy fields outside `core/HotpathPolicy.ts` + - Add as required CI gate alongside `guard:model-derived` + - Add `npm run guard:hotpath-policy` script to `package.json` + +**Exit Criteria:** CI runs on every PR; merge blocked if tests or guards fail; both model-derived and policy-derived constants are enforced. + +--- + +### P3-F: Documentation + +**Why:** Users need to know how to integrate CORTEX. + +- [ ] **P3-F1:** Update `docs/api.md` + - Document `ingestText(...)` API + - Document `query(...)` API + - Document `QueryResult` structure + +- [ ] **P3-F2:** Update `docs/development.md` + - Add troubleshooting section + - Add performance tuning guide + +- [ ] **P3-F3:** Add architecture diagrams + - Data flow: ingest path + - Data flow: query path + - Module dependency graph + +**Exit Criteria:** API docs complete; developer guide useful. + +--- + +### P3-G: Product Surface UX Contract + +**Why:** v1.0 needs an explicit UX contract for the standalone app while keeping the library surface headless and integration-first. + +- [ ] **P3-G1:** Add `docs/product-surfaces.md` + - Define app-vs-library scope, boundaries, and non-goals + - Define standalone extension user journey: passive capture -> search -> revisit + - Define what remains local-only and private in the app shell + +- [ ] **P3-G2:** Add standalone search UX checklist to `docs/product-surfaces.md` + - Search-first information architecture (query bar, results, lightweight metrics) + - Result-card contract (title, URL, snippet/thumbnail, visit recency, relevance signal) + - UX states: empty index, no matches, loading/indexing, error recovery + +- [ ] **P3-G3:** Add model-mode UX contract to `docs/product-surfaces.md` + - Nomic mode: multimodal recall (text + images in shared latent space) + - Gemma mode: fine-grained text recall (no image embedding) + - UI copy rules that make image-recall availability explicit by mode + +- [ ] **P3-G4:** Add rabbit-hole recall acceptance checklist + - Vague text recollection scenario recovers a previously visited page path + - Vague visual recollection scenario recovers a previously seen image when Nomic mode is enabled + - Add manual validation steps for model toggle behavior and capability messaging + +**Exit Criteria:** Standalone app UX contract and model-mode behavior are documented; library boundary remains explicit and implementation-ready. + +--- + +## πŸ“‹ Summary by Phase + +| Phase | Items | Status | Blocking | +|-------|-------|--------|----------| +| v0.1 (Minimal Viable) | 30 tasks (P0-A through P0-G + P0-E + P0-X) | 🟑 In Progress (P0-A, P0-F, P0-G complete; P0-X architectural rename pending) | User cannot use system correctly; P0-X blocks MetroidBuilder | +| v0.5 (Hierarchical + Dialectical) | 20 tasks (P1-A through P1-F + P1-M + P1-N) | ❌ Not started | Blocked by v0.1 | +| v1.0 (Background Consolidation + Smart Sharing) | 20 tasks (P2-A through P2-G) | ❌ Not started | Blocked by v0.5 | +| Polish & Ship | 21 tasks (P3-A through P3-G) | ❌ Not started | Not blocking v1.0 | + +**Total:** ~91 actionable tasks + +--- + +## Quick Reference: Next Tasks to Unblock Everything + +If you're reading this and want to know "what do I work on right now?", here's the answer: + +**Immediate (unblock MetroidBuilder):** +1. **P0-X1–X7:** Fix architectural naming drift (`MetroidNeighbor` β†’ `SemanticNeighbor` and related renames) + +**After P0-X (complete v0.1):** +2. **P0-B1:** Implement `hippocampus/Chunker.ts` +3. **P0-C1/C2:** Implement `hippocampus/PageBuilder.ts` and `hippocampus/Ingest.ts` +4. **P0-D1:** Implement `cortex/Query.ts` (minimal) + +**After v0.1 (start v0.5):** +5. **P1-A1:** Implement `hippocampus/HierarchyBuilder.ts` +6. **P1-C1:** Implement `hippocampus/FastNeighborInsert.ts` +7. **P1-M1/M2:** Implement `cortex/MetroidBuilder.ts` with Matryoshka unwinding +8. **P1-N1/N2:** Implement `cortex/KnowledgeGapDetector.ts` +9. **P1-D1:** Implement `cortex/OpenTSPSolver.ts` +10. **P1-E1:** Rewrite `cortex/Query.ts` to full dialectical orchestrator (substantial rework; not backward-compatible with flat top-K version) + +--- + +## Notes + +- **Dependencies:** Items are ordered so that completing tasks in sequence minimises blocked work. P0-X (naming drift fix) must precede MetroidBuilder. P0-F and P0-G (Williams Bound foundation) must precede all hotpath-aware modules. +- **Estimates:** Each P0/P1/P2 task is roughly 1-4 hours for an experienced developer familiar with the codebase. +- **Testing:** Every implementation task should be accompanied by test coverage (explicitly called out). +- **TDD Approach:** Write failing tests first, then implement to green. +- **Documentation Sync:** Update PLAN.md module status as tasks complete. +- **Williams Bound Invariant:** The resident count must never exceed H(t). Every test that touches the hotpath should assert this. +- **Policy constants:** Never hardcode hotpath constants outside `core/HotpathPolicy.ts`. P3-E3 will add a guard to enforce this automatically; until then, enforce by convention. +- **Metroid vs medoid vs semantic neighbor graph:** These are three distinct concepts. `Metroid` = dialectical probe `{ m1, m2, c }` (ephemeral, query-time). `medoid` = cluster representative node. Semantic neighbor graph = sparse proximity edges used for BFS subgraph expansion. Do not conflate them. See P0-X for the code rename tasks that fix the current conflation.