P1 (v0.5): Hierarchical + dialectical memory — full implementation by Copilot · Pull Request #69 · devlux76/cortex

Copilot · 2026-03-13T22:14:28Z

Original prompt

This section details on the original issue you should resolve

<issue_title>P1</issue_title>
<issue_description>All P1 tasks, issues and sub issues belong here, either directly or by reference. Please subtask every issue with P1 in its name for ease of tracking.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes P1 #68

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

…d upgrade Ingest - HierarchyBuilder: cluster pages→books→volumes→shelves using cosine-distance medoid selection; persist each tier; run promotion sweeps after each level - FastNeighborInsert: build sparse SemanticNeighbor graph (NOT Hebbian); forward + symmetric reverse edges; degree-bounded with eviction; marks affected volumes dirty; runs promotion sweep for new pages - Ingest: replace manual single-book construction with buildHierarchy; add insertSemanticNeighbors call; extend IngestResult with volumes/shelves - Tests: 11 HierarchyBuilder tests + 8 FastNeighborInsert tests (230 total pass) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- cortex/OpenTSPSolver.ts: greedy nearest-neighbor open-path TSP heuristic with deterministic start (lex smallest node) and tiebreaker - cortex/MetroidBuilder.ts: dialectical probe builder selecting m1 (thesis) and m2 (antithesis medoid) with Matryoshka dimensional unwinding across standard tiers [32,64,128,256,512,768,1024,2048]; computes frozen synthesis centroid c - cortex/KnowledgeGapDetector.ts: detectKnowledgeGap + buildCuriosityProbe with deterministic SHA-256 probeId - cortex/QueryResult.ts: add coherencePath, metroid, knowledgeGap fields - cortex/Query.ts: populate new QueryResult fields with flat-query defaults - 32 new tests across OpenTSPSolver, MetroidBuilder, KnowledgeGapDetector Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add cortex/Ranking.ts: rankPages/rankBooks/rankVolumes/rankShelves use cosine similarity against entity representative vectors; spillToWarm('page') scores the full corpus as fallback - Rewrite cortex/Query.ts (P1-E1): full dialectical orchestrator - HOT path: rankPages() over resident hotpath entries - Warm spill: spillToWarm('page') when hot path is insufficient - buildMetroid() with book medoid + page candidates - detectKnowledgeGap() from metroid result - getInducedNeighborSubgraph() + solveOpenTSP() for coherence path - queryHitCount increment + runPromotionSweep() - QueryResult now fully populated (coherencePath, metroid, knowledgeGap) - Add tests/cortex/Ranking.test.ts: 12 tests covering empty inputs, cosine scoring, descending order, topK, spillToWarm all tiers - Update tests/cortex/Query.test.ts: add assertions for new fields (coherencePath, metroid, knowledgeGap); preserve all existing behavioral checks; update describe label to reflect new orchestrator All 274 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tor, P1-B Ranking, P1-E Query rewrite, P1-F integration tests Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com>

Copilot

Pull request overview

Implements the v0.5 (P1) “hierarchical + dialectical memory” slice by adding hierarchy construction (Pages→Books→Volumes→Shelves), semantic neighbor graph storage/retrieval, dialectical probing (Metroid + knowledge gap), coherence path derivation (open TSP), and updated query orchestration, alongside substantial new unit/integration test coverage.

Changes:

Adds hierarchy builder + semantic neighbor insertion to ingest, with IndexedDB schema updates for the neighbor graph.
Introduces cortex modules for ranking, dialectical Metroid construction, knowledge gap detection, and coherence path solving; rewrites cortex/query.
Adds/updates tests to cover hierarchy persistence, neighbor graph behavior, ranking/query behavior, and knowledge gap signaling.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/integration/IngestQuery.test.ts	Adds v0.5 integration tests for hierarchy, neighbors, Williams bound, knowledge-gap signaling.
tests/hippocampus/HierarchyBuilder.test.ts	New unit tests for hierarchy building + persistence + hotpath admission.
tests/hippocampus/FastNeighborInsert.test.ts	New unit tests for semantic neighbor insertion behavior and constraints.
tests/cortex/Ranking.test.ts	New unit tests for ranking and warm spill behavior.
tests/cortex/Query.test.ts	Updates query tests for new `QueryResult` fields and dialectical behavior.
tests/cortex/OpenTSPSolver.test.ts	New unit tests for deterministic greedy open-path TSP.
tests/cortex/MetroidBuilder.test.ts	New unit tests for Metroid construction + determinism + Matryoshka behavior.
tests/cortex/KnowledgeGapDetector.test.ts	New unit tests for knowledge gap detection and curiosity probe creation.
tests/SalienceEngine.test.ts	Updates mock `MetadataStore` to renamed neighbor-graph APIs.
tests/Persistence.test.ts	Updates persistence tests to renamed neighbor-graph types/APIs.
storage/IndexedDbMetadataStore.ts	Bumps DB version; renames neighbor store to `neighbor_graph`; renames APIs.
hippocampus/Ingest.ts	Integrates hierarchy building + semantic neighbor insertion into ingest pipeline.
hippocampus/HierarchyBuilder.ts	New hierarchy builder (books/volumes/shelves) + prototype vector creation.
hippocampus/FastNeighborInsert.ts	New semantic neighbor insertion implementation + dirty-volume recalc flagging.
cortex/Ranking.ts	New ranking functions for pages/books/volumes/shelves + warm spill for pages.
cortex/QueryResult.ts	Extends query result schema with coherencePath/metroid/knowledgeGap fields.
cortex/Query.ts	Rewrites query orchestration to use ranking→metroid→gap→subgraph→TSP pipeline.
cortex/OpenTSPSolver.ts	New deterministic greedy open-path TSP heuristic over neighbor subgraph.
cortex/MetroidBuilder.ts	New Metroid builder with Matryoshka unwinding and centroid freezing.
cortex/KnowledgeGapDetector.ts	New knowledge gap detection + curiosity probe builder.
core/types.ts	Renames Metroid neighbor graph types/APIs to SemanticNeighbor equivalents.

devlux76 · 2026-03-13T23:15:14Z

+  const pageChunks = chunkArray(pageIds, PAGES_PER_BOOK);
+  const books: Book[] = [];
+
+  for (const chunk of pageChunks) {
+    const sortedChunk = [...chunk].sort();
+    const bookId = await hashText(sortedChunk.join("|"));
+
+    const chunkVectors = chunk.map((id) => {
+      const idx = pageIds.indexOf(id);
+      return pageVectors[idx];
+    });
+


@copilot Also verify prev/next edges are added to nodes that follow one another directly from slicing.

Done in de785cd. HierarchyBuilder.buildHierarchy now inserts SemanticNeighbor edges between every consecutive pair of pages within each book slice — both the forward (A→B) and reverse (B→A) edges — using the actual cosine similarity. These document-order adjacency edges bypass the cosine-distance cutoff (adjacent text chunks of the same source are always related) but still respect a per-direction degree cap (16). A Map<Hash, Float32Array> was also introduced to replace the O(n²) indexOf calls. The FastNeighborInsert test fixture was updated to build pages directly without going through ingestText/buildHierarchy, keeping those tests isolated from the adjacency-edge insertion.

devlux76 · 2026-03-13T23:17:05Z

+    const chunkVectors = chunk.map((id) => {
+      const idx = pageIds.indexOf(id);
+      return pageVectors[idx];
+    });
+
+    const medoidIdx = selectMedoidIndex(chunkVectors);
+    const medoidPageId = chunk[medoidIdx];
+
+    const book: Book = { bookId, pageIds: chunk, medoidPageId, meta: {} };
+    await metadataStore.putBook(book);
+    books.push(book);
+  }
+
+  await runPromotionSweep(books.map((b) => b.bookId), metadataStore, policy);
+
+  // -------------------------------------------------------------------------
+  // Level 2: Books → Volumes
+  // -------------------------------------------------------------------------
+  const bookChunks = chunkArray(books, BOOKS_PER_VOLUME);
+  const volumes: Volume[] = [];
+
+  for (const bookChunk of bookChunks) {
+    const sortedBookIds = bookChunk.map((b) => b.bookId).sort();
+    const volumeId = await hashText(sortedBookIds.join("|"));
+
+    const medoidVectors = bookChunk.map((b) => {
+      const idx = pageIds.indexOf(b.medoidPageId);
+      return pageVectors[idx];
+    });


@copilot Ok I agree with this, just make sure to fix it using your best judgement and update the DESIGN.md to reflect this design decision. Insertion needs to be fast and lightweight. Daydreaming exists to find connections we never noticed before.

Fixed in b2fd983. The Map<Hash, Float32Array> (O(1) lookups) was already landed in the previous commit. DESIGN.md now explicitly documents the fast/lightweight ingest strategy: only document-order adjacency edges are created at ingest time using O(1) precomputed maps, and the Daydreamer is responsible for discovering the semantic cross-connections we never noticed before via its idle-pass graph walks.

devlux76 · 2026-03-13T23:20:49Z

+
+  if (oppositeSet.length === 0) return null;
+
+  const medoidIdx = findMedoidIndex(oppositeSet.map((s) => s.candidate.embedding));


@copilot Use your best judgement to resolve the discrepancy please.

Fixed in b2fd983. searchM2 now calls findMedoidIndex(oppositeSet.map((s) => s.candidate.embedding.slice(protectedDim))) — the medoid is selected over the free-dimension slices only, matching the scoring used to build the opposite set.

…ice pages in HierarchyBuilder Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com>

devlux76 · 2026-03-13T23:46:34Z

@copilot Apparently the PR touched on some P0 issues that were being worked on concurrently. As a result there has been drift on main and this PR has merge conflicts. Rebase to head main and then roll forward applying the changes that still make sense in light of the higher priority P0 issues that were recently closed.

Sorry about that!

…move vectorBackend, ONE Book per ingest, DESIGN.md update Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com>

…e ARCHITECTURE-REVIEW.md and close-legacy-issues.yml Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com>

Copilot · 2026-03-14T00:05:33Z

@copilot Apparently the PR touched on some P0 issues that were being worked on concurrently. As a result there has been drift on main and this PR has merge conflicts. Rebase to head main and then roll...

Done in dc04d6f. The branch is now aligned with main's HEAD (619374d):

ARCHITECTURE-REVIEW.md and close-legacy-issues.yml deleted (removed on main)
TODO.md / PLAN.md / README.md / docs/api.md taken from main (P0-X tasks marked done, American spelling)
DESIGN.md stale "MetroidNeighbor is a naming error" notes removed (P0-X is done; only our P1 additions remain)
storage/IndexedDbMetadataStore.ts STORE constant renamed metroidNeighbors → neighborGraph; applyUpgrade restructured to match main's clean v3 section (neighbor_graph created after v2 stores)
core/types.ts "neighbour" → "neighbor" comment spelling fixed

All 280 tests pass, build and lint clean.

devlux76

Excellent work!

Initial plan

6eb4c2f

Copilot AI assigned Copilot and devlux76 Mar 13, 2026

Copilot started work on behalf of devlux76 March 13, 2026 22:14 View session

Copilot AI and others added 4 commits March 13, 2026 22:28

feat: P1-D OpenTSPSolver, P1-M MetroidBuilder, P1-N KnowledgeGapDetec…

eafff4c

…tor, P1-B Ranking, P1-E Query rewrite, P1-F integration tests Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] [P1] Subtask all P1 tasks and issues for tracking~~ P1 (v0.5): Hierarchical + dialectical memory — full implementation Mar 13, 2026

Copilot AI requested a review from devlux76 March 13, 2026 22:49

Copilot finished work on behalf of devlux76 March 13, 2026 22:49

devlux76 marked this pull request as ready for review March 13, 2026 22:51

Copilot AI review requested due to automatic review settings March 13, 2026 22:51

Copilot started reviewing on behalf of devlux76 March 13, 2026 22:51 View session

Copilot AI reviewed Mar 13, 2026

View reviewed changes

Copilot started work on behalf of devlux76 March 13, 2026 23:15 View session

fix: add prev/next SemanticNeighbor edges between consecutive book-sl…

de785cd

…ice pages in HierarchyBuilder Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com>

Copilot finished work on behalf of devlux76 March 13, 2026 23:32

Copilot started work on behalf of devlux76 March 13, 2026 23:32 View session

fix: FastNeighborInsert safety, MetroidBuilder m2 free-dim medoid, re…

b2fd983

…move vectorBackend, ONE Book per ingest, DESIGN.md update Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com>

Copilot finished work on behalf of devlux76 March 13, 2026 23:50

Copilot started work on behalf of devlux76 March 13, 2026 23:50 View session

chore: rebase onto main — align P0-X naming, remove stale docs, delet…

dc04d6f

…e ARCHITECTURE-REVIEW.md and close-legacy-issues.yml Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com>

Copilot finished work on behalf of devlux76 March 14, 2026 00:06

devlux76 approved these changes Mar 14, 2026

View reviewed changes

Merge branch 'main' into copilot/p1-subtask-issues

a58ddbc

github-actions Bot added layer: foundation Core types, model profiles, crypto layer: storage OPFS vector store and IndexedDB metadata store labels Mar 14, 2026

devlux76 mentioned this pull request Mar 14, 2026

P1 #68

Closed

devlux76 merged commit e6fc325 into main Mar 14, 2026
3 checks passed

devlux76 deleted the copilot/p1-subtask-issues branch March 14, 2026 00:38

devlux76 mentioned this pull request Mar 14, 2026

Priority List #74

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

P1 (v0.5): Hierarchical + dialectical memory — full implementation#69

P1 (v0.5): Hierarchical + dialectical memory — full implementation#69
devlux76 merged 9 commits intomainfrom
copilot/p1-subtask-issues

Copilot AI commented Mar 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

devlux76 Mar 13, 2026

Uh oh!

Copilot AI Mar 13, 2026

Uh oh!

devlux76 Mar 13, 2026

Uh oh!

Copilot AI Mar 13, 2026

Uh oh!

Uh oh!

Uh oh!

devlux76 Mar 13, 2026

Uh oh!

Copilot AI Mar 13, 2026

Uh oh!

Uh oh!

Uh oh!

devlux76 commented Mar 13, 2026

Uh oh!

Copilot AI commented Mar 14, 2026

Uh oh!

devlux76 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		if (oppositeSet.length === 0) return null;

		const medoidIdx = findMedoidIndex(oppositeSet.map((s) => s.candidate.embedding));

Conversation

Copilot AI commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comments on the Issue (you are @copilot in this section)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

devlux76 Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

devlux76 Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

devlux76 Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

devlux76 commented Mar 13, 2026

Uh oh!

Copilot AI commented Mar 14, 2026

Uh oh!

devlux76 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Mar 13, 2026 •

edited

Loading