feat(kb): AI-native representation Phase 1 — the ai:begin block foundation — 0.24.0 by Cain-Ish · Pull Request #13 · Cain-Ish/claude-code-plugin

Cain-Ish · 2026-06-02T16:43:15Z

Why

The KB is AI-to-AI (written by AI, read by AI). Prose forces every AI reader to re-derive structure on every read — "no shared intermediate" (Knows). SOTA agent memory is multi-granularity (TriMem: raw + atomic + synthesized), not prose-only. Design docs/specs/2026-06-02-ai-native-knowledge-representation-design.md; plan docs/plans/2026-06-02-ai-native-representation-phase1.md. The 3 open questions were web-researched and resolved (SOTA, not quick-win): flat-YAML block (model-preferred, token-efficient, no new dep), proposition-level retrieval unit, hybrid confidence (verbalized primary + corroboration).

What — Phase 1 (deterministic foundation + safety)

Makes a per-page  flat-YAML structured block (the "shared intermediate") a recognized, parsed, schema-validated, strip-safe construct:

new pure mcp/src/tools/ai-block.ts — parseAiBlock / stripAiBlock / validateAiBlock + per-type schemas (learnings/decisions/entities/issues/concepts/security);
parseDoc exposes doc.aiBlock; the body-[[link]]→related: fallback strips the block (block values are plain slugs, never [[links]]);
the block is excluded from every length/first-sentence count — firstSentence (reindex), FORGET wc -c + stub-floor, the search stub-penalty — so a uniform block can't skew FORGET scores or BM25;
knowledge_validate warns gently (not error) on a block missing a required field for its type.

MCP server → 2.5.0. Additive + back-compat: no block ⇒ behaviour unchanged. A graph-project regression test proves projection never clobbers the block.

Deferred (own phases): authoring at capture (extractor — Phase 1b); consumption (search weight/return + session-load injection + knowledge_fetch block tier + own embedding — Phase 2); dream/maintainer refresh + lint staleness + backfill (Phase 3).

State-check grounded (spec §5b)

Before building I mapped the live subsystems (forgetting / search-write / persona / automation) so the design didn't assume — that's where the length-exclusion + plain-slug + "authoring respects the explicit-invocation-only boundary (no auto-dispatch)" constraints came from.

Release-gate review — 4 findings, 0 false positives, all fixed + guarded

MEDIUM: an unterminated ai:begin made the shell FORGET strip eat the rest of the page → demoted a real page toward forgetting (fail-open). Now strips only on a complete block (matches the TS stripAiBlock no-op); guarded.
LOW ×3: AI_BLOCK_RE begin-tail hardened to same-line; validate message uses the resolved type; off-by-one byte count confined.

Verification

Full suite green (64 shell + vitest, 0 fail); lockstep 0.24.0 / MCP 2.5.0; validate + migration-row gates; 8 TDD tasks each RED→GREEN.

🤖 Generated with Claude Code

Reframe: the KB is AI-to-AI, so prose forces every reader to re-derive structure ("no shared intermediate"). Research-backed direction (approved): keep markdown + prose, but ADD a per-page machine-first, schema'd, atomic AI-block (the shared intermediate) — multi-granularity (TriMem): raw(episodic) + atomic(block) + synthesized(prose) + relations(edges). Per-type block schemas, first-class provenance/confidence, authored-not-projected, validated like the other marked regions, consumed by session-load + knowledge_search. Additive, flag-gated, reversible. Future track (0.24.0 candidate), orthogonal to the 0.23.x hierarchy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e-check grounding Resolved the 3 open questions via web research (SOTA over quick-win): - format: flat YAML key:value in the marked region (model-preferred, token-efficient, no new dep) — not JSON. - embedding: proposition-level retrieval unit (BM25 P1, own vector P2) — beats passage/sentence (Dense X Retrieval / AGRaME). - confidence: hybrid — LLM-verbalized primary (best-calibrated, 2026 study) + evidence/recurrence corroboration + recency. Added §5b integration constraints from a parallel state-check of the live subsystems (forgetting / search-write / persona / automation), so the design doesn't assume: - exclude the block from FORGET wc -c + stub-penalty length counts (else scores shift + a test breaks); - block field values are PLAIN SLUGS, never [[wikilinks]] (else they pollute the connectivity signal + the related: body-fallback); - firstSentence must strip ai:begin like graph:begin; - authoring respects the automation boundary: extractor authors at capture (auto), dream/maintainer refresh (explicit-invocation-only — no auto-dispatch); - knowledge_fetch gains a block-aware path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

8 tasks: pure ai-block module (parse/schema/validate/strip) -> parseDoc.aiBlock + plain-slug guard -> stub-penalty prose-only -> firstSentence strip -> validate gentle warning -> FORGET byte-count exclude -> graph-project safety -> build+gate+0.24.0. Authoring (extractor) + consumption (search/session-load) are Phase 1b/2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…he block Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…scription Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The ai:begin block is now a recognized/parsed/schema-validated/strip-safe construct across parseDoc, knowledge_validate, reindex firstSentence, FORGET byte-count, and the search stub-penalty. Additive/back-compat (no block => unchanged). Authoring (extractor) + consumption (search/session-load) are Phases 1b/2. Full suite green (64 shell + vitest). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ives) - MEDIUM: wiki-forget-score.sh now only strips the ai-block when a COMPLETE block exists (closing ai:end present); an unterminated ai:begin keeps the raw byte-exact count, matching the TS stripAiBlock no-op — fixes the fail-open where an unterminated block ate a real page's prose and demoted it toward FORGET. Guarded by a new unterminated case in test-wiki-forget-ai-block.sh. - LOW: AI_BLOCK_RE begin-tail is [^\n]*? (same-line marker close; a stray > in the annotation can't fold the tail into the body) — guarded by a new ai-block.test case. - LOW: knowledge_validate uses the resolved type (dir fallback) in the warning message, not the possibly-empty doc.type. Full suite green (64 shell + vitest). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Introduces a new per-page  flat-YAML “AI block” and threads it through the KB pipeline as a parsed/validated construct, while ensuring length-based heuristics (FORGET scoring, stub penalty, first-sentence extraction) can ignore it. This establishes a deterministic foundation for “AI-native” knowledge representation in the MCP KB server and the surrounding scripts/tests.

Changes:

Adds mcp/src/tools/ai-block.ts with parsing, stripping, and per-type required-field validation (+ tests).
Integrates ai-block awareness into doc parsing (parseDoc), validation warnings (knowledge_validate), reindex first-sentence extraction, search stub-penalty, and FORGET byte counting (+ regression tests).
Updates release/migration docs and bumps plugin/MCP versions (0.24.0 / MCP 2.5.0), rebuilding mcp/dist outputs.

Reviewed changes

Copilot reviewed 16 out of 47 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test-wiki-forget-ai-block.sh	Regression test ensuring FORGET byte-count ignores complete ai-blocks and doesn’t strip unterminated blocks.
skills/upgrade/SKILL.md	Adds 0.24.0 migration row describing Phase 1 AI-native representation.
scripts/wiki-forget-score.sh	Updates FORGET body byte-count to exclude ai-block content.
mcp/src/tools/knowledge-validate.ts	Adds gentle `ai_block_incomplete` warnings when required ai-block fields are missing.
mcp/src/tools/knowledge-validate.test.ts	Tests that ai-block incompleteness is a warning and absent ai-block is OK.
mcp/src/tools/knowledge-search.ts	Exposes `doc.aiBlock`, strips ai-block for related-fallback and stub-penalty length check.
mcp/src/tools/knowledge-search.test.ts	Adds tests for stub-penalty excluding ai-block and parseDoc ai-block behavior.
mcp/src/tools/knowledge-reindex.ts	Ensures first-sentence extraction ignores ai-block content.
mcp/src/tools/knowledge-reindex.test.ts	Adds tests for projection preserving ai-block and firstSentence ignoring it.
mcp/src/tools/ai-block.ts	New ai-block parsing/stripping/validation module and schemas.
mcp/src/tools/ai-block.test.ts	Unit tests for ai-block parsing, stripping, validation, and unterminated marker behavior.
mcp/src/server.ts	Bumps MCP server version to 2.5.0.
mcp/dist/tools/knowledge-validate.test.js.map	Rebuilt dist artifact for validate tests.
mcp/dist/tools/knowledge-validate.test.js	Rebuilt dist artifact for validate tests.
mcp/dist/tools/knowledge-validate.js.map	Rebuilt dist artifact for validate implementation.
mcp/dist/tools/knowledge-validate.js	Rebuilt dist artifact for validate implementation.
mcp/dist/tools/knowledge-validate.d.ts.map	Rebuilt dist typings map.
mcp/dist/tools/knowledge-validate.d.ts	Rebuilt dist typings.
mcp/dist/tools/knowledge-validate.bundle.js	Rebuilt bundled dist output including ai-block logic.
mcp/dist/tools/knowledge-search.test.js.map	Rebuilt dist artifact for search tests.
mcp/dist/tools/knowledge-search.test.js	Rebuilt dist artifact for search tests.
mcp/dist/tools/knowledge-search.js	Rebuilt dist artifact for search implementation.
mcp/dist/tools/knowledge-search.d.ts.map	Rebuilt dist typings map.
mcp/dist/tools/knowledge-search.d.ts	Rebuilt dist typings.
mcp/dist/tools/knowledge-search-cli.bundle.js	Rebuilt CLI bundle including ai-block stripping/parsing.
mcp/dist/tools/knowledge-reindex.test.js.map	Rebuilt dist artifact for reindex tests.
mcp/dist/tools/knowledge-reindex.test.js	Rebuilt dist artifact for reindex tests.
mcp/dist/tools/knowledge-reindex.js.map	Rebuilt dist artifact for reindex implementation.
mcp/dist/tools/knowledge-reindex.js	Rebuilt dist artifact for reindex implementation.
mcp/dist/tools/knowledge-reindex.d.ts.map	Rebuilt dist typings map.
mcp/dist/tools/knowledge-reindex.bundle.js	Rebuilt bundled dist output including ai-block stripping.
mcp/dist/tools/ai-block.test.js.map	New dist map for ai-block tests.
mcp/dist/tools/ai-block.test.js	New dist output for ai-block tests.
mcp/dist/tools/ai-block.test.d.ts.map	New dist typings map for ai-block tests.
mcp/dist/tools/ai-block.test.d.ts	New dist typings for ai-block tests.
mcp/dist/tools/ai-block.js.map	New dist map for ai-block module.
mcp/dist/tools/ai-block.js	New dist output for ai-block module.
mcp/dist/tools/ai-block.d.ts.map	New dist typings map for ai-block module.
mcp/dist/tools/ai-block.d.ts	New dist typings for ai-block module.
mcp/dist/server.js	Rebuilt dist server output with version bump.
mcp/dist/server.bundle.js	Rebuilt bundled dist server output including ai-block logic.
mcp/dist/cli/sb-entry.bundle.js	Rebuilt CLI entry bundle including ai-block logic.
docs/specs/2026-06-02-ai-native-knowledge-representation-design.md	New design doc describing the ai-block concept, constraints, and phased rollout.
docs/plans/2026-06-02-ai-native-representation-phase1.md	New implementation plan detailing Phase 1 tasks and tests.
.claude-plugin/plugin.json	Bumps plugin version to 0.24.0.
.claude-plugin/marketplace.json	Bumps marketplace version to 0.24.0.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    const kv = line.match(/^([a-z_][a-z0-9_]*):\s*(.*)$/i);
+    if (kv) { last = kv[1]; out[last] = kv[2].trim(); }
+    else if (last) { out[last] = (out[last] + ' ' + line.trim()).trim(); }
+  }


+  age=$(( (now - $(stat -c %Y "$f")) / 86400 ))
+  # body byte-count is PROSE-ONLY: strip the authored ai-block so a uniform block can't lift
+  # every page over the stub floor (spec §5b). Only strip when a COMPLETE block exists (a
+  # closing ai:end is present) — an unterminated ai:begin is NOT a block, so it stays a raw
+  # byte-exact count (matches the TS stripAiBlock no-op; never eats a real page toward FORGET).
+  if grep -qE '<!--[[:space:]]*ai:end[[:space:]]*-->' "$f"; then
+    body=$(awk '
+      /<!--[[:space:]]*ai:begin/ { skip=1 }
+      skip==1 { if ($0 ~ /<!--[[:space:]]*ai:end[[:space:]]*-->/) skip=0; next }
+      { print }
+    ' "$f" | wc -c)
+  else
+    body=$(wc -c < "$f")
+  fi


  // Stub penalty: auto-extracted skeletons and very short pages rank below real content
  for (let i = 0; i < scored.length; i++) {
    if (allDocs[i].source === 'local-doc') continue;
    const { doc, rawContent } = allDocs[i];
-    if (AUTO_EXTRACTED_RE.test(rawContent) || doc.body.trim().length < MIN_SUBSTANTIVE_LENGTH) {
+    if (AUTO_EXTRACTED_RE.test(rawContent) || stripAiBlock(doc.body).trim().length < MIN_SUBSTANTIVE_LENGTH) {
      scored[i].score *= STUB_PENALTY;
    }


+## 5. Consumption (where the win lands)
+
+- **The block is a first-class, proposition-level retrieval unit (RESOLVED — SOTA Q2).** "Dense X Retrieval"/factoid-wiki shows atomic propositions as retrieval units significantly outperform passage/sentence retrieval (sharper precision, fewer tokens, better downstream QA); AGRaME confirms proposition-granularity ranking. So:
+  - **Phase 1 (offline-first):** `knowledge_search` BM25-weights the block (high-signal, deduped) — `parseDoc` extracts it as `aiBlock`, scored above body, returned as the snippet. Works with no embeddings (the Pi default).
+  - **Phase 2 (embeddings present):** the block gets its **own embedding/vector** (proposition-level index), not folded into the page's body vector. Refs: [Dense X Retrieval](https://arxiv.org/html/2312.06648v2), [AGRaME](https://arxiv.org/pdf/2405.15028).
+- **session-load injection** prefers the `ai:begin` block over the prose body — token-cheap, no re-parse. The full prose stays one fetch away.


Cain-Ish and others added 12 commits June 2, 2026 17:15

feat(ai-block): pure parse + per-type schemas + validate + strip

9b0f7f4

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ai-block): parseDoc exposes aiBlock; body-link fallback strips t…

c3923ba

…he block Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(ai-block): stub penalty measures prose only (excludes ai-block)

aa61f65

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(ai-block): firstSentence strips the ai-block before extracting de…

ddcd9a0

…scription Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(ai-block): knowledge_validate warns on incomplete ai-block (gentle)

180c842

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(ai-block): FORGET body byte-count excludes the ai-block (prose-only)

030c243

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(ai-block): guard that reindex/projection preserves the ai-block

720dc3c

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings June 2, 2026 16:43

Copilot started reviewing on behalf of Cain-Ish June 2, 2026 16:43 View session

Cain-Ish merged commit b41488e into main Jun 2, 2026
2 checks passed

Cain-Ish deleted the feat/ai-native-representation-spec branch June 2, 2026 16:46

Copilot AI reviewed Jun 2, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(kb): AI-native representation Phase 1 — the ai:begin block foundation — 0.24.0#13

feat(kb): AI-native representation Phase 1 — the ai:begin block foundation — 0.24.0#13
Cain-Ish merged 12 commits into
mainfrom
feat/ai-native-representation-spec

Cain-Ish commented Jun 2, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Cain-Ish commented Jun 2, 2026

Why

What — Phase 1 (deterministic foundation + safety)

State-check grounded (spec §5b)

Release-gate review — 4 findings, 0 false positives, all fixed + guarded

Verification

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants