Phase 2 ARCHITECTURE_UPDATE loses all results when context compacts before final knowledge-graph.json write

[rilkerfc]

Summary

When auto-update triggers an ARCHITECTURE_UPDATE with many structural files (e.g. 98 files → 13 batches), the current Phase 2 flow accumulates all batch results in the main LLM context before performing a single final knowledge-graph.json write. If context compaction fires between the last batch completing and the write step, 100% of work is lost — knowledge-graph.json and meta.json remain unchanged, and the next session re-triggers the full update.

Reproduction

1. Let KG drift across many sprints (e.g. 9 sprints / 516 commits).
2. Session start triggers ARCHITECTURE_UPDATE with ~98 structural files → 13 parallel batches.
3. Batches complete (~1.2M tokens total in context).
4. Context compaction fires before Write knowledge-graph.json executes.
5. On next session start: hook fires again, same 13-batch cost, same outcome.

Root cause

Phase 2 step 4 reads:

4. After batch(es) complete, read each `batch-<N>.json` and merge results.

Then step 5 (merge) and Phase 3d (save) happen in the main thread. For large batch counts, the accumulated agent outputs fill the context window before the write executes.

The intermediate/batch-N.json files ARE written by agents (good), but the merge + final write is a single all-or-nothing operation in the main thread that is vulnerable to context compaction.

Proposed fix

Process batches in groups of 3-4. After each group: merge → write knowledge-graph.json immediately (checkpoint). Then continue with the next group.

4. Process batches in groups of **3-4 at a time**. After each group:
   a. Verify each `intermediate/batch-N.json` EXISTS on disk (agent must have written it)
      — if missing, the agent failed silently; log a warning and skip that batch
   b. Read the batch file(s) and merge their nodes/edges into the running KG
   c. Write the updated `knowledge-graph.json` immediately (checkpoint)
   Then continue with the next group of batches.
   **Rationale:** context compaction mid-merge loses all in-memory results.
   A per-group checkpoint write ensures partial progress survives.

This means context compaction only loses the tail (unprocessed batches), not everything already merged. On the next session start, the fingerprint check correctly identifies only the remaining unprocessed files as changed.

Additional note: P2 anchor drift

In v2.7.5, the LOAD-PATCH-SAVE section of auto-update-prompt.md was restructured. The apply_ua_patches.py script we use locally to patch the prompt has a stale find anchor for P2 (// fingerprint-update.mjs\n). The marker check still passes (patch is present), but the anchor string no longer matches. Worth noting for users who maintain local patch scripts.

Environment

• Plugin version: 2.7.5
• Project: large Python/FastAPI monorepo (~1700 KG nodes, 50+ services)
• Structural change count that triggered this: 98 files (9-sprint drift)
• Estimated tokens lost: ~1.2M

[lvca]

I'm Luca Garulli, creator of ArcadeDB (https://arcadedb.com, Apache 2.0). Not suggesting you need a database to fix this specific bug, your intermediate/batch-N.json approach is the right instinct, but the root cause here is more general and worth naming: the whole pipeline holds state in the agent context and commits it as one atomic knowledge-graph.json write at the end, so anything that interrupts before that write (compaction, crash, cancel) loses 100% of the work. That same single-file model is also what makes #219 (cross-repo graph) and #291 (persistence across sessions) hard.

The durable-incremental-write property you are reaching for with batch-N.json is exactly what a transactional store gives you for free. The shape of the fix:

• Each batch commits its nodes and edges in its own transaction as it completes. Compaction between batch 12 and batch 13 cannot lose batches 1 to 12, because they are already durably committed. The next session resumes from the last committed batch instead of re-running all 13.
• The agent never has to hold the full merged graph in context to do the write. It streams results out as it goes, which also relieves the ~1.2M-token accumulation you measured.

If or when you decide the JSON file has run its course, ArcadeDB is one option for the backend. A few things that line up with this project specifically:

• No JVM code in your stack. ArcadeDB runs as a single local process and speaks HTTP/REST, so the plugin writes batches with plain fetch calls from TypeScript. You can start it on demand and point it at a local data dir, similar to how you already manage the graph file.
• Native multi-model graph. Files, functions, and classes are nodes; imports, calls, and ownership are edges, queryable with Cypher or Gremlin. For [Feature Request] Multi-repo workspace support and cross-repo knowledge graph #219, cross-repo edges (this service calls that API) are just edges between subgraphs in one database, which the single-file-per-repo model cannot express.
• Built-in vector search. Since the point is to "ask questions about" the code, you can store embeddings alongside the graph nodes and do semantic lookup in the same store, no second system.
• Apache 2.0 with governance guarantees (no CLA, no relicensing path), which matters if this becomes infrastructure many teams depend on.

Happy to help sketch a GraphStore-style interface so the JSON writer and a database backend are swappable, or to prototype the per-batch transactional write against ArcadeDB. Either way, committing each batch as it finishes is the fix that makes this class of bug go away. Feel free to ping me here.