Single-replica: doc permanently stuck in yjs_fallback/repairPending after failed edit verification clears Yjs state (session's own lease blocks reseed + repair)

[nishfaria]

Summary

On a single-replica self-host (PROOF_SINGLE_REPLICA=1, per #58), a document can get permanently stuck in a degraded state where:

• GET /documents/:slug/state returns revision: null, mutationReady: false, repairPending: true, projectionFresh: false, readSource: "yjs_fallback"
• POST /edit/v2 always returns 409 FRAGMENT_DIVERGENCE ("Live block at bN no longer matches the base snapshot")
• POST /ops rewrite.apply rejects with "Invalid baseRevision" (because state's revision is null)
• The advertised repair never completes — repairPending stays true indefinitely (observed > 2 min of polling; in practice it persists as long as a browser tab has the doc open, and 5 minutes beyond)

Observed twice on 2026-07-07 on the same document lineage; a sibling doc on the same server kept working. Only recovery found from the API surface alone was delete + recreate, which changes the share URL and breaks shared links.

This is the same class of single-replica false positive as #58 and #50 — a recentCollabSessionLeases entry being read as "another replica may hold this doc live" — but in a different and more destructive lane: it blocks the legacy Yjs reseed and the on-demand projection repair, after a failed mutation verification has already cleared the persisted Yjs state.

Environment

• proof-sdk self-hosted, single Docker container (PROOF_ENV=production, hosted runtime semantics)
• PROOF_SINGLE_REPLICA=1 (with the Add PROOF_SINGLE_REPLICA mode to avoid false LIVE_DOC_UNAVAILABLE on single-replica self-hosts #58 patch applied), COLLAB_ON_DEMAND_PROJECTION_REPAIR_ENABLED=true
• SQLite storage

Root-cause chain (from code + DB + logs)

Step 1 — a failed mutation verification destroys the canonical Yjs state.
executeCanonicalMutation's FRAGMENT_DIVERGENCE branch (server/canonical-document.ts, ~line 1252) and agent-edit-v2's unconfirmed-collab branch (server/agent-edit-v2.ts, ~line 673) both call invalidateCollabDocument(slug) with clearPersistedState=true, which runs clearYjsState(slug) — deleting all rows in document_y_updates and document_y_snapshots. If the document never had a snapshot compacted (typical for a doc that has only seen one live browser session), the canonical Yjs history is now empty. The forensic log confirms pending deltas being discarded:

[collab] Pending Yjs delta before clear (forensic only) {
  slug: '3sa0mrqa', reason: 'invalidate:pre', seq: 103, bytes: 34256,
  sourceActor: 'collab', createdAt: '2026-07-07T15:07:52.565Z', autoQuarantined: false
}

invalidateCollabDocumentInner also calls clearProjectionRepairState(slug), killing any queued background repair.

Step 2 — the DB is left with an unsatisfiable freshness invariant.
After the clear, both stuck docs looked like this (live DB):

documents:             y_state_version = 0, revision = 1
document_projections:  y_state_version = 103 (other doc: 106), health = 'healthy'
document_y_updates:    0 rows
document_y_snapshots:  0 rows

isProjectionFresh requires projection_y_state_version === documents.y_state_version (103 ≠ 0), so every read degrades to yjs_fallback with repairPending: true. Nothing ever rewrites the projection row, so the mismatch is permanent until some repair path runs.

Step 3 — every self-heal path is gated on the same ghost-lease heuristic, and the session's own lease trips it.

• shouldBlockLegacyReseed (server/collab.ts ~4477) blocks reseeding the canonical Yjs baseline from the legacy row whenever getRecentCollabSessionLeaseCount(slug, epoch) > 0 — a 300 s TTL entry written by buildCollabSession.
• shouldDeferOnDemandProjectionRepair (server/canonical-document.ts ~213) skips on-demand repair whenever activeCollabClients > 0 || recentLeases > 0 (live_collab_present).
• Neither consults isSingleReplicaDeployment().

The killer detail: buildCollabSession notes the lease before the WS connection hydrates the doc, so the very session trying to open the document blocks its own reseed. Log timestamps from the incident — same millisecond:

[collab] buildCollabSession lease noted { slug: '3sa0mrqa', role: 'editor', accessEpoch: 0, tokenId: null, ttlSeconds: 300 }   // 14:54:50.180
[collab] blocked legacy Yjs reseed during active live collab lease {
  slug: '3sa0mrqa', source: 'read_persisted_doc_state_async',
  blockedReason: 'recent_live_collab_lease', markdownChars: 12697,
  updatedAt: '2026-07-07T14:54:50.181Z', projectionHealth: 'healthy'
}

While the tab stays open the lease keeps renewing, so the deadlock never resolves; after closing the tab it still takes the full 300 s TTL. Every further /edit/v2 attempt hits the degraded fallback doc, fails verification, and re-invalidates — reinforcing the broken state.

On a single replica the lease can only ever refer to this node. When there is no live hocuspocus doc locally and the persisted Yjs state is empty, "another replica might hold the live doc" is impossible, and the reseed/repair should proceed — exactly the reasoning behind PROOF_SINGLE_REPLICA in #58, which currently only covers the rewrite live-client gate.

Why the sibling doc survived

A larger sibling doc on the same server hit authoritative_read_mismatch too, but its rewrites took the fallbackBarrier lane (prepareRewriteCollabBarrier → non-destructive invalidateLoadedCollabDocument, clearPersistedState=false) with no browser tab open, so the reseed ran and it recovered (revision kept advancing). The destructive combination is specifically: live browser session + failed edit/v2 verification.

Suggested fix

1. In shouldBlockLegacyReseed and shouldDeferOnDemandProjectionRepair: when isSingleReplicaDeployment(), ignore getRecentCollabSessionLeaseCount and gate only on actual local state (live hocuspocus doc / getActiveCollabClientBreakdown exact-epoch connections), mirroring the Add PROOF_SINGLE_REPLICA mode to avoid false LIVE_DOC_UNAVAILABLE on single-replica self-hosts #58 fix for the rewrite gate. A reseed requested by a doc-load path when no live doc exists locally cannot be racing another replica.
2. Independently: when invalidateCollabDocument clears persisted Yjs state, reset document_projections.y_state_version (or stamp it from getLatestYStateVersion, which is 0 after the clear) so the projection row doesn't strand a version the canonical store can never reach again.
3. Consider compacting pending deltas into a snapshot before clearYjsState in the invalidate path, so a failed verification doesn't silently discard user collab edits (34 KB of deltas in our incident).

Workaround for other self-hosters

POST /api/agent/:slug/repair (owner-secret auth) calls repairCanonicalProjection directly, skips the lease heuristics, and heals the doc in place — share URL preserved. It converges once the stale lease expires (≤ 5 min after all tabs close); retry with backoff. We now keep a small ops script that resolves the owner secret, hits this endpoint with retries, and verifies projectionFresh flips back to true.

Happy to test a patch — we run this in production daily and can reproduce the trigger conditions (browser session + agent edit/v2 pushes on the same doc).