feat(recovery): detect incompatible indexes, refuse stale reads, and recover the records - #60
feat(recovery): detect incompatible indexes, refuse stale reads, and recover the records#60pablontiv wants to merge 23 commits into
Conversation
PrefixSpan's defining property is that recursion works on a database that shrinks at each level. mineExtensions passed the original sequences down instead, so every node of the search tree re-scanned and re-projected all 305 sessions and 36,956 events with a longer prefix each time. That is why 'patterns --kind sequences --all-projects' hung. Project incrementally: the top level projects once per frequent item, and each recursive step projects the current database by the single new item. This is equivalent because project() completes a pattern greedily left-to-right, so projecting by [a,b] equals projecting by [a] then by [b]. Pinned by an equivalence test that runs 200 random corpora through both this implementation and the original full-database recursion and requires identical output. Measured on the real corpus at --min-support 20: max-length 4 now returns in 2s and 5 in 14s. The default of 6 still exceeds a reasonable wait — the frequent-pattern set is genuinely exponential in length here — so this fix makes the command usable at explicit lengths without yet making its default usable. Claude-Session: https://claude.ai/code/session_01XsmeJSWsTTZPSzfk5kgXo7
pablontiv
changed the title
|
Review found the branch ships the design and plan for recovered source accounting but not the implementation — the four planned functions ( |
|
Superseded. Every capability this branch carries has since landed on The known limitation this PR documented is also already fixed on main: Rebasing would mean resolving 260 conflicts to land nothing new. |
1 participant
What this does
Makes an incompatible or stale index a reachable condition instead of a silent data hazard: it is detected, reads are refused, and the records are recoverable.
Compatibility
internal/compat).Refusal
cmd/backscroll/index_policy.go). This is the one deliberate behaviour change: commands that previously returned data from an incompatible index now stop, and the diagnostic points atrecover --from <path> --dry-run.Recovery
recovercommand with a genuinely read-only dry run (cmd/backscroll/recover.go).internal/recovery), with per-platform no-clobber guards:RENAME_EXCLon darwin,RENAME_NOREPLACEon linux, and a safe fallback.Performance
internal/sequences/prefixspan.go: PrefixSpan now recurses on the projected database rather than the full one. Previously every node of the search tree re-scanned and re-projected all sessions and events, which is whypatterns --kind sequences --all-projectshung. Pinned by a 114-line equivalence test.Known limitation — read before merging
A recovered database does not yet pass
validate --indexed-only.CreateRecoveryDestinationinstalls the canonical union of records but leavesindexed_filesempty, andverifyRecoveryDestinationRowscurrently requires that emptiness (recovery_destination.go:243).Database.Validaterequires everysearch_items.source_pathto have anindexed_filesrow, so the recovered database trips the public orphan check.The fix is designed and planned in this very branch —
docs/superpowers/specs/2026-08-19-recovered-source-accounting-design.mdand the matching plan specify a reservedbackscroll:recoveredmarker with a NULLlast_indexed, one row per distinct planned path, plus consumer branches inGetFileHashes,GetStatsandBackfillDerived— but it is not implemented here:recoveredSourceHash,isRecoveredSourceHash,recoveryDestinationSourcePathsandinsertRecoveryDestinationAccountingdo not exist yet.So this PR delivers detection, refusal, atomic recovery and the perf fix; the accounting that makes a recovered database validate is a follow-up.
Verification
go build ./...clean;go test ./...— 22 packages pass, 0 failures, same asmain.Reviewer notes
Worth a look at: whether the refusal in
index_policy.goblocks every consumer it should without blocking ones it shouldn't; whether the atomic replace stays atomic on partial write / interrupted rename / disk full; and whether the three no-clobber implementations are behaviourally equivalent.