Record which bytes every m.db was built from (W1) - #9
Merged
Conversation
…oracle that moved The phantom-entry incident could not be attributed after the fact: convert parsed export.pdb bytes that the settled file no longer contained (torn read, 14 s after rekordbox's last write on exFAT), verify later compared against the settled bytes, and nothing recorded which side had moved. Separately, Engine DJ provably rewrites this stick's m.db (desktop-library merge), and one convert "succeeded" with its drive unmounted, dropping the report — the only record — into the source repo. This is W1/W1c of the phantom-entry plan. Fingerprint at read time ------------------------ parse_export_pdb hashes the buffer it actually parsed — never a re-read, which under a torn read would fingerprint the wrong bytes — and carries SourceFingerprint (sha256/size/mtime) on SourceLibrary. The field is excluded from to_json_obj, which is load-bearing for golden byte-identity. Provenance at publish time -------------------------- build_library pairs that fingerprint with the staged m.db sha256 and the dense id watermarks (max Playlist.id / PlaylistEntity.id) on the report. The record is deliberately time-free: writer/ sits under the no-wallclock determinism gate, so the timestamp is added only where the journal line is written, in report.py, called from cli.py after publish. Nothing new goes inside m.db — a timestamp would break byte-identical rebuilds and the m.db hash cannot contain itself. Append-only journal beside the report ------------------------------------- Engine Library/rb2engine-journal.jsonl gains one line per publish, capped at 64 KB by dropping oldest lines. The fixed-name report alone is not enough: "re-run convert" is exactly the remedy verify prescribes on staleness, and it overwrites the report — destroying the fingerprint of the suspect bytes, which is how the original incident's evidence was lost. A successful run whose report falls back to cwd now warns loudly that no provenance travels with the stick. verify: attribution before accusation, exit codes partitioned ------------------------------------------------------------- verify compares the recorded fingerprint against the source it just parsed and reports source changed / db changed / both / neither as top-level provenance findings, never as playlist discrepancies. Exit contract: 0 ok / 1 discrepancies / 2 cannot-verify / 3 not attributable Source-independent findings (broken nextEntityId chains, undecodable blobs) always exit 1 — a stale source must not launder a real defect into "re-run convert". Source-dependent comparisons under a fingerprint mismatch become informational with exit 3, whose finding text also states the irreducible residue: corruption co-occurring with a changed source is undetectable until convert re-runs. "db changed" is classified, not alarmed — Engine rewriting m.db is legitimate. A missing record (0.4.0 stick, off-stick report) is its own visible finding and degrades gracefully; exit 2 keeps precedence over both. Tests ----- 27 new tests: the fingerprint is proven to hash the parsed buffer against a file that settles underneath it; golden serialization unchanged; journal append/cap/truncation; the co-occurrence cases that decide whether exit 3 can swallow a defect (chain fork and garbage blob under a stale fingerprint both force 1); missing/unreadable provenance reported without a crash; the CLI journal write and the off-stick warning. 731 passing (704 before), coverage 88%, ruff and mypy clean; test_no_wallclock still holds — the clock lives outside writer/ and mapper/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3aF4y4GdsuJta2bfYUZ8w
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements W1 from
.omc/plans/playlist-phantom-entry-root-cause.md, including the exit-code partition (W1c) and the fail-loud report fallback.Why
The July 29 incident was undiagnosable for days because nothing tied an
m.dbto the bytes it was built from. Whenconvertandverifydisagreed, there was no way to tell whether the database was wrong or the source had moved — the two most likely explanations demanded opposite responses, and the evidence to choose between them was destroyed by the recommended remedy (re-run convert).This records the source fingerprint at read time and lets
verifyname which oracle moved.What lands
parse_export_pdb, never by re-reading — the whole point is capturing what was parsed, and a torn read must be fingerprinted as torn.SourceFingerprintonSourceLibrary, optional and defaulting toNone, and excluded fromto_json_obj— that serialization is load-bearing for golden byte-identity.Engine Library/rb2engine-journal.jsonl, one line per publish (pdb sha/size/mtime, staged m.db sha,max(Playlist.id),max(PlaylistEntity.id)), capped at 64 KB by dropping oldest lines, tolerant of a truncated final line. A journal survives repeated re-converts, which single-generation rotation does not.m.db, and no clock inwriter/ormapper/.build_libraryproduces a time-free record; the onlydatetime.nowis inreport.py, invoked from the CLI. A test pins the record's field set so a timestamp can't drift into the writer's side.Exit-code contract
FatalError)The partition matters: a stale source must never launder a real fault into "just re-run convert". The finding text also states the irreducible residue — source-dependent corruption co-occurring with a stale source is undetectable until convert re-runs.
db_changed(recorded staged sha ≠ current m.db) is informational and never alters the exit code, because Engine DJ legitimately rewritesm.dbon open — measured this session.Judgment call worth review
Missing provenance (a 0.4.0 stick, or the off-stick report case) is a rendered, visible finding but does not change the exit code. Routing it to 3 would fail every existing stick until re-convert. Easy to tighten later if you want mismatch-equivalent treatment.
Verification
731 tests (+27), coverage 88.45% (floor 85), ruff + mypy clean,
test_no_wallclockgreen, goldens and determinism unchanged.🤖 Generated with Claude Code
https://claude.ai/code/session_01V3aF4y4GdsuJta2bfYUZ8w