Skip to content

WI-143: per-article damage ledger in nzb-core - #140

Merged
thedancingdeveloper merged 2 commits into
mainfrom
feat/wi-143-damage-ledger
Sep 10, 2026
Merged

WI-143: per-article damage ledger in nzb-core#140
thedancingdeveloper merged 2 commits into
mainfrom
feat/wi-143-damage-ledger

Conversation

@thedancingdeveloper

Copy link
Copy Markdown
Collaborator

First code payload of the parity-in-flight initiative (Vogt WI-143, initiative parity-in-flight-damage-ledger-in-memory-par2-streaming-repair). Parent audit: WI-142.

What

A persisted per-article damage ledger in nzb-core — the shared truth the download engine (WI-145/146) and the streaming provider (WI-156+) both consult and update, so a hole learned once is not rediscovered.

  • Migration v10: damage_ledger table keyed (scope_id, file_index, segment_number), created empty. No backfill from v9 history.retry_data: that blob (WI-133) is a retry checkpoint, not damage evidence — it has no failure reason, per-server evidence, TTL, or server fingerprint to reconstruct a confirmed miss from. (Convergence decision recorded on WI-133/WI-143.)
  • crates/nzb-core/src/damage_ledger.rs: LedgerState (unknown / provisional_missing / confirmed_missing / present / unservable_file), per-server evidence (430 vs transport + timestamp), first_refused_at / confirmed_at / expires_at, server_fingerprint.
  • All verdict rules in one function, apply_outcome (from zurg holes.go):
    • a transport failure never creates or refreshes a missing record;
    • a single 430 is provisional, confirming only once every enabled server has refused and ≥1s has passed since the first refusal;
    • a confirmed hole is believed for 24h, then a caller re-probes;
    • a change to the server set (fingerprint) invalidates the record.
  • API on Database: ledger_note, ledger_consult, ledger_held_provisionally, ledger_file_gone_ratio, ledger_load_file. Callers translate their own failure taxonomy (nzb-dispatch::ArticleFailureKind) into Outcome (Present / Refused / Transport) — nzb-core does not depend on the dispatcher.

Tests

6 deterministic unit tests (injected clocks, in-memory DB) covering transient 430-then-200, 430 on every server, transport-on-the-last-server (never confirms), TTL-expiry re-probe, fingerprint invalidation, and the file-gone ratio. Full nzb-core suite: 128 passed. cargo fmt --check and cargo clippy -D warnings clean.

Scope notes

  • The ledger is a pure nzb-core SQLite/state-machine unit with no NNTP dependency, so it is tested directly with controlled outcomes rather than via mock-nntp-server; end-to-end mock-NNTP coverage lands with WI-151 fixtures and the WI-144/145 wiring.
  • Public-surface change is additive only, so the published nzbdav 0.5.7 consumer keeps building.
  • The 16 KiB MD5 file-matcher move from nzbdav-pipeline is deferred to WI-144 (its consumer).
  • Independent of the WI-168 version-floor PR (WI-168: nzb-* version floor + crate sync tooling and drift gate #139); no version bump here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TpoFnScdkRq7qxQLQZe1we

Add a persisted per-article damage ledger to nzb-core, the shared truth
the download engine and the streaming provider both consult and update
so a hole learned once is not rediscovered.

Migration v10 adds a `damage_ledger` table keyed
(scope_id, file_index, segment_number), created empty. There is no
backfill from v9 `history.retry_data`: that blob is a retry checkpoint,
not damage evidence — it carries no failure reason, per-server
evidence, TTL, or server fingerprint to reconstruct a confirmed miss
(see WI-142/WI-143 convergence).

The rules that decide a verdict live in exactly one function,
`apply_outcome`, following zurg's holes.go:
- a transport failure never creates or refreshes a missing record;
- a single 430 is provisional and confirms only once every enabled
  server has refused and >= 1s has passed since the first refusal;
- a confirmed hole is believed for 24h, then a caller re-probes;
- the record is keyed to the set of servers asked (server_fingerprint);
  adding or removing a provider invalidates it.

API on Database: ledger_note, ledger_consult, ledger_held_provisionally,
ledger_file_gone_ratio, ledger_load_file. Callers translate their own
failure taxonomy into Outcome (Present / Refused / Transport).

Deterministic unit tests cover transient 430-then-200, 430 on every
server, transport-on-the-last-server (never confirms), TTL expiry
re-probe, fingerprint invalidation, and the file-gone ratio. Change is
additive to the public surface, so the published nzbdav consumer keeps
building.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TpoFnScdkRq7qxQLQZe1we
@thedancingdeveloper
thedancingdeveloper merged commit 7e14bd3 into main Sep 10, 2026
11 checks passed
@thedancingdeveloper
thedancingdeveloper deleted the feat/wi-143-damage-ledger branch September 10, 2026 04:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant