Validate download integrity: evict corrupt cache hits, verify downloaded sizes - #29
Merged
Merged
Conversation
Truncated or corrupt downloads could poison the local cache: a partial file (e.g. a truncated ATT.OBX.gz) was cached, then treated as a satisfied dependency on every later run and silently handed to pdp3 until a user manually found and deleted it. Two changes in WormHole.download_one: - Cache hits (including the decompressed-.gz path) are now validated against their sidecar lockfile SHA-256 in strict mode. On mismatch the file and its stale sidecar are evicted and the download re-runs in the same call. Files without a sidecar keep the old trust-if- non-empty behavior. - Downloaded files are verified against the remote size already fetched via fs.info() before the transfer. On mismatch the partial file is deleted and the download retried once; a second mismatch deletes it and fails the resolution rather than caching it. New local-only test suite exercises the real file-protocol download path: truncated transfers (fail and retry-succeed), valid/corrupt/ sidecar-less cache hits, and the gzip decompressed-cache path. Fixes symptom (c) of #26 (phase 2 of 3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 2 of 3 for #26 — fixes symptom (c): truncated/corrupt downloads poisoning the local cache.
Previously a partial download (e.g. a truncated
ATT.OBX.gz) was cached, "resolved" as a satisfied local dependency on every later run, and silently handed topdp3until someone manually found and deleted it. The cache-hit check only required the file to exist withst_size > 0, and the post-download check only rejected zero-byte files.Changes (both in
WormHole.download_one)Cache-hit validation. Existing files (including the decompressed-
.gzfast path) are now validated against the sidecar_lock.jsonSHA-256 thatDownloadPipelinewrites, in strict mode. On mismatch the file and its stale sidecar are evicted and the download re-runs in the same call. Files without a sidecar keep the previous trust-if-non-empty behavior.Post-download size verification. The remote size was already fetched via
fs.info()before every download but never compared afterward. Now a size mismatch deletes the partial file and retries once; a second mismatch deletes it and fails the resolution instead of caching it.Tests
New
test_download_integrity.py(10 tests, local-only — the "remote" is afile-protocol directory, so the realConnectionPoolFactorydownload path is exercised):Full local run: 422 + 43 + 35 tests pass across the three packages; ruff check/format clean.
Remaining for #26
🤖 Generated with Claude Code