(remote): fetch a grown transcript by byte range, stop one large file from stalling the cycle - #274
Merged
Merged
Conversation
… from stalling the cycle Closes #257. A mirrored .jsonl that only grew is completed with tail -c from the local byte count over the existing ssh path, appended copy-then-rename; a shrink, a same-size rewrite, a torn local prefix or a sidecar fall back to the full scp. The cycle sorts transfers by size and checks the budget per file instead of latching cycleFull on the first oversized one.
This was referenced Sep 12, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #257 (audit finding 8). Measured tonight on v0.0.76: 65 remote poll cycles in 19 min, one transcript re-fetched whole in 52 of them — the live session's file, every time it grew.
What
remote-transport.js:fetchIncremental(alias, requests, destRoot)— for{ rel, offset }, runstail -c +<offset+1> '.claude/projects/<rel>'over the existing non-interactive ssh path (relpassesisSafeMirrorRelPath,offsetmust be a non-negative integer), captures stdout as a raw Buffer, writes local copy + tail to.part, renames. A failed tail leaves the previous mirror intact.remote-mirror.js: per file, incremental when the remote size grew and the local prefix is trusted (manifest size matches the on-disk size); full fetch on shrink, same-size rewrite (mtime changed), missing manifest entry, torn prefix, or.meta.jsonsidecar. Cycle: transcripts first, then transfer size ascending, budget checked per file —cycleFullno longer latches on the first oversized file; skip/defer logged once per file per streak.session-cache.md"Remote hosts — incremental fetch" (decision rule, invalidation, mechanics, ordering, known gaps: parse cost unchanged — the indexer still re-reads the file (perf(remote): rescan granularity is the project folder, not the file #216); a live session with no.jsonlyet is invisible to the inventory).Proof
test/remote-transport-incremental.test.js(7): range command shape, exact byte count for a 4 KB append, binary fidelity, failure leaves the mirror intact, no local file → skipped, unsafe rel rejected.test/remote-mirror.test.js(+5): growth → incremental, shrink → full, stale prefix → full, sidecars full, oversized straggler no longer defers smaller files.offset = 0in the incremental branch → "only the new bytes are requested" red (4168 vs 4096 bytes); reverted.task check: 1244 + 119 pass, 0 fail, pre-existing skips.Not verified on the wire:
tail -cagainst the real host (busybox/GNU both accept+N; tests only). Growth with an unchangedmtimeMsis accepted as append evidence — stated in the doc as a heuristic.