fix(search): stop pipeline work on cancellation, fatal errors, and panics - #22
Merged
Merged
Conversation
14 tasks
…nics ExecuteContext ran everything it was handed no matter what went wrong. It never derived a cancellable context, so a fatal error only landed in the 1-slot errCh while every remaining blob was still searched, and --quiet spin-drained the whole queue through a `stop atomic.Bool` after the answer was already known. The closer then republished ctx.Err() unconditionally, so a SIGINT arriving just after the last worker exited turned a complete result set into exit code 2. Work now runs under a cancellable child of the caller's context. The first fatal error and the first quiet-mode hit call cancelWork(), which unblocks the dispatcher and lets workers return at their existing ctx.Done() arm instead of draining. The closer defers cancelWork() so the derived context is always released, and publishes a cancellation only when the caller's own context was cancelled *and* fewer results were delivered than there were deduplicated blobs -- so an internal early stop is never misreported as context.Canceled, and a cancel that lands after the caller already received everything cannot fail a complete run. processTask inflates and reconstructs bytes from an arbitrary .git directory, and a panic on a worker goroutine cannot be recovered by runContext: it killed the process, skipping the reader's cleanup and the exitCodeForError contract. safeProcessTask now recovers per task and turns a panic into an ordinary fatal *BlobResult.Error routed through the normal fatal-error path. Also: deleted ExecuteStream, which hardcoded context.Background() and had no callers, so every streaming consumer must supply a cancellable context; extracted dispatch() and runWorker() with directional channel parameters so the compiler enforces the sender/receiver roles whose violation panics; and documented the errCh capacity-1 invariant, the drain-or-cancel precondition behind the no-goroutine-leak claim, and the immutability of Pipeline and Matcher. Closes #14
hammadmajid
force-pushed
the
fix/14-pipeline-cancellation
branch
from
September 5, 2026 14:14
d36923e to
3bb1827
Compare
hammadmajid
marked this pull request as ready for review
September 5, 2026 14:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The pipeline's goroutine lifecycle and channel closure were already correct. These defects are all about what it did after something went wrong: it kept working.
Closes #14
Type of Change
Key Changes
Derived cancellable context.
ExecuteContextnow doescallerCtx := ctx; ctx, cancelWork := context.WithCancel(callerCtx). One change subsumes three defects:errChwith adefaultarm and every remaining blob was still processed; the caller could not observe it untilresultsChclosed. Now the worker publishes, delivers the result, and callscancelWork().--quietstops instead of spin-draining.stop atomic.Booland itsstop.Load()skip are gone. A quiet hit callscancelWork(), the dispatcher drops out at its existingctx.Done()arm, and workers return.callerCtx.Err()— never the derived context — and only whensearched.Load() < len(jobOrder). The parent gate alone was insufficient: a cancel landing after the caller already received every result still reportedcontext.Canceled, which is precisely the defect.searchedis one atomic add per blob, read only afterwg.Wait(), sosearched == len(jobOrder)deterministically means the full set was handed over.Worker panics contained.
processTaskdrives zlib inflate and delta reconstruction over bytes from an arbitrary.gitdirectory, and there was norecoveranywhere in the repository — a panic on a worker goroutine killed the process with a raw stack dump, skippingdefer reader.Close()and theexitCodeForErrorcontract. NewsafeProcessTaskrecovers per task into an ordinary fatalBlobResult.Error, routed through the existing fatal branch. The recover is scoped to one task, not the worker loop.ExecuteStreamdeleted. It hardcodedcontext.Background(), so an abandoned pipeline leaked for the process lifetime with no recovery. No callers.Channel directions are now compiler-enforced. Extracted
dispatch(ctx, jobOrder, tasksCh chan<- *blobTask)(sole sender and closer) andrunWorker(ctx, cancelWork, searched, <-chan *blobTask, chan<- *BlobResult, chan<- error). Previously these were bidirectional locals captured by closures — the invariants whose violation panics were verified only by inspection.Stale-result drop. After
safeProcessTask, an internal cancellation (quiet/fatal) no longer gets re-published as a boguscontext.Canceledfatal error viaprocessTask's own ctx checks.Docs. The
errChcap-1 + non-blocking-send invariant is now written down: it implements first-fatal-error-wins and is what guarantees a worker never blocks publishing an error, since the consumer may not readerrChuntilresultsChis drained — raising the capacity reintroduces the deadlock.ExecuteContext's leak-freedom claim now states its precondition (drain or cancel).PipelineandMatcherdocumented as immutable after construction and safe for concurrent use.Untouched: single-owner closure,
wg.Addbeforego,ctx.Done()on every blocking arm, both fast paths,Execute's deduplicated ordering.Verification & Testing
go test -v -count=1 ./...go test -race -shuffle=on -count=1 ./...go vet ./...Mutation results — each new assertion proven to fail on pre-fix behaviour, then the file restored byte-identically and re-run green:
context canceledsearchedgateTestPipelineCancelAfterCompletionfails withcontext canceledprocessTaskinstead ofsafeProcessTaskNew tests assert observable work, not just returned errors: a counting
ObjectReadershows--quietwith one worker reads exactly 1 of 200 blobs, and the first fatal of 300 stops after exactly 1 read.Note
Stacked on #21.
Checklist
gofmtclean