fix(aggregator): drain resultsCh on early return and stop polling a closed errCh - #23
Merged
Conversation
14 tasks
…losed errCh AggregateChannel had three early returns (context cancellation, an error on errCh, and a fatal res.Error) that all walked away from a still-running producer with resultsCh open and undrained. Producers stream over a bounded channel, so once the buffer fills every worker blocks on its send, the task dispatcher blocks behind them, wg.Wait never returns and neither channel is ever closed: the worker pool, its dispatcher and the object store they pin leak for the life of the process, and a later range over the same channel hangs too. A named-result defer now drains resultsCh whenever the function exits with an error or a cancelled context, so the producer always gets the receives it needs to run itself down. The drain is skipped when the loop already observed the close, since ranging over the nilled channel would block forever. A receive on a closed channel is ready forever, so the exhausted errCh arm re-armed itself on every pass and spun the select at full CPU until the producer caught up (measured: ~214k passes across 100ms of streaming). Exhausted arms are now disabled by nilling their channel, which is what the pre-existing "if errCh != nil" guard had always intended. Finalization no longer depends on an undocumented ordering contract. The old post-close check was a non-blocking peek, so a producer that closed resultsCh before publishing its terminal error had that error silently swallowed and the search reported as successful. Aggregation now disables the results arm on close and keeps looping until errCh is closed as well, so a late error is surfaced. The producer and caller contracts this relies on - close both channels, share a cancellable context - are documented on AggregateChannel. Tests: a real search.Pipeline is driven into both early-return paths and the goroutine count is polled back to baseline; a custom context counts select passes to prove the closed errCh is no longer polled; a producer that closes resultsCh before publishing its error must still surface it. The existing channel tests now close both channels, as the documented producer contract requires. Closes #15
hammadmajid
force-pushed
the
fix/15-aggregator-drain
branch
from
September 5, 2026 14:14
831accb to
a987873
Compare
hammadmajid
marked this pull request as ready for review
September 5, 2026 14:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
AggregateChannelreturned on three paths withresultsChstill open and undrained, permanently wedging its producer.Closes #15
Type of Change
Key Changes
Drain on every early return. The pipeline's workers block on
resultsCh <- reswhose only escape isctx.Done(). Once themax(32, NumCPU*4)buffer filled: every worker wedged, the dispatcher wedged ontasksCh <- task,wg.Wait()never returned, and neither channel was ever closed — so a laterrangeby the same caller hung too. That isNumCPU+2leaked goroutines plus a pinnedRepositoryReader.AggregateChannelneither cancelled nor drained, and does not own aCancelFunc. Now a named-resultdeferdrainsresultsChwhenever the function exits with an error or a cancelled context. The drain is skipped whenresultsChhas already been nilled, because ranging a nil channel blocks forever; the termination argument is written out in a comment.Disable the exhausted
errCharm. A receive on a closed channel is permanently ready, sook == falsefell through and re-entered theselectforever. Measured pre-fix: 214,261 select passes over ~100 ms of streaming. NowerrCh = nil; continue. The deadif errCh != nilguard that was already in the file was residue of this intended fix; it finally means something.No more unwritten ordering contract. The post-close error check was a non-blocking
default:peek, so it only worked if the producer published its terminal error strictly before closingresultsCh. Nothing documented or enforced that, and this is exported API taking arbitrary channels. NowresultsChclose setsresultsCh = nil, the loop runs whileresultsCh != nil || errCh != nil, and finalization happens only once both are closed — an error published afterresultsChcloses is surfaced instead of swallowed.Contracts documented on
AggregateChannelandAggregateStream: the producer owns and must close both channels and may publish its terminal error in either order; the caller must pass a context the producer also observes, since the drain only terminates when the producer closes.Verification & Testing
go test -v -count=1 ./...go test -race -shuffle=on -count=1 ./...go vet ./...Mutation: restored the pre-fix
aggregator.goand re-ran — all four new tests fail there (6 leaked goroutines and an undrained channel in both leak subtests, 214,261 select passes in the spin test, error swallowed in the late-error test), then restored the fixed file byte-identically.TestAggregator_AggregateChannel_ReleasesProducerOnEarlyReturndrives a realsearch.Pipeline(256 in-memory blobs, deliberately above the result buffer) through both early-return paths and pollsruntime.NumGoroutineback to baseline with a 5 s deadline. The existing aggregator tests only ever fed synthetic channels, so the pipeline integration had never been leak-checked. ArunWithinhelper bounds every call so a blocking regression fails instead of hanging the suite.Note
Stacked on #22.
API change for reviewers:
AggregateChannelnow finalizes only when both channels are closed, so producers must closeerrChtoo.search.Pipelinealready does on every exit path, and there are no non-test callers (cmd/grguses the slice-formAggregate).Checklist
gofmtclean