[Claude] [AUTOTEST-466] Log benign setMaxInFlight teardown race at DEBUG#113
Open
sprout-autoagent wants to merge 1 commit into
Open
[Claude] [AUTOTEST-466] Log benign setMaxInFlight teardown race at DEBUG#113sprout-autoagent wants to merge 1 commit into
sprout-autoagent wants to merge 1 commit into
Conversation
When the read thread closes the socket concurrently (peer EOF / nsqd shutdown), SubConnection.setMaxInFlight flush() fails with "Socket closed". This is already handled and the connection is rebuilt, but logging it at ERROR causes Sentry to file it as an issue. Branch on the existing isReading liveness flag: ERROR for genuine failures while still reading, DEBUG for the benign teardown race. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sentry issue EVERYDAY-POLLER-75Z files a Jira ticket whenever
SubConnection.setMaxInFlightlogs aSocketException: Socket closedat ERROR. The exception is a benign teardown race: during normal connection churn (nsqd rollout, k8s pod cycling, peer EOF) the dedicated read thread callsConnection.close()— settingisReading = falseand closing the socket — whileSubscription.distributeMaxInFlightis concurrently callingsetMaxInFlight→flush()on that same connection. TheIOExceptionis already caught, the connection is closed and rebuilt bycheckConnections, and no messages are lost (inFlight:0). Only the log level is wrong. This change branches the existing catch block on theisReadingliveness flag — the same discriminator the siblingPubConnection.checkConnectionLiveness()already uses — so genuine failures stay at ERROR while the benign race logs at DEBUG and no longer surfaces in Sentry.Change Summary
src/main/java/com/sproutsocial/nsq/SubConnection.java— In thecatch (IOException e)block ofsetMaxInFlight(int, boolean), replaced the unconditionallogger.error("setMaxInFlight failed. con:{}", ...)with a branch onisReading: log at ERROR when the connection still believes it is reading (a genuine, actionable failure), and at DEBUG whenisReading == false(the read thread already tore the socket down — a benign race).close()is still called unconditionally in both branches, so behaviour is unchanged. No method signatures, control flow, or callers were touched.src/test/java/com/sproutsocial/nsq/SubConnectionTest.java(new) — First unit test in the repo; runs under surefire with no Docker. Builds aSubConnectionwith mockedClient/Subscription/Subscriber, swaps in anOutputStreamwhoseflush()throwsIOException("Socket closed")(reproducing the race), and attaches a LogbackListAppenderto thecom.sproutsocial.nsq.SubConnectionlogger.logsErrorWhenFlushFailsWhileStillReadingasserts the failure is logged at ERROR whenisReading == trueand thatclose()ran;logsDebugWhenFlushFailsAfterConnectionTornDownsetsisReading = falseand asserts the failure is logged at DEBUG with no ERROR event.Suggested Assignee
@alex-dyla — alex.dyla@sproutsocial.com (Alex Dyla)
Related Links