fix: Handle non-429 retriable errors in backoff_wait_generator#61
Open
ReubenFrankel wants to merge 2 commits into
Open
fix: Handle non-429 retriable errors in backoff_wait_generator#61ReubenFrankel wants to merge 2 commits into
backoff_wait_generator#61ReubenFrankel wants to merge 2 commits into
Conversation
The backoff wait generator assumed every exception it received was a RetriableAPIError with a populated response and read `response.status_code` directly. But the SDK retries on a wider set of exceptions (ConnectionError, Timeout, ConnectionResetError, etc.), none of which carry a response, so a dropped connection crashed the tap with `AttributeError: 'NoneType' object has no attribute 'status_code'`. Rewrite `backoff_wait_generator` to decide per-exception: HTTP 429s use the `rate-limit-msec-left` header wait, everything else falls back to the default exponential backoff (driven via an inline `backoff.expo` generator). This removes the previous raise/`suppress` handshake, which only recognised RetriableAPIError and could not fall back cleanly for connection-level errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
backoff_wait_generator
Contributor
Author
|
@edgarrmondragon Do you know if there is a better way we can be handling backoff with respect to rate-limit response headers here? |
No, I don't think there's a better way to respect RateLimit headers when they're available, and fall back to exponential backoff when they're not. |
Contributor
Author
|
Curious if meltano/sdk#2012 (comment) would make this easier... 🤔 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A transient
requests.exceptions.ConnectionError(RemoteDisconnected) crashed the tap during retry handling:The backoff wait generator assumed every exception it received was a
RetriableAPIErrorwith a populated.response. But the SDK'sbackoff.on_exceptionretries on a wider family —ConnectionResetError,RetriableAPIError,Timeout,ConnectionError,ChunkedEncodingError,ContentDecodingError— and the connection-level ones carry no response, so.response.status_coderaisedAttributeError.Fix
backoff_wait_generatornow decides per-exception:rate-limit-msec-leftheader (unchanged behaviour).backoff.expogenerator.This replaces the previous
backoff_runtimeraise/suppresshandshake, which only recognisedRetriableAPIErrorand couldn't fall back cleanly for connection-level errors. Because the check runs on every retry, a connection blip followed by a real 429 now correctly honours the 429's header instead of being locked into exponential backoff.Verification
Drove the generator manually across the relevant exception sequences — all return sensible waits with no crash:
Unit/schema tests pass. The 6 failing tests in
test_core.pyare pre-existing live-API integration tests (record-limit / schema-match against real data) that fail identically on the base commit and don't exercise backoff.🤖 Generated with Claude Code