fix: make sample-catalog fetches resilient to transient failures#580
Merged
Conversation
Replaces the silent catch-all in the raw.githubusercontent fetch layer (which randomly dropped templates on rate-limit/5xx) with a retrying fetch: distinguishes 404 (genuinely missing) from transient 429/5xx/network errors, retries the latter with jittered exponential backoff honoring server Retry-After, and caps per-wait and total backoff. A transient failure that survives retries now aborts the run loudly instead of producing an incomplete catalog.
huimiu
approved these changes
Jul 23, 2026
Yimin-Jin
added a commit
that referenced
this pull request
Jul 23, 2026
After merging template/dev (PR #580 added fetch-layer retry with delay/parseRetryAfterMs/computeBackoffMs), the LLM retry path's local sleep/retryAfterMsFromResponse duplicated those helpers. Drop the duplicates and reuse the shared delay/parseRetryAfterMs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The catalog generator fetched each sample's
azure.yaml/README.mdfromraw.githubusercontent.comand swallowed every error asreturn null, treating it asfile does not exist. Under the ~90-request serial scan, an occasional rate-limit (429) / 5xx / network blip therefore silently dropped a real template — the same upstream commit produced 88 vs 89 templates across two runs (the C# hello-world sample vanished from one).Changes
Replaces the silent catch-all with a resilient fetch layer:
HttpErrorcarries the status code so callers can distinguish outcomes.fetchWithRetryretries only transient failures (429 / 5xx / network); a 404 is permanent and not retried.Retry-Afterheader when present.FETCH_MAX_DELAY_MS) and a total backoff budget (FETCH_TOTAL_BACKOFF_BUDGET_MS) prevent a rate-limit storm from stalling past the job timeout — once exceeded the run fails fast.fetchAzureYamlnow returnsnullonly on a genuine 404; a transient failure that survives retries is re-thrown, so the run fails loudly instead of emitting an incomplete catalog.Notes
sample-catalog.jsonis regenerated by the sync workflow and is intentionally NOT included.Validation
node --checkpasses. Recommend a manualworkflow_dispatchrun before merge to confirm the full scan still succeeds.