Skip to content

fix: make sample-catalog fetches resilient to transient failures#580

Merged
Yimin-Jin merged 1 commit into
template/devfrom
yimin/fix-catalog-fetch-retry
Jul 23, 2026
Merged

fix: make sample-catalog fetches resilient to transient failures#580
Yimin-Jin merged 1 commit into
template/devfrom
yimin/fix-catalog-fetch-retry

Conversation

@Yimin-Jin

Copy link
Copy Markdown
Collaborator

Summary

The catalog generator fetched each sample's azure.yaml / README.md from raw.githubusercontent.com and swallowed every error as return null, treating it as file does not exist. Under the ~90-request serial scan, an occasional rate-limit (429) / 5xx / network blip therefore silently dropped a real template — the same upstream commit produced 88 vs 89 templates across two runs (the C# hello-world sample vanished from one).

Changes

Replaces the silent catch-all with a resilient fetch layer:

  • HttpError carries the status code so callers can distinguish outcomes.
  • fetchWithRetry retries only transient failures (429 / 5xx / network); a 404 is permanent and not retried.
  • Backoff uses full jitter exponential backoff and honors a server Retry-After header when present.
  • Per-wait cap (FETCH_MAX_DELAY_MS) and a total backoff budget (FETCH_TOTAL_BACKOFF_BUDGET_MS) prevent a rate-limit storm from stalling past the job timeout — once exceeded the run fails fast.
  • fetchAzureYaml now returns null only on a genuine 404; a transient failure that survives retries is re-thrown, so the run fails loudly instead of emitting an incomplete catalog.

Notes

  • All knobs are overridable via env vars for local debugging; defaults are conservative.
  • Zero new dependencies (the script is intentionally dependency-free).
  • sample-catalog.json is regenerated by the sync workflow and is intentionally NOT included.

Validation

  • node --check passes. Recommend a manual workflow_dispatch run before merge to confirm the full scan still succeeds.

Replaces the silent catch-all in the raw.githubusercontent fetch layer (which randomly dropped templates on rate-limit/5xx) with a retrying fetch: distinguishes 404 (genuinely missing) from transient 429/5xx/network errors, retries the latter with jittered exponential backoff honoring server Retry-After, and caps per-wait and total backoff. A transient failure that survives retries now aborts the run loudly instead of producing an incomplete catalog.
@Yimin-Jin
Yimin-Jin requested a review from huimiu as a code owner July 23, 2026 06:19
@Yimin-Jin
Yimin-Jin merged commit 68b7b31 into template/dev Jul 23, 2026
5 checks passed
@Yimin-Jin
Yimin-Jin deleted the yimin/fix-catalog-fetch-retry branch July 23, 2026 08:09
Yimin-Jin added a commit that referenced this pull request Jul 23, 2026
After merging template/dev (PR #580 added fetch-layer retry with delay/parseRetryAfterMs/computeBackoffMs), the LLM retry path's local sleep/retryAfterMsFromResponse duplicated those helpers. Drop the duplicates and reuse the shared delay/parseRetryAfterMs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants