webhookDispatcher.backoffMs() has no jitter — synchronized retries thundering-herd a subscriber's endpoint right as it recovers

[prodbycorne]

Overview

webhookDispatcher.js's retry backoff is purely deterministic — the same attempt count always produces the exact same delay — with no randomization (jitter) at all. Combined with webhookRetryWorker's fixed 5-second poll interval and 25-item batch size, this creates a textbook synchronized-retry thundering-herd risk against any subscriber whose endpoint experiences an outage affecting multiple in-flight deliveries at once.

function backoffMs(attemptsCompleted) {
  const base = config.webhooks.retryBaseMs;
  const factor = config.webhooks.retryFactor;
  return base * factor ** (attemptsCompleted - 1);
}

With the documented defaults (WEBHOOK_RETRY_BASE_MS=30000, WEBHOOK_RETRY_FACTOR=2), every delivery that fails on its first attempt computes nextRetryAt = now + 30000ms — exactly. Every delivery that fails on its second attempt (regardless of which delivery, which webhook, or which subscriber) computes nextRetryAt = now + 60000ms — exactly. There is no + Math.random() * jitterMs term anywhere in this function or anywhere else in the retry path (attempt(), deliveryRepo.scheduleRetry).

Concretely: if a subscriber's endpoint goes down for, say, 90 seconds during a deploy or a brief outage, every delivery attempted against it during that window fails at roughly the same wall-clock moments (driven by real, independent event timing — new pool.*/airdrop.failed events as they occur), and — because backoffMs is a pure function of attempt count only, not of the delivery's specific identity or any random offset — every one of those failed deliveries computes the same backoff delay for the same attempt number, converging their nextRetryAt values into tight clusters. deliveryRepo.scheduleRetry stores these in a single shared Redis sorted set (webhooks:retries), and webhookRetryWorker.tick() drains up to retryBatchSize (default 25) due items every retryPollMs (default 5000ms) — so when a cluster of retries becomes due, the worker processes them in a burst of up to 25 near-simultaneous outbound POST requests to what may be the same recovering subscriber endpoint, right as (or before) it's likely still coming back up from whatever caused the original outage. This is the exact self-inflicted thundering-herd pattern jitter exists to prevent, and it applies with equal force to every subscriber in the system whenever their endpoint has any correlated multi-delivery outage — not a rare edge case, but the expected behavior for the single most common failure mode a webhook consumer will ever experience.

Requirements

• Add jitter to backoffMs() — a randomized offset (commonly "full jitter": Math.random() * computedDelay, or "decorrelated jitter") so that deliveries failing at the same attempt count and around the same wall-clock moment do not all compute identical nextRetryAt values.
• Ensure the jitter is deterministic-enough for testability (inject a random source, as this codebase already does elsewhere — e.g. CircuitBreaker's options.now/options.logger injection pattern in utils/circuitBreaker.js — rather than calling Math.random() directly and making the function hard to test precisely).
• Consider whether webhookRetryWorker's retryBatchSize/retryPollMs need retuning once jitter spreads retries out more evenly, though that's a secondary concern to adding the jitter itself.

Acceptance Criteria

• backoffMs(attemptsCompleted) no longer returns an identical value for every call with the same attemptsCompleted — repeated calls with the same input produce a distribution of delays within a documented, bounded range around the previous deterministic value.
• A test simulates many deliveries failing at the same attempt count within the same tick and asserts their computed nextRetryAt values are spread across a range, not identical.
• The jitter source is injectable/mockable for deterministic test assertions (min/max bounds respected, not just "looks random").
• shouldRetry/maxAttempts semantics are unaffected — only the timing of retries changes, not whether/how-many retries occur.

Additional Notes

More precise references

• src/services/webhookDispatcher.js:13-17 (backoffMs): confirmed the exact formula base * factor ** (attemptsCompleted - 1), confirmed no randomization term anywhere in the function.
• src/services/webhookDispatcher.js:108-118 (attempt, retry-scheduling branch): confirmed const delayMs = backoffMs(attempts); const nextRetryAt = new Date(Date.now() + delayMs).toISOString(); await deliveryRepo.scheduleRetry(delivery.id, Date.now() + delayMs); — confirmed Date.now() + delayMs is the only source of variation between two deliveries at the same attempt count, and that variation is purely a function of when each delivery happened to fail, not of anything designed to spread retries apart.
• src/repositories/deliveryRepository.js:114-117 (scheduleRetry): confirmed this writes to a single shared Redis sorted set webhooks:retries, keyed by nextRetryAtMs as the score — confirming that clustered nextRetryAt values across many different deliveries/webhooks genuinely collide into the same narrow score range in one shared structure, not just conceptually but in the literal data structure a single worker drains from.
• src/jobs/webhookRetryWorker.js:21 (tick): confirmed deliveryRepo.popDueRetries(Date.now(), config.webhooks.retryBatchSize) — default retryBatchSize = 25 per src/config.js (WEBHOOK_RETRY_BATCH default 25), confirmed retryPollMs default 5000.
• src/config.js webhooks.retryBaseMs/retryFactor: confirmed defaults 30000/2, giving deterministic delays of exactly 30s, 60s, 120s for attempts 1, 2, 3 (with maxAttempts default 3, meaning realistically most deliveries only ever see the 30s and 60s tiers before exhausting retries).

Additional edge cases

• Jitter needs to be careful not to ever produce a negative or zero delay, and should probably stay bounded close to (not wildly exceeding) the deterministic base delay, so retry timing remains roughly predictable for operators reasoning about worst-case delivery latency — "full jitter" (Math.random() * delay) versus "jitter added on top" (delay + Math.random() * someSpreadMs) have different tradeoffs here and are worth deciding deliberately rather than picking arbitrarily.
• This compounds with the already-open No ordering guarantee for webhook deliveries to the same endpoint across independent dispatch() calls #78 ("No ordering guarantee for webhook deliveries to the same endpoint across independent dispatch() calls") in that both concern the timing behavior of the retry/delivery system as experienced by a subscriber — worth a shared review pass, though the fixes are largely independent.

Test/reproduction plan

const delays = new Set();
for (let i = 0; i < 100; i++) {
  delays.add(webhookDispatcher.backoffMs(1)); // simulate 100 deliveries all failing on attempt 1 "at once"
}
expect(delays.size).toBeGreaterThan(1); // currently fails: delays.size === 1, every call returns exactly 30000

Cross-references

• No ordering guarantee for webhook deliveries to the same endpoint across independent dispatch() calls #78 (open) — "No ordering guarantee for webhook deliveries to the same endpoint across independent dispatch() calls." Related timing/delivery-experience concern on the same subsystem, different specific mechanism.
• Missing production metrics for webhook delivery outcomes, retry-queue depth, and price-source failure rates #93 (open) — "Missing production metrics for webhook delivery outcomes, retry-queue depth, and price-source failure rates." A retry-queue-depth metric (which Missing production metrics for webhook delivery outcomes, retry-queue depth, and price-source failure rates #93 asks for) would make the clustering this issue describes directly observable as periodic depth spikes — worth noting as motivating evidence for whoever implements that metric.

[circleboyslimited]

The webhook retry mechanism has a classic synchronized-retry, or thundering-herd, risk because its exponential backoff is completely deterministic.

backoffMs() produces the exact same delay for every delivery at a given attempt count. With the documented defaults, all first-attempt failures are scheduled for exactly 30 seconds later, all second-attempt failures for exactly 60 seconds later, and so on. There is no jitter in backoffMs(), attempt(), or deliveryRepo.scheduleRetry().

Combined with the retry worker's fixed 5-second polling interval and 25-item batch size, an outage affecting multiple deliveries to the same subscriber can cause those deliveries to become eligible simultaneously and be retried in synchronized batches. This can concentrate load precisely when the subscriber is recovering, potentially prolonging the outage or creating a retry storm.

I have experience working with distributed systems where retry coordination can amplify transient failures. Adding bounded random jitter to the exponential backoff would spread retry attempts over time while preserving the existing backoff growth, substantially reducing the synchronized-load risk without changing the fundamental retry semantics.

[grantfox-oss]

🦊 GrantFox — @circleboyslimited has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #128)
2. Your PR will be reviewed by the SmartDropLabs maintainers

Good luck! Track your progress on GrantFox.