Skip to content

Bounded concurrency for the gig/repin/event sweeps + fix the dead webhook retry helper (#236–#239) - #359

Merged
meshackyaro merged 11 commits into
trustflow-protocol:mainfrom
stephanieoghenemega-eng:perf/bounded-concurrency-sweeps-236-239
Aug 31, 2026
Merged

Bounded concurrency for the gig/repin/event sweeps + fix the dead webhook retry helper (#236–#239)#359
meshackyaro merged 11 commits into
trustflow-protocol:mainfrom
stephanieoghenemega-eng:perf/bounded-concurrency-sweeps-236-239

Conversation

@stephanieoghenemega-eng

Copy link
Copy Markdown
Contributor

Four issues, all about the background workers processing their sweeps one item at a time. A shared mapWithConcurrency helper (commit 1) is applied to the three sweeps; #239 is a separate dead-code + bug fix.

#236GigExpiryWorkerService sweeps sequentially

runOnce() awaited gigService.expire(gig.id) per gig in a plain loop; each expire() appends an outbox row the relay delivers with up-to-3 retries, so a batch of simultaneously-expiring gigs against a slow endpoint serialised all of that latency and could still be running when the next tick fired.

  • Bounded-concurrent (GIG_EXPIRY_SWEEP_CONCURRENCY, default 8).
  • A failed expire is counted + logged, not fatal to the sweep.
  • New sweeping flag: a tick is skipped (and logged) while a previous runOnce() is still in flight. (The Redis lock already covered the multi-instance case; this covers a single instance whose sweep outruns its interval.)

#237RepinWorkerService reconciles sequentially

Same shape: await pinningService.reconcile(cid) per degraded/failed pin, each doing per-provider network calls. Now bounded-concurrent (IPFS_REPIN_SWEEP_CONCURRENCY, default 8), with the per-CID try/catch kept inside the worker so one unreachable provider stays isolated to its CID, plus the same overlapping-sweep guard.

#238EventIngestionService processes events sequentially

ingestEvents / ingestSingleLedger looped await processEvent(event) for every fetched event, so a 100-ledger window's events processed fully serially each 5s poll tick.

New processEventBatch(events) (used by both):

  • Phase 1 — every event with no topic[1] (escrow_created, unknown types) runs strictly in original order. escrow_created carries no escrow id, so it can't be correlated to a specific later escrow_funded; running all id-less events first guarantees no keyed handler observes a missing escrow.
  • Phase 2topic[1]-keyed events are grouped per escrow id; groups run in parallel (EVENT_PROCESSING_CONCURRENCY, default 8), sequential within a group, so same-escrow ordering (funded → released) is preserved while independent escrows don't block each other.

Full concurrency-safety analysis is in the method's doc comment. Test asserts a slow escrow_funded for escrow A does not delay escrow B and does not reorder A's own events.

#239retry.helper.ts is dead code + isRetryable() bug

withRetry/isRetryable were exported but never imported — WebhookService.sendWithRetry re-implemented the loop inline. And isRetryable did error.message.includes('5'), matching the digit 5 anywhere (a URL like /v5/, 512 bytes, port 5000).

  • sendWithRetry now calls withRetry(() => this.send(...), retries, 1000, isRetryable) — one implementation.
  • isRetryable(error: unknown) matches transient network errors (ECONNREFUSED, ETIMEDOUT, …) and a real HTTP 5xx — the bare 3-digit message send() throws (new Error(String(statusCode))), or "HTTP 5xx …". A 4xx or a message that merely contains a 5 is not retried.
  • withRetry gained a shouldRetry predicate so a non-retryable failure stops immediately instead of burning all attempts.
  • retry.helper.spec.ts added.

Not built or run in this environment (per your instruction). New/changed code targets tsc --noEmit and the repo's prettier/eslint config; tests follow the existing *.spec.ts patterns (jest.Mocked<Pick<…>>, fakeLock, flushPromises, Test.createTestingModule).

Closes #236, closes #237, closes #238, closes #239

`mapWithConcurrency(items, n, worker)` runs `worker` over `items` with at
most `n` in flight, returns `PromiseSettledResult`s in input order, and
never rejects. Replaces the `for..of await` sweeps whose runtime was the
sum of every item's latency (trustflow-protocol#236, trustflow-protocol#237, trustflow-protocol#238).
…rlapping sweeps (trustflow-protocol#236)

`runOnce()` fanned out `expire()` one gig at a time; each `expire` appends an
outbox row the relay delivers with retries, so a big batch with a slow
endpoint serialised all that latency and could outrun the sweep interval.
Now bounded-concurrent (GIG_EXPIRY_SWEEP_CONCURRENCY, default 8), a failed
expire is counted+logged not fatal, and a `sweeping` flag skips a tick while
a previous sweep is still running.

Closes trustflow-protocol#236
…low-protocol#237)

`runOnce()` awaited `pinningService.reconcile(cid)` serially; each call makes
per-provider network requests, so sweep time scaled with pin count x latency.
Now bounded-concurrent (IPFS_REPIN_SWEEP_CONCURRENCY, default 8) with the
per-CID try/catch kept inside the worker so one bad CID stays isolated, plus
the same overlapping-sweep guard.

Closes trustflow-protocol#237
…rve per-escrow order (trustflow-protocol#238)

New `processEventBatch()` (used by `ingestEvents` and `ingestSingleLedger`):
phase 1 runs every id-less event (escrow_created, unknown types) strictly in
original order so no later keyed event can observe a missing escrow; phase 2
groups the `topic[1]`-keyed events per escrow and runs the groups in
parallel (EVENT_PROCESSING_CONCURRENCY, default 8), sequential within a
group. Full concurrency-safety analysis is in the method's doc comment.

Closes trustflow-protocol#238
trustflow-protocol#239)

`retry.helper.ts` was dead code — `WebhookService.sendWithRetry` re-
implemented the loop inline. It now calls `withRetry(...)`, and `isRetryable`
no longer classifies any message containing the digit 5 as retryable: it
matches transient network errors and a real HTTP 5xx status (the bare
3-digit message `send()` throws, or `HTTP 5xx`). `withRetry` gained a
`shouldRetry` predicate so a 4xx stops immediately.

Closes trustflow-protocol#239
@drips-wave

drips-wave Bot commented Aug 31, 2026

Copy link
Copy Markdown

@stephanieoghenemega-eng Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants