[FEAT] Distributed Multi-Node Webhook Event Delivery Engine & DLQ Recovery

[jotel-dev]

telegram link : t.me/nullifiersystem

1. Summary & Core Promise

Velo's existing webhook dispatch logic in apps/api/src/lib/webhook.ts sends HTTP notifications synchronously. If an external client webhook endpoint is down or slow, the API request thread blocks, leading to socket exhaustion and missed payment notifications (sendRefundAlert()).
This feature implements a Distributed Multi-Node Webhook Event Delivery Engine & Dead-Letter Queue (DLQ) Recovery System. It offloads webhooks to a Redis Stream queue (velo:webhook-delivery-queue), signs payloads with HMAC-SHA256 signatures (x-velo-signature), enforces exponential backoff retries (up to 5 attempts), provides a manual DLQ replay API, and locks DB event deliveries (SELECT FOR UPDATE).

2. Background & Architectural Risks

• Synchronous Thread Blocking: External webhook failures block Fastify server event loops, causing cascading API timeouts.
• Un-Signed Webhooks: Un-signed webhook payloads allow malicious third parties to spoof trade status alerts (COMPLETED / REFUNDED).

---

3. Database Layer Specifications

Migration SQL (030_add_distributed_webhook_pipeline.sql)

CREATE TYPE webhook_delivery_status AS ENUM ('QUEUED', 'DELIVERED', 'FAILED', 'DEAD_LETTER');
CREATE TABLE webhook_endpoints (
    endpoint_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id VARCHAR(64) NOT NULL,
    target_url TEXT NOT NULL,
    secret_key VARCHAR(64) NOT NULL,
    is_active BOOLEAN NOT NULL DEFAULT TRUE,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE webhook_delivery_logs (
    delivery_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    endpoint_id UUID NOT NULL REFERENCES webhook_endpoints(endpoint_id) ON DELETE CASCADE,
    event_type VARCHAR(64) NOT NULL,
    payload JSONB NOT NULL,
    signature_header VARCHAR(64) NOT NULL,
    attempt_count INT NOT NULL DEFAULT 0,
    status webhook_delivery_status NOT NULL DEFAULT 'QUEUED',
    last_response_code INT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

---

4. Backend Route & Service Layer Specifications

Route: POST /api/v1/webhooks/endpoints & POST /api/v1/webhooks/dlq/replay

1. Endpoint Registration: Generates 32-byte HMAC-SHA256 secret key for clients.
2. DLQ Replay: Re-enqueues failed dead-letter webhook deliveries (SELECT FOR UPDATE).

---

5. Background Processors / Workers

Redis Webhook Delivery Worker (apps/api/src/lib/workers/webhookDeliveryWorker.ts)

• Queue: Redis Stream velo:webhook-delivery-queue.
• Backoff: Exponential retry (delayMs = 1000 * 2^attempt). Max 5 retries before routing to DLQ.
• Signature: Generates x-velo-signature using HMAC-SHA256(payload, secret_key).

---

6. Frontend / UI Component Specifications

Component: mobile/frontend/src/pages/WebhookSettings.tsx

• Developer portal UI to register target URLs, view HMAC secret keys, monitor delivery logs, and trigger manual DLQ retries.

---

7. Rigor & Test Plan

1. HMAC Signature Unit Test (webhook.test.ts): Verify x-velo-signature calculation against payload.
2. Delivery Retry & DLQ Integration Test: Simulate 5 consecutive HTTP 500 errors. Assert job routes to DLQ state.

---

8. Relevant Files Inventory

New Files to Create

• apps/api/src/db/migrations/030_add_distributed_webhook_pipeline.sql
• apps/api/src/routes/webhooks.ts
• apps/api/src/lib/workers/webhookDeliveryWorker.ts
• apps/api/src/routes/__tests__/webhooks.test.ts
• mobile/frontend/src/pages/WebhookSettings.tsx

Existing Files to Modify

• apps/api/src/lib/webhook.ts
• apps/api/src/routes/cash.ts
• apps/api/src/app.ts
• packages/shared/src/index.ts

---

9. Acceptance Criteria

• All webhook deliveries offloaded asynchronously to Redis Streams.
• Webhook payloads include valid x-velo-signature HMAC headers.
• Failed deliveries retry up to 5 times before moving to DLQ.
• Developers can manually replay DLQ events via API/UI.

---

10. Contributor Notes

• ⚠️ Security Rule: NEVER send webhook payloads over cleartext HTTP; enforce HTTPS target URLs in production.

[chiemeliesteve4-art]

Good day maintainer, i will implement the distributed webhook delivery pipeline using Redis Streams, move deliveries to asynchronous workers with exponential retries and DLQ handling, and secure payloads with HMAC-SHA256 signatures over HTTPS endpoints. I will also add endpoint registration, manual DLQ replay through the API/UI, database migrations, and tests for signatures and retry-to-DLQ behavior. Thank you.

[ReinaMaze]

GM Maintainer, Can I get assigned to work on this?

I’ll implement the distributed webhook delivery system with Redis Streams, HMAC-SHA256 signing, retry/backoff handling, and DLQ replay.
I’ll also add the webhook registration/replay APIs, delivery worker, frontend settings UI, and tests covering signatures, retries, and DLQ behavior.

ETA: 10 hours

[wheval]

Hello i would love to work on this,i can open the PR in a few hours completing the task requirements.

[grantfox-oss]

🦊 GrantFox — @wheval has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #445)
2. Your PR will be reviewed by the Nullifier-Systems maintainers

Good luck! Track your progress on GrantFox.

[grantfox-oss]

🎉 This issue has been marked as completed on GrantFox!

@wheval's PR #450 was approved and merged by @jotel-dev.

🏆 @wheval: You earned 40 FoxPoints for this contribution! Your current tier: Builder (1,302 total points). Track your full progress on GrantFox.

👏 Great work, @wheval! Keep contributing to Nullifier-Systems.