testnet: dead-letter queue for events that repeatedly fail to persist

[Depo-dev]

Related to #208. Today a persistently failing event can either block the pipeline or be dropped. Neither is acceptable on testnet, where the failure will be someone else data.

Scope

• Route events that exhaust retries to a dead-letter store with the failure reason and raw payload.
• Ensure the pipeline advances past a poison event instead of stalling.
• Alert when the DLQ is non-empty; a silent DLQ is the same as data loss.
• Provide a documented replay path once the underlying bug is fixed.

Done when

• A poison event lands in the DLQ, ingestion continues, an alert fires, and replay works.

[obswrld]

@obswrld has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

I would like to work on this Issue

ℹ️ Repo Maintainers: To accept this application, review their application or assign @obswrld to this issue.

[Ola-Olamijuwon]

@Ola-Olamijuwon has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

I'd like to work on this please.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Ola-Olamijuwon to this issue.

[Salmatcre8]

@Salmatcre8 has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

I’d like to claim this issue. Implementing a resilient Dead-Letter Queue (DLQ) for ingestion failures is essential to isolate poison-pill events, prevent head-of-line blocking, and guarantee zero silent data loss on testnet.

My implementation plan:

Bounded Retry & DLQ Routing: Wrap the event persistence pipeline in an exponential-backoff retry policy (e.g., max 3 attempts). When retries are exhausted, route the event to a dedicated dead_letter_events store containing the raw XDR/JSON payload, error stack trace, attempt count, contract/topic IDs, and ledger sequence.

Non-Blocking Ingestion Continuity: Ensure routing to the DLQ returns a success signal to the batch orchestrator, allowing the cursor/pipeline to safely advance past poison events without stalling.

DLQ Monitoring & Metric Alerting: Export a Prometheus gauge metric indexer_dlq_unresolved_events_total and fire an immediate alert on any non-zero DLQ insertion.

Replay & Reprocessing CLI/Utility: Build a deterministic replay command (trident replay-dlq --id= | --all) that re-ingests resolved dead-letter records through the parser and purges/marks them upon successful write.

End-to-End Verification Test: Add integration tests proving a poison event routes to the DLQ, pipeline ingestion advances unblocked, metrics/alerts trigger, and subsequent replay succeeds after the fix.

Please assign this to me!

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Salmatcre8 to this issue.

[drips-wave]

Congratulations, @Salmatcre8! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on August 31, 2026.

🧑‍💻 @Salmatcre8: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊