feat(indexer): dead-letter queue for events that repeatedly fail to persist

[Salmatcre8]

Closes #508 (related to #208)

Problem

A persistently failing event blocked the pipeline forever: commit failures are StorageError → Retryable, so the streamer refetched and re-failed the identical page every poll with no budget and no capture path. Parse failures have had a DLQ since #414; persistence failures had nothing — e.g. an event whose malformed ledger timestamp fails column conversion deterministically.

What this does

• Bounded budget, then isolation: transient failures behave exactly as before (error, hold cursor, retry). Only the same page (identity = its first event's paging token, stable under tip growth) failing PERSIST_FAILURE_BUDGET consecutive polls triggers isolation: events commit one at a time (each with its token projection, and each with its own bounded retry so a transient DB blip during isolation cannot mass-dead-letter healthy events); events that still fail are captured in the new dead_letter_events table (migration 0027) with full payload, reason, and attempt count, deduplicated on the natural key.
• Never data loss: an event leaves the pipeline only by committing or by a durable DLQ row — if the capture write itself fails, the cursor holds and the page retries. The realistic tail-poison case (malformed ledgerClosedAt poisoning the page's last event AND the LedgerMeta row built from it) advances via a cursor-only fallback that drops the poisoned ledger row rather than wedging — regression-tested.
• A silent DLQ is the same as data loss: trident_indexer_persist_dead_letter_backlog (published at the top of every poll so failing cycles still refresh it) drives the new TridentIndexerPersistDeadLetterBacklog alert, which fires for as long as any row exists; rows are removed only by the documented replay procedure. trident_indexer_persist_dead_lettered_total counts captures only after a durable insert. Both metrics described/seeded; promtool clean.
• Replay documented honestly in the runbook: fix the bug, re-ingest with the idempotent backfill CLI (scope stated explicitly: backfill restores soroban_events rows only — no outbox delivery or token projections), verify, then clear replayed rows via a natural-key join DELETE.
• Fixes the /admin/dead-letter endpoint (selected a non-existent column and scanned a UUID into an i64 — it 500'd against the real schema) and extends it to list both queues.

Done-when check (integration-tested against real Postgres + Redis + mock RPC)

A poison event lands in the DLQ with its reason and round-trippable payload → ingestion continues (good events in the same page indexed, cursor durably advanced) → the backlog alert has a live series → replay is documented. Budget-reset and tail-poison edge cases have dedicated tests. Full workspace suite green.

[drips-wave]

@Salmatcre8 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Salmatcre8]

Rebuilt rather than rebased: #208 landed the dead-letter queue itself on dev (failed_events, whole-page → per-event isolation, cursor advance past captured events) while this PR was in flight, and mechanically merging two parallel implementations of the same queue would have doubled the machinery. The PR is now the small delta that completes #508's remaining criteria on top of #208's foundation:

• Dedup (migration 0030): pending rows unique per event, keyed by the same (contract, ledger, index) triple 0025 made canonical — redelivered failures fold their attempt counts into one row, so the queue's pending count means distinct poisoned events; replayed rows are history and never block a fresh failure. Existing duplicates are collapsed with attempts preserved.
• Separated metrics: the persist path no longer increments the parse-DLQ counter — trident_indexer_persist_dead_lettered_total is its own series, and both counters are now described accurately.
• Backlog observability: trident_indexer_persist_dead_letter_backlog (pending rows, refreshed each active cycle and on every dead-letter write), the TridentIndexerPersistDeadLetterBacklog alert that stays up while any pending row exists, and a runbook covering diagnosis, backfill replay (with its outbox/projection scope limits stated), and the replayed_at bookkeeping that resolves the alert.
• Env-gated test: redelivery collapses to one row with folded attempts; a replayed row doesn't block recording a regression.

Verified against a live Postgres with the migration applied; promtool-clean; fmt/clippy clean. (Two pre-existing db::tests failures on my local throwaway DB reproduce identically on the base commit — partition-provisioning environment gaps, not this diff; the rust-integration job is the arbiter.)