feat(indexer): reconcile indexed event counts against the RPC source

[Miracle656]

Closes #511

Problem

Nothing proved that what Trident indexed matches what the chain actually emitted — silent under-indexing was invisible, and the API returned confident, incomplete answers.

What this does

A reconciliation loop inside the indexer (crates/indexer/src/reconcile.rs): every RECONCILE_INTERVAL_MS (default 10 min) it re-fetches the most recent RECONCILE_LEDGER_SPAN settled ledgers (default 400 — deliberately the same window as the nightly testnet-correctness suite, i.e. a meaningful testnet window) sitting RECONCILE_TIP_MARGIN behind the tip and never past the ingest cursor, and compares per-ledger event counts against the database.

The comparison mirrors the ingest pipeline exactly — same server-side filter plan (allowlist + topics, incl. the degraded index-all fallback), the diagnostic gate, the failed-call skip, per-contract index_from — because indexed counts are not raw RPC counts and comparing unlike sets would make every report a false positive. The DB side counts soroban_events plus parse_errors (seen-but-undecodable is accounted for, not missing). A parity test pins the mirrored rules.

Discrepancies are reported with the specific ledger ranges involved (contiguous discrepant ledgers coalesced, each logged with both counts), in both directions — missing AND extra events. A page-cap-truncated walk clamps the compare window to the fully-walked prefix so un-walked ledgers can never surface as fake discrepancies. Six new trident_indexer_reconcile_* metrics (described/seeded) feed two alerts with runbook entries: TridentIndexerReconciliationMismatch (any discrepant ledger, sustained across passes) and TridentIndexerReconciliationFailing (passes keep aborting — silence is unknown, not clean).

Continuous vs on-demand — decided and documented: a slow continuous in-process loop, default ON (disabling logs a loud warning), because that surfaces under-indexing in minutes at negligible RPC cost; trident-backfill --dry-run remains the on-demand path for arbitrary historical ranges. RpcClient::get_latest_ledger loses its #[cfg(test)] gate — its own comment asked for exactly this production caller.

Done-when check

Integration tests (real Postgres + mock RPC) prove it: an under-indexed and an over-indexed ledger are reported as exactly the right ranges with the right counts; failed-call/diagnostic events and allowlist/index_from boundaries never read as missing; parse-error rows count as accounted for. Runs over a 400-ledger window continuously; choice documented. Full workspace suite green; promtool clean; env vars documented (CI-enforced).

[drips-wave]

@Miracle656 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Miracle656]

Rebased onto current dev. Two things beyond the mechanical rebase: (1) the metrics module lost its debug-fallback counter on dev (the decoder is exhaustive now), so this PR's stale references to it are gone and only the six reconcile series remain; (2) the reconcile tests' fixtures used a global table wipe and ledgers 850–900 — a neighbourhood dev's new commit_page test now also seeds — so under the parallel harness the suites destroyed each other's rows. The fixtures moved to an untouched window (30201–30600), cleanup is window-scoped, and the two tests serialize over the genuinely global state they mutate (allowlist + cursor row). Full suite green locally apart from two pre-existing failures that reproduce identically on the base commit in this environment.