Skip to content

The indexer does not track which ledgers were partially processed due to page-level RPC errors, creating undetectable gaps in the index #295

Description

@Lost-Z

Summary

When the indexer's pagination loop encounters an RPC error on a non-first page (e.g. page 3 of a 5-page batch), it exits the loop and advances the cursor to the tip. Events on the failed pages are silently skipped. The error is logged and counted in errors_total, but the specific ledger range that was missed is not recorded. There is no way to identify or backfill the gap without replaying the entire retention window.

Expected Behavior

Partial-page failures should record the affected ledger range so operators can trigger a targeted backfill.

Proposed Solution

  • Add a missed_ranges table to the schema: (from_ledger, to_ledger, reason, detected_at)
  • When a page fetch fails mid-batch, insert a row into missed_ranges
  • Add a GET /health field listing the count of unrecovered missed ranges
  • Add a CLI command lumenqraph-indexer recover-gaps that replays all recorded missed ranges

Affected Files

  • crates/lumenqraph-indexer/src/poller.rs
  • crates/lumenqraph-api/src/routes/health.rs
  • migrations/ (new migration)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingindexerIndexer / ingestion pipelineobservabilityMetrics, logging, tracing, health

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions