Skip to content

fix: remove deleted dag/captive-core imports and wire persistent cursor into indexer worker - #386

Merged
Osuochasam merged 3 commits into
Open-audit-foundation:mainfrom
Abd-Standard:issue-283-280
Jul 27, 2026
Merged

fix: remove deleted dag/captive-core imports and wire persistent cursor into indexer worker#386
Osuochasam merged 3 commits into
Open-audit-foundation:mainfrom
Abd-Standard:issue-283-280

Conversation

@Abd-Standard

Copy link
Copy Markdown
Contributor

Closes #283
Closes #280


Issue #283 — Remove deleted dag and captive-core module references

Problem

lib/stellar/indexer.ts imported reconstructDagFromMetaXdr from
../dag/engine and the ExecutionDag type from ../dag/types. Both files
were deleted in a prior cleanup commit, causing a build failure. The same
commit also deleted lib/stellar/captive-core.ts, leaving
CaptiveCoreSupervisorOptions, CaptiveCoreControls, and
startCaptiveCoreIndexer referenced in startResilientEventIngestion with
no backing module.

Changes

  • Remove the two dead dag import statements
  • Remove the onDag field from StreamingIndexerOptions
  • Remove the DAG reconstruction block from the Horizon streaming callback
  • Remove the captiveCore option from ResilientStreamingOptions and the
    startCaptiveCoreIndexer call from startResilientEventIngestion
  • Add the missing IngestionStateStore import from ./ingestion-state and
    re-export createMemoryIngestionStateStore / createFileIngestionStateStore
    so existing consumers (server.ts, tests) require no import-path changes
  • Add stateStore and coldStartLookbackLedgers to StreamingIndexerOptions
    to satisfy the call-sites that were already using them
  • Fix duplicate paginationCursor property in the poll loop
  • Remove the now-invalid captiveCore option from server.ts
  • Update lib/stellar/__tests__/indexer.test.ts to drop the captive-core
    test path (which depended on the deleted module)

Result

File compiles cleanly. All 7 existing indexer unit tests pass.


Issue #280 — Wire persistent cursor into the standalone indexer worker

Problem

src/worker/indexer.ts never called getCursor() on startup or
updateCursor() after processing batches, so every restart re-indexed from
ledger 0 — wasting RPC quota and delaying the dashboard while backfilling.
lib/stellar/indexer-persistent.ts and lib/db/utils.ts already provided
the full cursor infrastructure; it just wasn't wired up.

Changes

  • Startup read: call getCursor() on boot; resume from the stored ledger.
    Fall back to the START_LEDGER environment variable (default 0) on first
    run when no cursor row exists yet.
  • Mid-run write (streaming mode): call updateCursor(rawEvent.ledger)
    after every successfully handled event so the cursor advances continuously.
  • Mid-run write (polling mode): call updateCursor(cursor.lastLedger)
    after every successfully processed batch, not only at shutdown.
  • Graceful SIGTERM / SIGINT: stop() now (1) signals the underlying
    indexer to stop producing, (2) awaits the in-flight batch via a Promise
    sentinel so no partially-processed ledger is ever lost, (3) writes the
    final cursor to the database, then (4) tears down Redis and exits.
  • Convert SorobanRpc.Api.EventResponseRawEvent via
    eventResponseToRawEvent in polling mode before publishing.
  • Document START_LEDGER in .env.example with usage notes.

Tests added (src/worker/__tests__/indexer.test.ts)

11 unit tests with a fully mocked Prisma client covering all three scenarios
required by the issue:

Suite Tests
Startup cursor read returns 0 on first run; returns stored ledger on resume; queries correct table/id
Mid-run cursor write upserts after each batch independently; creates row on first write; updates on subsequent writes
Graceful shutdown persists cursor before stop() resolves; skips write when ledger is 0; in-flight batch awaited before final write; restarts resume from persisted ledger

Result

18/18 tests pass (11 new + 7 pre-existing).


What was tested

  • lib/stellar/__tests__/indexer.test.ts — 7/7 ✅
  • src/worker/__tests__/indexer.test.ts — 11/11 ✅
  • TypeScript compilation: tsc --noEmit --project tsconfig.server.json — no
    errors in any file touched by this PR

Note: 11 pre-existing test failures exist in lib/translator,
lib/resilience, and lib/stellar/__tests__/ingestion-pool.test.ts.
These were already failing on main before this branch and are unrelated
to the changes here.

Closes Open-audit-foundation#283

- Remove import of reconstructDagFromMetaXdr from ../dag/engine
- Remove import of ExecutionDag type from ../dag/types
- Remove onDag field from StreamingIndexerOptions interface
- Remove DAG reconstruction block from the Horizon streaming callback
- Remove CaptiveCoreSupervisorOptions / CaptiveCoreControls / startCaptiveCoreIndexer
  references from ResilientStreamingOptions and startResilientEventIngestion since
  lib/stellar/captive-core.ts was deleted in the same cleanup commit as lib/dag/
- Add missing IngestionStateStore import from ./ingestion-state
- Re-export createMemoryIngestionStateStore and createFileIngestionStateStore from
  indexer.ts so existing consumers (tests, server.ts) do not need import-path changes
- Add stateStore and coldStartLookbackLedgers to StreamingIndexerOptions to satisfy
  existing call-sites in startHorizonStreamingIndexer and startResilientEventIngestion
- Import eventResponseToRawEvent from ./events where it is defined
- Fix duplicate paginationCursor property in startEventIndexer poll loop
- Remove captiveCore option from server.ts call to startResilientEventIngestion
- Update indexer.test.ts: drop PassThrough/EventEmitter imports and update the
  startResilientEventIngestion test to reflect the removed captive-core path
- All 7 indexer unit tests pass after these changes
Closes Open-audit-foundation#280

Problem: src/worker/indexer.ts started re-indexing from ledger 0 (or a
hard-coded value) on every restart because it never read or wrote the
IndexerCursor table, even though lib/stellar/indexer-persistent.ts and
lib/db/utils.ts already provided getCursor() / updateCursor().

Changes:
- Import getCursor and updateCursor from lib/db/utils into the worker.
- On startup, call getCursor() to read the last persisted ledger. Fall back
  to the START_LEDGER environment variable (default 0) when no row exists
  (i.e. first run). Log which path was taken so operators can see behaviour.
- Streaming mode (INDEXER_MODE=stream): call updateCursor(rawEvent.ledger)
  after every successfully handled event so the cursor advances continuously.
- Polling mode (INDEXER_MODE=poll): call updateCursor(cursor.lastLedger)
  after every successfully processed batch.
- In both modes, use an in-flight Promise sentinel so stop() can await the
  currently executing batch before writing the final cursor.
- Graceful SIGTERM / SIGINT: stop() now (1) signals the underlying indexer,
  (2) awaits the in-flight batch, (3) writes the final cursor, (4) tears
  down Redis. Restores are therefore cheap: next start resumes from the
  ledger after the last completed batch.
- Polling mode: convert SorobanRpc.Api.EventResponse to RawEvent via
  eventResponseToRawEvent before passing to the shared handleEvent().
- Remove the unused fetchContractEventsResilient import.
- Document START_LEDGER in .env.example with usage notes.
- Add src/worker/__tests__/indexer.test.ts with 11 unit tests (mocked
  Prisma) covering all three required scenarios:
    * startup read (returns 0 when no row, returns stored ledger otherwise)
    * mid-run write (upsert called after each batch, not only at shutdown)
    * graceful shutdown (in-flight awaited before final cursor write,
      no write when ledger is 0, restarts resume from persisted ledger)
@Osuochasam
Osuochasam merged commit 4bd4697 into Open-audit-foundation:main Jul 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants