fix: remove deleted dag/captive-core imports and wire persistent cursor into indexer worker - #386
Merged
Osuochasam merged 3 commits intoJul 27, 2026
Conversation
Closes Open-audit-foundation#283 - Remove import of reconstructDagFromMetaXdr from ../dag/engine - Remove import of ExecutionDag type from ../dag/types - Remove onDag field from StreamingIndexerOptions interface - Remove DAG reconstruction block from the Horizon streaming callback - Remove CaptiveCoreSupervisorOptions / CaptiveCoreControls / startCaptiveCoreIndexer references from ResilientStreamingOptions and startResilientEventIngestion since lib/stellar/captive-core.ts was deleted in the same cleanup commit as lib/dag/ - Add missing IngestionStateStore import from ./ingestion-state - Re-export createMemoryIngestionStateStore and createFileIngestionStateStore from indexer.ts so existing consumers (tests, server.ts) do not need import-path changes - Add stateStore and coldStartLookbackLedgers to StreamingIndexerOptions to satisfy existing call-sites in startHorizonStreamingIndexer and startResilientEventIngestion - Import eventResponseToRawEvent from ./events where it is defined - Fix duplicate paginationCursor property in startEventIndexer poll loop - Remove captiveCore option from server.ts call to startResilientEventIngestion - Update indexer.test.ts: drop PassThrough/EventEmitter imports and update the startResilientEventIngestion test to reflect the removed captive-core path - All 7 indexer unit tests pass after these changes
Closes Open-audit-foundation#280 Problem: src/worker/indexer.ts started re-indexing from ledger 0 (or a hard-coded value) on every restart because it never read or wrote the IndexerCursor table, even though lib/stellar/indexer-persistent.ts and lib/db/utils.ts already provided getCursor() / updateCursor(). Changes: - Import getCursor and updateCursor from lib/db/utils into the worker. - On startup, call getCursor() to read the last persisted ledger. Fall back to the START_LEDGER environment variable (default 0) when no row exists (i.e. first run). Log which path was taken so operators can see behaviour. - Streaming mode (INDEXER_MODE=stream): call updateCursor(rawEvent.ledger) after every successfully handled event so the cursor advances continuously. - Polling mode (INDEXER_MODE=poll): call updateCursor(cursor.lastLedger) after every successfully processed batch. - In both modes, use an in-flight Promise sentinel so stop() can await the currently executing batch before writing the final cursor. - Graceful SIGTERM / SIGINT: stop() now (1) signals the underlying indexer, (2) awaits the in-flight batch, (3) writes the final cursor, (4) tears down Redis. Restores are therefore cheap: next start resumes from the ledger after the last completed batch. - Polling mode: convert SorobanRpc.Api.EventResponse to RawEvent via eventResponseToRawEvent before passing to the shared handleEvent(). - Remove the unused fetchContractEventsResilient import. - Document START_LEDGER in .env.example with usage notes. - Add src/worker/__tests__/indexer.test.ts with 11 unit tests (mocked Prisma) covering all three required scenarios: * startup read (returns 0 when no row, returns stored ledger otherwise) * mid-run write (upsert called after each batch, not only at shutdown) * graceful shutdown (in-flight awaited before final cursor write, no write when ledger is 0, restarts resume from persisted ledger)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #283
Closes #280
Issue #283 — Remove deleted
dagandcaptive-coremodule referencesProblem
lib/stellar/indexer.tsimportedreconstructDagFromMetaXdrfrom../dag/engineand theExecutionDagtype from../dag/types. Both fileswere deleted in a prior cleanup commit, causing a build failure. The same
commit also deleted
lib/stellar/captive-core.ts, leavingCaptiveCoreSupervisorOptions,CaptiveCoreControls, andstartCaptiveCoreIndexerreferenced instartResilientEventIngestionwithno backing module.
Changes
dagimport statementsonDagfield fromStreamingIndexerOptionscaptiveCoreoption fromResilientStreamingOptionsand thestartCaptiveCoreIndexercall fromstartResilientEventIngestionIngestionStateStoreimport from./ingestion-stateandre-export
createMemoryIngestionStateStore/createFileIngestionStateStoreso existing consumers (
server.ts, tests) require no import-path changesstateStoreandcoldStartLookbackLedgerstoStreamingIndexerOptionsto satisfy the call-sites that were already using them
paginationCursorproperty in the poll loopcaptiveCoreoption fromserver.tslib/stellar/__tests__/indexer.test.tsto drop the captive-coretest path (which depended on the deleted module)
Result
File compiles cleanly. All 7 existing indexer unit tests pass.
Issue #280 — Wire persistent cursor into the standalone indexer worker
Problem
src/worker/indexer.tsnever calledgetCursor()on startup orupdateCursor()after processing batches, so every restart re-indexed fromledger 0 — wasting RPC quota and delaying the dashboard while backfilling.
lib/stellar/indexer-persistent.tsandlib/db/utils.tsalready providedthe full cursor infrastructure; it just wasn't wired up.
Changes
getCursor()on boot; resume from the stored ledger.Fall back to the
START_LEDGERenvironment variable (default0) on firstrun when no cursor row exists yet.
updateCursor(rawEvent.ledger)after every successfully handled event so the cursor advances continuously.
updateCursor(cursor.lastLedger)after every successfully processed batch, not only at shutdown.
stop()now (1) signals the underlyingindexer to stop producing, (2) awaits the in-flight batch via a Promise
sentinel so no partially-processed ledger is ever lost, (3) writes the
final cursor to the database, then (4) tears down Redis and exits.
SorobanRpc.Api.EventResponse→RawEventviaeventResponseToRawEventin polling mode before publishing.START_LEDGERin.env.examplewith usage notes.Tests added (
src/worker/__tests__/indexer.test.ts)11 unit tests with a fully mocked Prisma client covering all three scenarios
required by the issue:
stop()resolves; skips write when ledger is 0; in-flight batch awaited before final write; restarts resume from persisted ledgerResult
18/18 tests pass (11 new + 7 pre-existing).
What was tested
lib/stellar/__tests__/indexer.test.ts— 7/7 ✅src/worker/__tests__/indexer.test.ts— 11/11 ✅tsc --noEmit --project tsconfig.server.json— noerrors in any file touched by this PR