Skip to content

Add integration test for resetIndexer/replayFromLedger racing concurr… - #1411

Open
Fury03 wants to merge 1 commit into
LabsCrypt:mainfrom
Fury03:test/issue-1293-reset-replay-race
Open

Add integration test for resetIndexer/replayFromLedger racing concurr…#1411
Fury03 wants to merge 1 commit into
LabsCrypt:mainfrom
Fury03:test/issue-1293-reset-replay-race

Conversation

@Fury03

@Fury03 Fury03 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Closes #1293

Problem Statement (The Bug)

The resetIndexer and replayFromLedger functions are tested in isolation but never interleaved with SorobanEventWorker's own mutex-protected poll cycle. This means the race described in Functional Edge Case #19 is untested: when an operator triggers incident-recovery tooling while a poll is mid-flight, the poll's stale cursor can silently overwrite the reset/replay cursor, rendering the recovery action ineffective.

This issue cannot be fixed with a local patch because it requires concurrent execution of two independently-tested code paths — something isolated unit tests structurally cannot cover.

Solution Comparison and Decision

Option Description Why rejected
A. Unit test with mocked concurrency Mock both paths and assert ordering Does not exercise real async interleaving; the race depends on actual promise resolution ordering
B. Mutex extension only (no test) Extend the worker mutex to cover reset/replay Addresses the symptom without proving it works; regressions go undetected
C. Integration test with deferred promise Use a deferred getEvents mock to inject reset/replay mid-flight, assert reset cursor wins Chosen — directly reproduces the real race, fails against current code, passes once fixed

The Change (Code modifications)

Added backend/tests/integration/reset-replay-race.test.ts — a single new test file containing two integration tests.

Core test mechanism:

// Deferred promise controls when the poll's getEvents call resolves
let resolveGetEvents!: (value: any) => void;
const getEventsDeferred = new Promise((resolve) => { resolveGetEvents = resolve; });

// Shared mutable state simulates Postgres — both worker and reset/replay write here
let dbIndexerState = { lastLedger: 199, lastTimestamp: "2024-01-01T00:00:00Z" };

// Upgraded mock: first call blocks, second call (after reset) reads from shared state
server.getEvents
  .mockImplementationOnce(() => getEventsDeferred)
  .mockImplementationOnce(() => ({ events: [mockEvent], latestLedger: 210 }));

// Inject reset mid-flight while poll is awaiting RPC
resetIndexer(50, "incident recovery");

// Release the poll — it now writes its stale cursor, overwriting 50
resolveGetEvents({ events: [mockEvent], latestLedger: 210 });

Test 1 — resetIndexer race: Starts a poll, injects resetIndexer(50) mid-flight via deferred promise, asserts reset cursor wins. Current code fails: poll overwrites 50 with stale 200.

Test 2 — replayFromLedger race: Starts a second poll mid-flight, injects replayFromLedger(100), asserts replay cursor wins. Current code fails: second poll overwrites 100 with stale 300.

Compatibility Note

No INTERFACE_VERSION change. This PR adds only a test file; no production code is modified.

Testing

Test Current code Expected after fix
resetIndexer cursor survives concurrent poll FAILED (got 200, expected 50) PASSED
replayFromLedger cursor survives concurrent poll FAILED (got 300, expected 100) PASSED

Pre-existing failures in indexer-state.test.ts, soroban-event-worker.test.ts, and eventRace.test.ts are unrelated.

Additional Notes

Scope: Only backend/tests/integration/reset-replay-race.test.ts is added. No production code is modified.

…ent poll

Adds an integration test that exercises the race condition described in
Functional Edge Case #19 (issue LabsCrypt#1293): resetIndexer and replayFromLedger
bypass the SorobanEventWorker's batchMutex, so their DB cursor writes can
be overwritten by a concurrent poll's stale upsert.

The test deliberately fails against current code and will pass once the
race is fixed (the poll must re-read the cursor before writing it back,
or resetIndexer/replayFromLedger must go through the batchMutex).

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Audit] No test covers resetIndexer/replayFromLedger racing a concurrent scheduled poll

1 participant