Skip to content

fix(actions): make validator sync atomic to stop dropped validators#48

Merged
lucca65 merged 1 commit into
masterfrom
fix/validator-sync-race
Jun 29, 2026
Merged

fix(actions): make validator sync atomic to stop dropped validators#48
lucca65 merged 1 commit into
masterfrom
fix/validator-sync-race

Conversation

@lucca65

@lucca65 lucca65 commented Jun 29, 2026

Copy link
Copy Markdown
Member

Problem

upsertAction replaced an action's validators by deleting on db (a separate connection, unawaited) and inserting on tx (also unawaited). The transaction committed before the inserts completed, and the out-of-transaction delete could race and wipe the just-inserted rows. Result: actions left with zero validators. A claimable action with no validators is invisible to validators in the app, so its claims sit pending forever.

A manual reindex on 2026-06-13/14 replayed cambiatus.cm::upsertaction as updates (action_id > 0) and tripped this race at scale.

Impact (prod, read-only audit 2026-06-29)

  • 262 claimable actions reindexed 06-13/14 with 0 validators
  • 187 stuck pending claims, ~80 users, across 8 communities (AGL + MUDA worst hit)
  • Surfaced via a user report (validators couldn't see the claims to approve them)

Fix

Run the delete + inserts inside tx, awaited via Promise.all, with the whole chain returned so the transaction commits only after they land. Drop the per-insert swallowing .catch so any failure rolls the action back instead of persisting a half-written action. Re-running the indexer over the affected blocks now repopulates validators reliably (also makes future reindexes safe).

Follow-up (separate, not in this PR)

Backfill the already-broken 262 actions: re-process the affected upsertaction blocks with this fix deployed, then verify 0 actions with missing validators and the 187 claims become visible.

Testing

  • npx standard clean.
  • No automated suite in this repo. Behavior validated by static analysis of the tx/promise ordering; recommend a local indexer replay of an upsertaction update against a dev DB before deploy.

🤖 Generated with Claude Code

upsertAction replaced an action's validators by deleting on `db` (a
separate connection, unawaited) and inserting on `tx` (also unawaited).
The transaction committed before the inserts landed, and the out-of-tx
delete could race and wipe them — leaving actions with zero validators.
Such claimable actions become invisible to validators in the app, so
their claims sit pending forever.

A reindex on 2026-06-13/14 replayed upsertaction as updates and hit this
race, stranding 187 pending claims across 8 communities (~80 users).

Run the delete + inserts inside `tx`, awaited via Promise.all, and let
any failure roll the action back instead of swallowing it. Re-running the
indexer over the affected blocks now repopulates validators reliably.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@lucca65
lucca65 merged commit 7816716 into master Jun 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant