Skip to content

Indexer crash-loops on "relation \"accounts_to_delete\" already exists" when a gap triggers a second snapshot run #18

Description

@artemrootman

Version: v0.1.2

Summary: When self-healing triggers a second snapshot-processing run within the same process, the indexer panics with relation "accounts_to_delete" already exists and exits, entering an unbreakable crash loop.

What happens:

  1. The first snapshot run calls create_accounts_to_delete_table (crates/snapshot/src/db_queries.rs), which executes db/create_accounts_to_delete_table.sql — a plain CREATE UNLOGGED TABLE accounts_to_delete AS SELECT ... (no IF NOT EXISTS, no preceding DROP). The table is never dropped within the process.
  2. A live slot gap (e.g. a single missed slot) triggers self-healing, which starts a second snapshot-processing run.
  3. That run calls create_accounts_to_delete_table again → relation "accounts_to_delete" already exists → panic at crates/index/src/modules/snapshot.rs:76 → process exits with code 1 → restart → full re-bootstrap → next gap → same panic.

The deploy-time migration m20260325_000000_drop_temp_tables drops accounts_to_delete (along with temp_snapshot_account_versions and temp_closed_accounts), but only during migrations — not between snapshot runs within the same process. temp_closed_accounts is also cleaned up at runtime via its own drop_* SQL, while accounts_to_delete and temp_snapshot_account_versions are not.

Log:

ERROR cloudbreak_index::modules::snapshot: Failed to process snapshot: Execution Error: error returned from database: relation "accounts_to_delete" already exists
thread 'tokio-runtime-worker' (13) panicked at crates/index/src/modules/snapshot.rs:76:13:
Failed to process snapshot: ... relation "accounts_to_delete" already exists
index-1 exited with code 1 (restarting)

Suggested fix: DROP TABLE IF EXISTS accounts_to_delete before creating it (or drop it at the end of each snapshot run), mirroring the existing cleanup for temp_closed_accounts.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions