Skip to content

Owner routed deferred cleanup - #49

Open
ArthurPaivaT wants to merge 3 commits into
mainfrom
owner-routed-deferred-cleanup
Open

ArthurPaivaT wants to merge 3 commits into
mainfrom
owner-routed-deferred-cleanup

Conversation

@ArthurPaivaT

Copy link
Copy Markdown
Contributor
  • Takes the finalize-slot account cleanup off the finalize worker and routes every delete by owner and pubkey together, so on the owner-partitioned tables each delete prunes to a single partition instead of probing all of them. One new key, cleanup-interval-slots, defaulting to 1.
  • Each account carries its own exclusive cutoff, merged by max, so a deferred delete prunes each key against its own newest write rather than against whatever slot the drain happens to run in. Finalizing a slot now costs ~<1ms on the worker.
  • Fails in the safe direction. The snapshot_accounts delete is awaited before the accounts delete is issued, so a failure leaves the closed-account tombstone in place and the account still reads as closed. A failed drain returns its keys to the queue and is attempted again on the next round, never inside the same one.

Indexer (crates/index, crates/core)

  • New module at crates/index/src/modules/cleanup, split into the handle, the pending map, the drainer and the statements.
  • The finalize worker writes the finalized marker, turns the slot's accounts into keys, hands them to the queue and returns. Its cleanup body drops from 170 lines to 43.
  • An open account takes the finalized slot as its cutoff. A closed or owner-moved account takes the slot after it, so the tombstone written at that slot goes with the rows it was hiding. One exclusive cutoff expresses both shapes, so the four cleanup query files collapse to two.
  • A drain takes the whole queue at once and runs one statement per table. There is no batching and no concurrency to size.
  • The owner is resolved from the in-memory owner map rather than the incoming block, because a close arrives from geyser carrying the system program as its owner. The map already holds the prior owner plus that slot's changed owners, so this needs no extra database read.
  • Accounts closed while the startup snapshot is still loading are cached and cleared by the existing one-shot once the load finishes. The owner map is still seeding during that window, so a close it has not learned yet can still have rows.

Query tracker (crates/query-tracker)

  • Index DDL now defers on either of two indexer gauges rather than one. The finalize queue stops reflecting database pressure once cleanup runs off that worker, so the cleanup backlog is the second signal.
  • An indexer that does not publish the new gauge reads as no pressure, so the change is inert until the indexer side deploys. An endpoint that cannot be read at all still defers, as before.

Config

  • One new indexer key, cleanup-interval-slots, default 1. There is no enable flag, because cleanup is not optional.
  • One new query tracker key, indexer-cleanup-lag-threshold, default 32.
  • Values above one coalesce repeat touches of an account into a single delete. The deletes saved and the extra account versions every latest-version read then scans past are the same quantity, so the default is the setting that pays nothing. A threshold set near or below the interval would defer index DDL permanently, so the two move together.

Memory

  • The queue holds one slot of keys at the default interval, about 0.4 MB estimated. It is bounded by coalescing rather than by arrival rate.
  • No change to the account owner map. The 37.7 GiB resident on the soak node is that map, which this change neither adds nor requires.

Database

  • No schema change and no migration. Two runtime query files, one routed and one not, replacing four.

Docs

  • README gains the new config key on the indexer table, the new threshold on the query tracker table, and the cleanup lag gauge on the indexer metrics table. The finalize-slot histogram origin list is corrected, since the cleanup statements now run off that worker.
  • Both example configs gain their new key.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants