Skip to content

Implement processed commitment module - #50

Draft
ArthurPaivaT wants to merge 2 commits into
mainfrom
feat/processed-accounts
Draft

ArthurPaivaT wants to merge 2 commits into
mainfrom
feat/processed-accounts

Conversation

@ArthurPaivaT

@ArthurPaivaT ArthurPaivaT commented Sep 14, 2026

Copy link
Copy Markdown
Contributor
  • Adds processed commitment for getAccountInfo, getMultipleAccounts, getBalance, getTokenAccountBalance, getTokenSupply and getSlot. The
    API holds the newest Yellowstone blocks in memory, around the slot Postgres has confirmed.
  • A request answers every key written in those blocks from memory and reads the rest from Postgres at the confirmed slot the blocks link
    to. The context slot and the block time come from the newest block.
  • Is fail-closed. When the blocks cannot be linked by blockhash down to the Postgres confirmed slot, a processed request answers exactly as
    a confirmed request would. Forks are served as the feed reports them. Dead or restarted slots, duplicate blocks and feed gaps fall back
    until Postgres confirmed passes them.
  • Every other method keeps following processed-commitment. A node without the new section behaves as main.

Core (crates/core)

  • Processed is its own module tree, split by role: mod.rs for the public surface and the constants, ingest.rs to build one block per slot,
    store.rs for the block store, read.rs for ProcessedBlocks and get_account, and subscribe.rs for the feed thread.
  • The feed subscribes to blocks with accounts and to slot statuses at processed commitment, on its own thread. A single writer, FeedWriter,
    owns the store and publishes the latest chained blocks as an immutable ProcessedBlocks after every event. A request clones one Arc and
    holds no lock while it reads, and ProcessedBlocks get_account answers a key as Live, Closed or Unknown.
  • The head is the highest stored slot. Its parent walk is checked by blockhash down to the Postgres confirmed slot, whose blockhash comes
    from recent_blockhashes. The walk fails when a link is missing, a hash differs, the head has no block time, or a slot on the way is dead,
    restarted, has no bank creation in this session, or was seen before the last reconnect.
  • A second blockhash for a slot, or a parent blockhash that does not match the stored parent, deletes that block and its descendants.
    Processed then falls back until Postgres confirmed passes that slot.
  • The store keeps the confirmed slot, 4 slots below it and at most 32 slots above it. An account written by an owner outside the program
    filter is treated as closed.
  • The gRPC connect, subscribe and reconnect loop moves from the indexer into core as a shared client. The indexer keeps its from_slot
    replay and its give-up panic. The API feed never replays, retries forever and never changes node health.

API (crates/api)

  • getAccountInfo, getMultipleAccounts, getTokenAccountBalance and getTokenSupply read their keys through one shared read at every
    commitment, with their own SQL. Each method keeps its own owner filter, decode, encode and errors. Without processed blocks the read is the
    same Postgres query as before.
  • read decides how each request reads. With processed blocks, a live key answers from memory, a closed key is absent and never reads
    Postgres, and an unknown key reads Postgres with the same SQL at the confirmed slot the blocks link to. jsonParsed token accounts resolve
    their mint through the blocks first.
  • getBalance keeps getBalance.sql, which now takes a slot bound. A key found in the blocks answers from memory.
  • getSlot answers the newest block slot from the blocks, and otherwise runs its Postgres slots read unchanged. The slot mismatch log no
    longer underflows when the slot cache is ahead of Postgres.
  • getTokenAccountBalance reads through getAccountInfoWithMintData.sql and slices the amount and the mint in Rust, with the same bytes and
    errors as before. getTokenAccountBalance.sql is removed.
  • Processed blocks serve only while the node is healthy, their head is not below the cached confirmed slot, and the cached finalized slot
    is not above their anchor.
  • The slot syncronizer publishes the confirmed slot and its blockhash to the feed.
  • Tracing: the four shared-read methods now use one SQL debug target, account_sql, and one database span, account_db, with a method field.
    They replace gai_sql, gma_sql, get_token_account_balance_sql, get_token_supply_sql and the matching per-method span names. The in-memory
    lookup runs in a processed_read span with the store size in stored_blocks and stored_bytes.

Config

  • A new processed-accounts section on the API with enabled, endpoint and x-token. It requires the slot-syncronizer section, and startup
    fails without it.
  • For the six methods, a processed request that falls back gets confirmed data even with processed-commitment set to reject.
  • No indexer, owner map or schema change. Each API instance carries its own full block feed.

Metrics

  • cloudbreak_api_processed_requests_total by method, route (view or degraded) and reason (none, no_blocks, unhealthy, head_behind,
    finalized_above_anchor).
  • cloudbreak_api_processed_confirm_latency_ms, from block receipt to the Postgres confirmed anchor covering it.
  • Both register only when the section is enabled.
  • cloudbreak_api_request_duration_ms records fractional milliseconds for the five account methods, with buckets down to 0.05 ms.

Integration tests

  • The new compare-processed-accounts command checks processed getMultipleAccounts against reference Agave nodes at equal context slots,
    with per-source latency. It checks that a processed answer at slot S matches confirmed at exactly S, and compares getBalance,
    getTokenAccountBalance and getTokenSupply against a reference. It reports processed getSlot against confirmed getSlot and the references.
  • A check fails when it compares nothing, and forks are classified with getBlocks on a reference.

@ArthurPaivaT
ArthurPaivaT force-pushed the feat/processed-accounts branch from ebbe21a to b30d7ae Compare September 16, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant