transfer-lens

A focused Solana SPL Token transfer indexer written in Rust.

Tracks balance-changing instructions for a configured set of mints — Transfer, TransferChecked, MintTo, MintToChecked, Burn, BurnChecked — and writes each one as a row to Postgres, resolving each side to the owning wallet (not just the token account).

Built on sevenlabs-hq/carbon for decoding. The live tail runs on a custom Yellowstone gRPC datasource (published yellowstone-grpc-client); a getBlock backfill/reconcile sweep fills history and guarantees completeness. All sources feed one writer that dedups on (signature, instruction_index), so overlap is harmless.

Scope: SPL Token only. Token-2022 (Token Extensions) is not yet supported (blocked on an upstream Carbon pin — see Notes). Approve/Revoke/Delegate and token name/symbol metadata are also out of scope.

Prerequisites

• Rust (stable; minimum version is pinned as rust-version in Cargo.toml).
• Postgres — any reachable instance, which you provide. The indexer needs durable storage (resume cursors, slot coverage, the token-account cache survive restarts), so this is not an ephemeral container.
• A Solana JSON-RPC endpoint (rpc.http_url) — any provider. Drives the token-account resolver and the getBlock reconcile/backfill crawls.
• A Yellowstone gRPC endpoint (rpc.grpc_url, plus grpc_x_token if your provider requires it) — only for the live tail, and usually a paid/premium product (Triton, Helius, QuickNode, Shyft, …). No gRPC endpoint? Set enable_live = false to run RPC-only (reconcile + onboarding still give the full completeness guarantee, just at higher latency).
• just (optional, recommended) — the bundled Justfile wraps every command and auto-strips the Homebrew binutils path that otherwise breaks the macOS build (see Build notes).
• Docker — only for the Postgres integration tests (just test-all); not needed to build or run the indexer.

Quickstart

git clone <repo> && cd transfer-lens

# 1. Configure: copy the templates, then fill in endpoints + the token list.
cp config/indexer.example.toml config/indexer.toml   # set rpc.http_url and [[tokens]]
cp .env.example .env                                  # set INDEXER__POSTGRES__URL, RPC/gRPC

# 2. Apply the schema, then run. `just` handles the macOS build quirk for you.
just migrate
just run

Prefer raw cargo? cargo run -p transfer-lens-migrations then cargo run -p transfer-lens-indexer --release — but on macOS read Build notes first, or you'll hit the binutils/ar wall. Once rows are landing, see Querying.

Layout

transfer-lens/
├── Cargo.toml                              # workspace
├── config/
│   └── indexer.example.toml                # copy → indexer.toml
└── crates/
    ├── transfer-lens-indexer/              # main binary
    └── transfer-lens-migrations/           # schema migrations runner

Postgres schema

The schema name is configurable (default in the example config: token_transfers). Tables created:

Table / view	Purpose
transfer_events	One row per indexed transfer. Owner-resolved from/to; amount; mint; slot.
block_time	(slot, block_time) — wall-clock time per slot, recorded independently of rows (no row/time race). Join on slot.
mint	(mint, decimals, fetched_at) — per-mint decimals, fetched once on startup. SPL decimals are immutable, so write-once. Powers ui_amount.
transfer_events_with_time	View: transfer_events LEFT JOINed to block_time and mint — exposes block_time, decimals, and ui_amount = amount / 10^decimals for a flat, human-readable read shape. Self-healing: a not-yet-recorded slot/mint reads NULL and fills in once that row lands.
indexer_cursor	(source, last_slot) — one row per datasource so restarts resume cleanly.
token_account_owner	Durable token-account → (owner, mint) cache, populated lazily. Works for ATAs and any other SPL Token account.
live_covered_slots	Slots the live stream confirmed via blocks_meta. The reconciler diffs produced slots against this and fetches only the gaps the live tail missed.
incomplete_slots	Slots with a transfer that failed to resolve (RPC error). The reconciler re-crawls these until they process cleanly, so a covered slot is genuinely complete.
token_coverage	(mint, covered_from_slot) — per-token indexed floor, used by onboarding to auto-backfill a newly added token's history.

Consumer ordering contract

Rows within a transaction are ordered by instruction_index (a packed, prefix-preserving encoding of the instruction's CPI path). Consumers should always ORDER BY slot ASC, instruction_index ASC for canonical execution order — the indexer guarantees this for any number of outer instructions and CPI depths up to 7 levels (Solana caps CPI at 4).

Mints and burns use the all-zeros pubkey (11111111111111111111111111111111) as the counterparty in from/to — handy if you want to detect them in SQL with from = '11111111111111111111111111111111' (mint) or the same on to (burn).

Querying

Examples use the default schema token_transfers and the transfer_events_with_time view (rows + slot time + decimals + ui_amount).

-- A wallet's transfers, newest first, in human units.
SELECT slot, "from", "to", ui_amount, mint
FROM token_transfers.transfer_events_with_time
WHERE "from" = $1 OR "to" = $1
ORDER BY slot DESC, instruction_index DESC;

-- Mints and burns (all-zeros counterparty convention).
SELECT * FROM token_transfers.transfer_events
WHERE "from" = '11111111111111111111111111111111';   -- mints; use "to" = … for burns

-- Daily volume per mint (human units).
SELECT date_trunc('day', to_timestamp(block_time)) AS day,
       mint,
       sum(ui_amount) AS volume
FROM token_transfers.transfer_events_with_time
WHERE block_time IS NOT NULL
GROUP BY 1, 2
ORDER BY 1 DESC, 3 DESC;

-- All transfers in one transaction, in canonical execution order.
SELECT * FROM token_transfers.transfer_events
WHERE signature = $1
ORDER BY slot ASC, instruction_index ASC;

Configuration

Copy config/indexer.example.toml to config/indexer.toml and fill in the endpoints + token list. Every field can be overridden via env vars using the prefix INDEXER__ and __ as a separator:

INDEXER__POSTGRES__SCHEMA=my_schema \
INDEXER__RPC__GRPC_X_TOKEN=secret \
cargo run -p transfer-lens-indexer

The schema name must match [A-Za-z0-9_]+.

Pipeline modes

The completeness sweep (HTTP getBlock reconcile) is always on — it is the completeness guarantee and cannot be disabled. One optional toggle in [indexer] (default-safe; see indexer.example.toml for the rest):

Flag	Default	What it does
enable_live	true	Live Yellowstone gRPC tail (the latency layer). Set false to run RPC-only with no gRPC endpoint.

Per-token onboarding is also always on: on startup a background task automatically backfills [start_slot, watermark] for any newly-added token (see Adding a new token).

The reconcile sweep self-paces via reconcile_parallel_workers (default 4): up to N adjacent slot-chunks ahead of its cursor are crawled concurrently, so it runs full parallel catch-up when far behind the tip and collapses to single-worker hole-patching at the tip (where the live tail already covers most slots). Set reconcile_parallel_workers = 1 for the old single-chunk behavior. Completeness comes from this HTTP-RPC sweep, independent of the gRPC tail.

First-run tip — RPC rate limits. A cold catch-up fans out up to reconcile_parallel_workers × backfill_concurrency concurrent getBlock calls (default 4 × 8 = 32). On a rate-limited endpoint that will earn you 429s; lower those two and/or raise backfill_throttle_ms (the per-request delay). All three knobs are documented in config/indexer.example.toml.

Running

# 1. Apply migrations.
cargo run -p transfer-lens-migrations

# 2. Start the indexer.
cargo run -p transfer-lens-indexer --release

Useful env vars (also see .env.example):

Var	Default	Purpose
INDEXER_CONFIG	./config/indexer.toml	Path to TOML config.
RUST_LOG	info	Tracing filter.

Observability

At RUST_LOG=info the indexer logs a periodic stats line so you can watch live ingest and reconcile progress. To check how far the reconcile sweep has advanced (compare against the chain's finalized tip to gauge lag):

SELECT source, last_slot FROM token_transfers.indexer_cursor;

RUST_LOG=info,transfer_lens_indexer=debug (the .env.example default) adds per-batch detail.

Backfilling a historical range (sandbox → promote)

To add an older range (e.g. you started at slot 380M and now want [360M, 370M]) without touching live data until you've checked it:

# 1. Create + migrate a temp schema.
INDEXER__POSTGRES__SCHEMA=token_transfers_backfill cargo run -p transfer-lens-migrations

# 2. Backfill the range into the temp schema (honors the range over start_slot).
INDEXER__POSTGRES__SCHEMA=token_transfers_backfill \
  cargo run -p transfer-lens-indexer --release -- backfill --start 360000000 --end 370000000

# 3. Inspect token_transfers_backfill.transfer_events, then merge + drop.
cargo run -p transfer-lens-indexer --release -- promote --from token_transfers_backfill --drop

backfill writes no resume cursor and is idempotent; promote merges on (signature, instruction_index) so it never duplicates rows you already have. You can also backfill straight into the main schema (skip steps 1 and 3) if you trust the range.

Adding a new token

The live tail subscribes to the whole SPL Token program and filters by mint, so adding a [[tokens]] entry and restarting indexes that token from then on automatically. Its history (transfers before the restart, up to the reconcile watermark W) is filled automatically: on startup, a background onboarding task detects that the new token has no token_coverage record and no existing rows at slot <= W, then crawls [start_slot, W] for it — coalesced across multiple new tokens, honoring each token's start_slot, resumable. Reconcile covers (W, tip] forward with the new mint already in config, so history is complete with no manual step and no range is crawled twice.

If you want to fill deep history below your RPC's retention (where the normal reconcile endpoint cannot reach), use the manual one-shot command:

cargo run -p transfer-lens-indexer --release -- \
  backfill --start <new_token.start_slot> --end <current_tip> --mint <NEW_MINT>

Overlap with ongoing coverage is always idempotent.

Notes

• Commitment: default is finalized — rooted, no reorgs to worry about, ~13s lag. confirmed cuts latency but can reorg, and the indexer has no reorg rollback — a reverted confirmed block's rows would persist. Only use it if you accept that tradeoff. (processed is rejected: getBlock/getBlocks need a rooted commitment.)
• Token-account resolution: each side of a transfer is resolved to its owning wallet via a four-tier lookup — Moka in-memory cache → the transaction's own pre/post_token_balances hint → the durable token_account_owner Postgres table → JSON-RPC getAccount as a last resort. Owner is read from the standard 165-byte Token Account layout, so any account — ATA or otherwise — resolves the same way; the balance hint also covers accounts closed mid-transaction with no RPC call. Cache size is indexer.token_account_cache_capacity.
• Reconcile: always on (no flag). A bounded, resumable sweep diffs produced slots (getBlocks) from the lowest configured start_slot to the finalized tip against live_covered_slots and getBlock-fetches only the slots the live stream missed — the completeness guarantee. It self-paces: up to reconcile_parallel_workers adjacent chunks are crawled concurrently, so it does fast parallel catch-up when behind and collapses to single-worker hole-patching at the tip (where the live tail covers most slots and reconcile does almost no getBlock). Its first contiguous pass also serves as the historical fill. For deep history below your RPC's retention, run the one-shot backfill command (below) with rpc.http_url pointed at an archival endpoint — standard providers prune old blocks, so getBlock of an old slot fails on them.
• Atomic coverage: a slot is trusted only when it's covered and not flagged incomplete. A transfer that fails to resolve (RPC error, vs a clean "not a token account" skip) flags its slot in incomplete_slots; the reconciler re-crawls flagged slots every pass until they process cleanly. The getBlock crawl is owned in-process (not carbon's RpcBlockCrawler) and applies backpressure rather than dropping transactions when the pipeline is busy. So a "covered" slot is genuinely complete.
• Block time: recorded per slot in the block_time table — live from blocks_meta, crawls from each getBlock. It's keyed by slot and written independently of transfer rows, so there's no row/time ordering race. Join transfer_events.slot = block_time.slot, or read the transfer_events_with_time view (a LEFT JOIN; a not-yet-recorded slot reads NULL and self-heals).
• Idempotency: the writer upserts on (signature, instruction_index), so overlap among backfill, live, and reconcile is harmless. Restarts are safe.
• Out of scope: Approve/Revoke/Delegate, Token-2022 (re-enabled once carbon-token-2022-decoder ships with a compatible spl-token-2022 pin — upstream 0.12 pulls 10.0.0 which does not compile against current Solana releases), token name/symbol metadata.

Build notes

On macOS, the yellowstone-grpc-proto build script compiles protobuf from source via autotools. If you have Homebrew binutils on your PATH it shadows Apple's ar and the link step fails. Two workarounds:

# Option 1 (recommended): temporarily unlink binutils for the build.
brew unlink binutils && cargo build --release && brew link binutils

# Option 2: build with a PATH that excludes /opt/homebrew/opt/binutils/bin.
# (NOTE: use `env` not `env -i` — `env -i` will also wipe any env vars you
#  rely on, e.g. credentials read from your shell rc.)
PATH="$HOME/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/bin" \
  cargo build --release

Testing

Two layers, both run by CI on every push:

• Unit tests — pure logic, no Docker or network: just test (cargo test --lib --all). Covers the instruction_index CPI-path packing, reconcile wave planning, verify range math, config validation, and mint-decimal parsing.
• Integration tests — the full write path against an ephemeral Postgres (testcontainers; needs Docker): just test-all (cargo test --test postgres). Applies the migrations and exercises the batched UPSERT, (signature, instruction_index) dedup, cursor/coverage bookkeeping, and the transfer_events_with_time view.

Correctness of the I/O paths leans on design as much as tests: the reconcile sweep plus incomplete_slots give the completeness guarantee, and idempotent writes make overlap and replay safe. There's intentionally no coverage-percentage badge — most lines are I/O exercised by the integration layer, so a unit-line figure would understate it; run cargo llvm-cov locally if you want a number.

Contributing

See CONTRIBUTING.md. In short: run just lint && just test before opening a PR, and read Build notes if you're on macOS.

License

MIT.