stateward

Offline, verified reading of Cosmos SDK application.db snapshots — no running node, no chain binary, no state replay.

stateward opens a node's on-disk IAVL state, recomputes the store roots and the application hash from the raw nodes, and compares them to the store's own CommitInfo and (optionally) to a signed block header. If the recomputed app hash equals the validator-signed Header.AppHash, the snapshot is the committed state at that height — a cryptographic fact, checked locally.

Everything it emits is tied to that check — the final RESULT line and the exit code tell you whether the tree verified; on a mismatch the output is explicitly marked NOT anchored.

Why

A snapshot tarball is bytes on a disk until something proves it is the state a chain actually committed. Running a full node to check one is slow and needs the matching chain binary. stateward does it offline from the data directory alone:

• verify — recompute a store's IAVL root and check it against CommitInfo; with --expect-app-hash, also against a block header.
• census — verify the bank store and report its balance / reverse-index / value-encoding census: per-denom holder counts, the modern/legacy encoding split, and reverse-index completeness (i.e. denoms where DenomOwners under-reports holders). With --check-supply, it also re-runs x/bank's TotalSupply invariant offline (Σ balances == stored supply, per denom) — a check production chains disable — and exits non-zero on any mismatch.
• raw — emit the verified (key, value) entries of a store, the anchored raw slice that makes any downstream decoding auditable back to the input boundary. With --with-version, each row also carries the IAVL version (block height) at which that key was last written — a property of the committed tree that no node API serves.

Install

# from source (Go 1.25+)
go install github.com/ny4rl4th0t3p/stateward/cmd/stateward@latest

# or build a checkout
git clone https://github.com/ny4rl4th0t3p/stateward && cd stateward
go build ./cmd/stateward

Prebuilt binaries (linux amd64/arm64, darwin arm64) are attached to each release.

With Docker

A prebuilt image is published to GHCR on each release. It's the complete build — both backends (goleveldb + pebbledb) — so it reads any snapshot:

# verify a node home, mounted read-only
docker run --rm \
  -v /path/to/node:/data:ro \
  ghcr.io/ny4rl4th0t3p/stateward:latest verify --home /data

# census, persisting the result to a writable output dir
docker run --rm \
  -v /path/to/node:/data:ro \
  -v /path/to/results:/out \
  ghcr.io/ny4rl4th0t3p/stateward:latest census --home /data --cache-dir /out

The image runs as a non-root user, so pass --user "$(id -u):$(id -g)" when it needs to read host-owned files or write to /out. Add -t for the live progress bar.

Usage

stateward verify  --home <dir> [--store bank] [--expect-app-hash <hex|base64>]
stateward census  --home <dir> [--denom <ibc/...>] [--cache-dir <dir>] [--check-supply]
stateward raw     --home <dir> --store <name> [--prefix <hex>] [--with-version]

common flags:
  --db <dir>            path to application.db (or use --home)
  --home <dir>          node home; uses <home>/data/application.db
  --version <V>         commit version to read (default: latest)
  --workers <N>         concurrent verify workers (default: NumCPU)
  --max-open-files <N>  cap open table-file descriptors (0 = adaptive); lower on shared/network FS
  --expect-app-hash <h> compare the recomputed app hash to a header value (hex or base64)

Verify a snapshot against a signed header

stateward verify --home ~/.gaia --expect-app-hash <Header.AppHash of block V+1>

stateward prints the recomputed app hash and, if you supply one, the comparison:

recomputed app hash:  b30e758b8f37c34ed58df7dd8df0c4ef7453152d4494e906d5e09c36637d2662
  (compare against Header.AppHash of block V+1 — /cosmos/base/tendermint/v1beta1/blocks/V+1)
you supplied:         b30e758b8f37c34ed58df7dd8df0c4ef7453152d4494e906d5e09c36637d2662
  HEADER MATCH — this snapshot IS the committed state whose app hash you supplied (validator-signed, if that value came from a signed header)
RESULT: ANCHORED — the recomputed root equals the store's committed root at this version

The value you supply is whatever you pass — stateward fetches nothing. Because state committed at version V is carried in the header of block V+1 (a consensus off-by-one), fetch the header of block V+1. Cosmos LCDs serve Header.AppHash as base64; hex is also accepted.

Reproducing the cosmoshub-4 census

A full worked example, from a cold snapshot and nothing else — no running node, no chain binary. It reproduces the bank-store census stateward reports: the balance / reverse-index gap that makes DenomOwners under-report holders. Shown with the published Docker image, the most common way to run it; for a native binary, drop the sw wrapper and call stateward <cmd> --db data/application.db … directly.

1. Get a pruned cosmoshub-4 snapshot. Any public provider works. You only need the application state, so extract just data/application.db and skip blockstore.db / state.db / tx_index.db — that is most of the size and time. Providers serve only their latest snapshot, so use whatever height is current; for example, from Polkachu:

curl -sL https://snapshots.polkachu.com/snapshots/cosmos/cosmos_32538722.tar.lz4 \
  | lz4 -dc | tar -x --wildcards 'data/application.db/*'
# unsure of the archive layout? peek first: curl -sL <url> | lz4 -dc | tar -t | head
mkdir -p out   # a writable directory for the census cache

Run the tool through the image — the extracted DB mounted read-only, a writable out/ for the cache. This wrapper keeps each step to just the stateward command (-t gives the single-line progress bar; drop it if you pipe the output):

IMG=ghcr.io/ny4rl4th0t3p/stateward
sw() { docker run --rm -t --user "$(id -u):$(id -g)" \
         -v "$PWD/data/application.db":/db:ro -v "$PWD/out":/out "$IMG" "$@"; }

2. Set the snapshot height V. Polkachu encodes it in the filename (cosmos_<V>.tar.lz4); otherwise sw verify --db /db --store bank prints it (a full walk).

V=32538722

3. Seal it to the validator-signed header of block V+1 (state committed at version V is carried in the header of V+1), fetched from any LCD:

APPHASH=$(curl -s "https://cosmos-api.polkachu.com/cosmos/base/tendermint/v1beta1/blocks/$((V+1))" | jq -r '.block.header.app_hash')
sw verify --db /db --expect-app-hash "$APPHASH"   # → HEADER MATCH

4. Run the census. One walk verifies the bank store, reports the gap, lists a denom's holders, and writes a reusable cache — sealed to the same header:

$ sw census --db /db --cache-dir /out --expect-app-hash "$APPHASH" \
    --denom ibc/04AA0759B8FF54DFEE563C6046050953D77079A7D212AF59E4C80FCBF98247CA
store: bank  version: 32538722  (59122661 nodes, 29561331 leaves, 2m11s)
balance entries (0x02):                  15759976
reverse-index entries (0x03):            13796409
missing from reverse index:               1963567
legacy-encoded balances (Coin proto):     1963567
IDENTITY CHECK: (balances − index entries) == legacy-encoded balances — EXACT MATCH

and, from the same walk, that denom's full holder list — the amount-string rows are what DenomOwners returns; the coin-proto rows are the ones it silently drops (each confirmable via balances/{addr}/by_denom):

holders of ibc/04AA0759B8FF54DFEE563C6046050953D77079A7D212AF59E4C80FCBF98247CA (9):
  cosmos1zkdeaqkt48j4k3wgwqjvgz4wflx23jnj8k933c    3000        coin-proto
  cosmos1zu668p3g4d97gwycvejkutqllug4vpcvqdjtc0    4700000     coin-proto
  cosmos19p463jhn6he6a7d7pp6pqn932pw7vqneu2kdty    27488000    amount-string
  cosmos1xpl2sx64ra5san4su0r0hj8z9eh5cf5dmlp576    500000      coin-proto
  cosmos1x54ltnyg88k0ejmk8ytwrhd3ltm84xehrnlslf    12756       coin-proto
  cosmos12g3lgc6rx2getny35887sq4gdkvqd8wdvk7wan    152000      amount-string
  cosmos1d8754xqa9245pctlfcyv8eah468neqznjwmh3v    1000000     coin-proto
  cosmos1wew52s06l3k3cz4r5ttjjaxya57sv8xk5e6tlu    1000000     coin-proto
  cosmos17svmv92kwaljn7rxh86f9fwnmnvc0a3ar4mjmh    2000000     coin-proto

The final RESULT: ANCHORED line confirms the app hash recomputed from the snapshot equals the header you supplied — this data is the committed state at V. The walk also wrote out/census.json + out/fossils.csv, so any re-run on this snapshot (another --denom, a different query) returns instantly from the cache instead of walking again.

Your numbers will differ from these, and that is expected. The run above is height 32,538,722; every snapshot is a different height, and the legacy population only shrinks over time — each time an old balance is touched it re-encodes to the Int string and gains its index entry. What reproduces at every height is the invariant: missing from reverse index == legacy-encoded balances, exactly, to the entry. A slightly different missing count is the fossils being worn down, not a discrepancy.

Runtime: the full verified walk + census of the bank store (≈59M nodes, ≈15.8M balances) is ~2 minutes at 16 workers on a laptop (native SSD; a bank-only verify is the same walk). The walk streams with bounded working memory; peak footprint is dominated by goleveldb's 256 MiB block cache (set in the reader), not by the size of the store.

Exit codes

A tri-state contract, so scripts can distinguish "wrong" from "broke":

code	meaning
0	verified / anchored / pass
1	FAIL — data does not match its commitments (root or app-hash mismatch)
2	ERROR — operational failure (bad flags, unreadable DB, decode error)

Backends

The default build reads goleveldb, the Cosmos default, and pulls in no C dependencies. pebble support is behind a build tag — either grab the prebuilt stateward-pebble release archive, or build it yourself:

go build -tags pebble ./cmd/stateward

Without pebble support, opening a pebble directory returns a clear ERROR telling you to rebuild with -tags pebble (or grab the -pebble release archive). The default and -pebble binaries are otherwise identical — same name, same usage; the -pebble one just also reads pebbledb, at ~3.7× the binary size.

rocksdb is not supported. A rocksdb store is detected (by its IDENTITY file) and refused with a clear ERROR rather than misread — rocksdb needs cgo and a linked C++ library, which stateward deliberately avoids. Cosmos nodes only write rocksdb when explicitly built and configured for it, so in practice snapshots are goleveldb (or, rarely, pebble).

Importing stateward? The tag propagates to your build, not just this repo's. If your binary needs to read pebbledb snapshots, build it with the tag too: go build -tags pebble ./cmd/mybin. Without it your build still compiles (pebble and its heavy deps are simply not linked) and istore.Open returns the rebuild-with-the-tag error only if it actually meets a pebbledb store. goleveldb needs no tag.

Use as a library

stateward is a library first; the CLI is a thin shell over it. The library packages never print and never call os.Exit — they return data and typed errors, so you can embed the verified walk and extract your own datasets from it:

package main

import (
	"context"
	"fmt"

	"github.com/ny4rl4th0t3p/stateward/istore"
	"github.com/ny4rl4th0t3p/stateward/verify"
)

func main() {
	kv, backend, err := istore.Open("/path/to/application.db") // read-only
	if err != nil {
		panic(err)
	}
	defer kv.Close()
	fmt.Println("backend:", backend)

	// A leaf visitor rides the verified walk: every leaf it sees comes from a tree
	// whose hashes have been checked. Here we just count; a real consumer decodes.
	var leaves int
	visitor := verify.SingleVisitor(func(n *istore.Node) error {
		leaves++
		return nil
	})

	// version 0 = latest; Workers defaults to sequential (set Workers>1 to parallelize).
	res, err := verify.VerifyStore(kv, "bank", 0, visitor, verify.Options{Ctx: context.Background()})
	if err != nil {
		panic(err) // a *verify.MismatchError means FAIL; anything else is operational
	}
	fmt.Printf("root match=%v leaves=%d app hash=%x\n", res.RootMatch, leaves, res.AppHash)
}

Options is additive (no signature churn across versions), long operations take a context.Context, and progress/leaf handling are optional callbacks rather than hardwired logging.

How it works

The readers are hand-rolled, spec-pinned decoders of the IAVL on-disk formats (legacy hash-keyed nodes, v1 structural keys, and hybrid trees from lazy migration), with per-node format detection from the key shape. They treat every snapshot as hostile input: no panics, bounded allocations, and the decoders are continuously fuzzed. CommitInfo is the only source of truth for which stores exist and what their committed roots are — there is no per-chain configuration.

Building & testing

The Makefile wraps the common tasks (see it for the full list):

make build          # default (goleveldb) binary
make build-pebble   # with pebbledb support
make test           # unit + fuzz seed corpus + consensus-golden (race; no snapshot needed)
make check          # everything CI runs: build both variants, vet, test, lint
make lint           # golangci-lint

The end-to-end test needs a real (multi-GB) snapshot and is build-tagged, so it never runs in make test:

make integration STATEWARD_TEST_DB=$(realpath /path/to/node/data/application.db)

See verify/integration_test.go.

License

Apache-2.0. See LICENSE.