flashblocks-recorder

Records Base Flashblocks (200ms sub-block preconfirmations) to a compact binary log, so we can post-hoc match our own tx hashes and competitors' tx hashes against exact flashblock index + wall-clock receive time. Flashblock data is a live streaming artifact — it is not part of finalized chain state and isn't retained anywhere after the fact, so this has to run continuously to be useful later.

Standalone project, independent of the main de-gate-evm poetry env.

Setup

poetry install
cp .env.example .env
poetry run flashblocks-recorder

Run

.venv/bin/flashblocks-recorder

Connects to FLASHBLOCKS_WS_URL (defaults to Base's public infra stream, no API key needed), reconnects with backoff on drop, and appends one fixed binary record per flashblock message to data/flashblocks-YYYYMMDD.bin (UTC-dated, rotates at midnight).

Running several recorders at once

To compare endpoints, run one instance per source. Set SUBSCRIBE=true for standard RPC endpoints, which need eth_subscribe(["newFlashblocks"]); leave it off for Base's infra stream, which pushes unprompted.

Give each instance a distinct SOURCE so their files don't collide — output becomes flashblocks-<SOURCE>-YYYYMMDD.bin:

SOURCE=infra FLASHBLOCKS_WS_URL=wss://mainnet.flashblocks.base.org/ws .venv/bin/flashblocks-recorder
SOURCE=rpc SUBSCRIBE=true FLASHBLOCKS_WS_URL=wss://<provider-endpoint> .venv/bin/flashblocks-recorder

A recorder takes an exclusive flock on its file and exits if another instance already holds it, so a forgotten SOURCE fails at startup instead of silently interleaving records. That check is reliable for processes on one host, but not between containers on Docker Desktop for Mac, where two opens of the same bind-mounted file get separate lock identities. make run therefore also names the container recorder-$SOURCE, so a duplicate SOURCE is refused by Docker itself.

Providers answer newFlashblocks in one of two shapes, both handled: the flashblock object (payload_id/index/diff, as Base documents), or a full block object whose transaction list grows with each frame. The second is converted to per-flashblock diffs on the way in — index recovered by counting frames per block, transactions de-duplicated against earlier frames — so records from either shape are directly comparable.

flashblocks-lookup --source rpc scans one source; without it, all are scanned.

Tests

pip install -e ".[dev]"
pytest

Unit tests cover parsing (both wire shapes), the writer's labelling and locking, and the comparison join. tests/test_e2e.py runs the recorder loop against a real local websocket server with canned frames — no network — and asserts the decoded output, including subscribe, reconnect, and junk-frame handling.

CI runs these before building the image, so a broken build costs nothing.

Compare sources

python -m analysis.compare_latency latency-compare-usa latency-compare-tokyo

Takes recording directories or individual .bin files, groups them by source label, and compares every pair on tx-hash first-seen (transport independent) and on (block_number, flashblock_index). Reports the median difference, who saw it first, and how often each source fell a whole flashblock behind — the tail matters more than the median, since sources tend to be near-identical until one drops a slot.

Sources recorded on different hosts include those hosts' clock offsets; the tool cannot separate NTP skew from propagation, so check chronyc tracking on both before reading a cross-host median as a latency advantage. Sources from one host share a clock and need no such caveat.

Record layout (little-endian, flashblocks_recorder.recorder.HEADER_FMT):

received_at_ms    uint64
payload_id        uint64  (low 8 bytes of the hex payload id)
flashblock_index  uint16  (0xFFFF = missing)
block_number      uint32  (0 = missing)
base_timestamp    uint32  (0 = missing)
tx_count          uint16
tx_hash[tx_count] 8 bytes each (first 8 bytes of keccak(raw_tx))

flashblock_index 0 is always the system deposit tx (empty of interest). User transactions start at index 1. base_timestamp is only present on index 0 (it's part of the block header, sent once per block).

Each tx_hash is truncated to its first 8 bytes (64 bits), not the full 32 — this plus dropping per-line JSON/key overhead is what gets storage down to roughly a tenth of the old JSONL format's size (full 66-char hex hashes, repeated per line, dominated it — and being cryptographic output they don't gzip). Collision odds within the scope this tool is used for (matching a known tx against a day's worth of flashblocks) are negligible. flashblocks-lookup truncates its query hash the same way before matching.

Look up a transaction

.venv/bin/flashblocks-lookup 0xeb236e8734e5d56245aabccdcdd4e1abbc567be1de4bac4e0b4c04e4aeb3c74e

Scans the recorded .bin files and prints which block/flashblock/wall-clock time that tx hash landed in, if present.

Notes

• The public wss://mainnet.flashblocks.base.org/ws endpoint pushes bare Flashblock JSON objects with no subscribe handshake (Base's own docs discourage depending on this directly for production trading decisions, but it's fine for a passive recorder). Messages may arrive brotli-compressed binary frames — handled transparently.
• If you point FLASHBLOCKS_WS_URL at a provider (Alchemy/QuickNode/Chainstack) instead, those wrap messages in a standard eth_subscription JSON-RPC envelope after you send an eth_subscribe request — parse_flashblock unwraps that shape too, but you'd need to send the subscribe request yourself (not currently wired up, since the default target doesn't need it).
• Throughput is trivial (~5 msg/s), so flush-per-write is used for crash-safety over raw throughput.