perf(l1): enable lz4 compression for trie and flat key-value tables#6530
perf(l1): enable lz4 compression for trie and flat key-value tables#6530dicethedev wants to merge 3 commits intolambdaclass:mainfrom
Conversation
Greptile SummaryThis PR adds Confidence Score: 4/5Safe to merge with minor concerns — LZ4 compression addition is correct; undocumented Only P2 findings: the core compression change is straightforward and RocksDB handles it gracefully for existing data; the crates/storage/backend/rocksdb.rs — review the full scope of
|
| Filename | Overview |
|---|---|
| crates/storage/backend/rocksdb.rs | Adds LZ4 compression for the four state CFs and also (without mention in the PR description) applies DataBlockIndexType::BinaryAndHash with hash_ratio=0.75 to all existing CF match arms; trailing whitespace on several added lines. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[RocksDBBackend::open] --> B{CF in compressible_tables?}
B -- Yes --> C[DBCompressionType::Lz4]
B -- No --> D[DBCompressionType::None]
C --> E{match cf_name}
D --> E
E --> F[HEADERS / BODIES 32KB block]
E --> G[CANONICAL_BLOCK_HASHES / BLOCK_NUMBERS 16KB + bloom + BinaryAndHash NEW]
E --> H[ACCOUNT_TRIE_NODES / STORAGE_TRIE_NODES 16KB + bloom + BinaryAndHash NEW + Lz4 NEW]
E --> I[ACCOUNT_FLATKEYVALUE / STORAGE_FLATKEYVALUE 16KB + bloom + BinaryAndHash NEW + Lz4 NEW]
E --> J[ACCOUNT_CODES 32KB blob + BinaryAndHash NEW]
E --> K[RECEIPTS 32KB + BinaryAndHash NEW]
E --> L[default 16KB + BinaryAndHash NEW]
Comments Outside Diff (1)
-
crates/storage/backend/rocksdb.rs, line 120-204 (link)Undocumented scope:
BinaryAndHashblock index applied to all CFsThe PR description states "No other logic changes," but
set_data_block_index_type(BinaryAndHash)andset_data_block_hash_ratio(0.75)are added to allmatcharms — including pre-existing CFs (CANONICAL_BLOCK_HASHES,BLOCK_NUMBERS,ACCOUNT_CODES,RECEIPTS, and the default branch) — not only the four newly compressed state tables. Athash_ratio = 0.75, each 16 KB data block gets ~12 KB of hash-index overhead appended, increasing SST file sizes for those existing CFs. This broader change should be called out explicitly and benchmarked separately from the LZ4 compression addition.Prompt To Fix With AI
This is a comment left during a code review. Path: crates/storage/backend/rocksdb.rs Line: 120-204 Comment: **Undocumented scope: `BinaryAndHash` block index applied to all CFs** The PR description states "No other logic changes," but `set_data_block_index_type(BinaryAndHash)` and `set_data_block_hash_ratio(0.75)` are added to _all_ `match` arms — including pre-existing CFs (`CANONICAL_BLOCK_HASHES`, `BLOCK_NUMBERS`, `ACCOUNT_CODES`, `RECEIPTS`, and the default branch) — not only the four newly compressed state tables. At `hash_ratio = 0.75`, each 16 KB data block gets ~12 KB of hash-index overhead appended, increasing SST file sizes for those existing CFs. This broader change should be called out explicitly and benchmarked separately from the LZ4 compression addition. How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: crates/storage/backend/rocksdb.rs
Line: 120-204
Comment:
**Undocumented scope: `BinaryAndHash` block index applied to all CFs**
The PR description states "No other logic changes," but `set_data_block_index_type(BinaryAndHash)` and `set_data_block_hash_ratio(0.75)` are added to _all_ `match` arms — including pre-existing CFs (`CANONICAL_BLOCK_HASHES`, `BLOCK_NUMBERS`, `ACCOUNT_CODES`, `RECEIPTS`, and the default branch) — not only the four newly compressed state tables. At `hash_ratio = 0.75`, each 16 KB data block gets ~12 KB of hash-index overhead appended, increasing SST file sizes for those existing CFs. This broader change should be called out explicitly and benchmarked separately from the LZ4 compression addition.
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: crates/storage/backend/rocksdb.rs
Line: 128-129
Comment:
**Trailing whitespace on newly added lines**
Several of the newly added lines carry trailing spaces (e.g. lines 128, 129, 143, 144, 158, 159, 175, 188, 201). Most CI linters and `rustfmt` will flag these; cleaning them up avoids noisy diffs in future PRs.
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "fix(l1): correct typo 10s24 → 1024" | Re-trigger Greptile
Motivation
State tables (
ACCOUNT_TRIE_NODES,STORAGE_TRIE_NODES,ACCOUNT_FLATKEYVALUE,STORAGE_FLATKEYVALUE) were previously uncompressed. On IO-bound nodes, reading fewer bytes from disk outweighs the decompression cost, LZ4 decompresses at ~4 GB/s, making it effectively free relative to disk latency.Description
Adds the four state CFs to
compressible_tablesso they receiveDBCompressionType::Lz4alongside the existing block/receipt tables. No other logic changes.Checklist
STORE_SCHEMA_VERSION(crates/storage/lib.rs) if the PR includes breaking changes to theStorerequiring a re-sync.Closes #5939