Review/fastsync migration#39
Open
i-naman wants to merge 79 commits into
Open
Conversation
…nagement and iteration.
…d server implementations.
…ead of fs.NodeInfo.
The inner loop used `for i := currentBlock; i >= startBlock; i--` with
uint64 variables. When startBlock is 0 (chain has fewer blocks than
batchSize), the loop processes block 0, then decrements i to uint64
max (~1.8e19) instead of stopping. Since uint64 max >= 0 is always
true, the loop spins indefinitely issuing VerifiedGet calls for
non-existent blocks, flooding logs with "tbtree: key not found" errors
and blocking transaction submission on fresh/low-block-count chains.
Fix: restructure as a top-decrement loop `for i := currentBlock + 1;
i > startBlock; { i-- }` so i never passes through zero and the
underflow is impossible.
Co-authored-by: Doc <saishibu@jupitermeta.io>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…priorRouter, and standardize NodeInfo import alias.
…ommitment serialization, and improved FastSync V2 documentation
* feat: Update checksum version to 2 and hardcode protocol version for priorRouter, and standardize NodeInfo import alias. * refactor: update sync protocol with extended transaction fields, ZK commitment serialization, and improved FastSync V2 documentation
…tion for bisection callbacks
…into fix/Fastsync
- Added a notification mechanism for the sequencer's vote collector in ListenerHandler. - Enhanced the Consensus struct to manage vote notifications and round contexts. - Updated vote result storage to be scoped by block hash, allowing for better tracking of votes per round. - Refactored vote processing to utilize event-driven architecture, improving efficiency and responsiveness. - Introduced targeted vote trigger broadcasts to committee peers, ensuring only relevant nodes participate in voting.
…into fix/Fastsync
… for Sequencer refactoring
… update ignore patterns
…database state retrieval
…ng zero balance or creating new accounts
…sses from peerstore
…se operation timeouts
- Added FastSync V2 settings to jmdn_default.yaml, allowing for enhanced control over sync behavior. - Updated main.go to conditionally initialize FastSync V2 based on configuration, including support for allowed peers. - Enhanced FastSyncSettings struct in config/settings/config.go to manage sync options and defaults. - Implemented default values for FastSync settings in loader.go to ensure proper initialization.
…onal initialization
…arker by restoring its original value after writes
…into fix/Fastsync
…econciliation fallback
…rations - Implemented deduplication for address and DID entries in BatchRestoreAccounts to ensure unique keys and maintain the latest updates. - Added logic to chunk operations into smaller batches for execution, adhering to ImmuDB's MaxTxEntries limit, improving transaction handling and performance. - Updated logging for better error tracking during batch execution.
…into fix/Fastsync
…ounts - Implemented a pre-fetch mechanism to retrieve all existing account values in a single GetAll RPC, reducing connection pool exhaustion during concurrent operations. - Refactored the logic to check for existing accounts using the pre-fetched data, improving efficiency and maintaining correctness in account updates. - Enhanced handling of account creation and updates to ensure only necessary writes occur, preserving the latest data while avoiding unnecessary operations.
- Updated the JMDN-FastSync dependency to version v0.0.0-20260520134039-a630fd87e742 in both go.mod and go.sum files for compatibility with recent changes. - Adjusted the scanning order in getKeysBatch to ascending for reliable prefix filtering.
- Added ListAccountsPaginatedCursor function to provide a cursor-based alternative to ListAccountsPaginated, improving performance by avoiding O(N²) re-scans. - Updated account manager to utilize the new cursor-based pagination in NewAccountNonceIterator and GetAccountsByNonces methods, enhancing efficiency in account retrieval. - Adjusted related logic to support cursor management, ensuring seamless iteration through accounts without unnecessary overhead.
* feat: add ListAccountsPaginatedFrom and GetZKBlockByNumberFast functions for improved account retrieval - Introduced ListAccountsPaginatedFrom for cursor-based pagination of accounts, allowing efficient retrieval starting from a specified key. - Added GetZKBlockByNumberFast for faster block retrieval without proof generation, optimizing performance for bulk reads. - Refactored existing account retrieval logic to utilize the new functions, enhancing overall efficiency and reducing connection pool strain. - Updated error handling and logging for better traceability during account and block operations. * chore: update JMDN-FastSync dependency version in go.mod and go.sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260526085909-5117eb5dffbf in both go.mod and go.sum files for compatibility with recent changes. --------- Co-authored-by: Neeraj Chowdary <57310710+neerajchowdary889@users.noreply.github.com>
* feat: add ListAccountsPaginatedFrom and GetZKBlockByNumberFast functions for improved account retrieval - Introduced ListAccountsPaginatedFrom for cursor-based pagination of accounts, allowing efficient retrieval starting from a specified key. - Added GetZKBlockByNumberFast for faster block retrieval without proof generation, optimizing performance for bulk reads. - Refactored existing account retrieval logic to utilize the new functions, enhancing overall efficiency and reducing connection pool strain. - Updated error handling and logging for better traceability during account and block operations. * chore: update JMDN-FastSync dependency version in go.mod and go.sum - Updated the JMDN-FastSync dependency to version v0.0.0-20260526085909-5117eb5dffbf in both go.mod and go.sum files for compatibility with recent changes. * feat: integrate Redis for asynchronous account synchronization - Added Redis support for the account sync worker, enabling WriteAccounts and BatchUpdateAccounts to enqueue operations to a Redis Stream, decoupling them from the ImmuDB commit latency. - Updated configuration to include Redis connection settings in jmdn_default.yaml and corresponding defaults. - Enhanced error handling to ensure proper initialization of the sync worker and logging for better traceability. - Refactored account management functions to utilize the new Redis streaming mechanism for improved performance and reliability. --------- Co-authored-by: Neeraj Chowdary <57310710+neerajchowdary889@users.noreply.github.com>
… immudb_account_manager.go to streamline code and improve maintainability.
This commit introduces proper support for Redis authentication and ensures all nested configuration structures perfectly map to environment variables via Viper. Changes: - Add Password field to RedisSettings struct in config.go. - Update main.go to pass cfg.Database.Redis.Password to the Redis client during initialization. - Update defaults.go and jmdn_default.yaml to explicitly bind Redis URL to 127.0.0.1:6379 instead of an empty string or wildcard. - Update loader.go's setDefaults() to explicitly register all top-level Security settings (e.g., global_rate_limit, trusted_proxies). - Introduce dynamic Viper default registration for Security.Services in loader.go, fully enabling runtime environment variable overrides for nested map fields (e.g. JMDN_SECURITY_SERVICES_EXPLORER_API_RATE_LIMIT).
- FastsyncV2 adapter now maps ChainID and AccessList into non-headers protobufs - immudb_data_writer and immudb_block_nonheaders properly serialize/deserialize these fields to defaultdb - immudb_headers_writer now correctly stores and serves LogsBloom This fixes the Merkle hash divergence on newly fast-synced nodes.
- account_sync_worker: parseAccountsPayload now correctly pushes the DID key into the stream so BatchRestoreAccounts successfully creates the bound did: reference for new accounts. - account_immuclient: BatchRestoreAccounts now performs field-merging (DIDAddress, CreatedAt, AccountType, Metadata) when processing partial account updates from Fastsync Reconciliation. This prevents data loss for active accounts. - account_immuclient: PutNonceofAccount migrated to a monotonic nanosecond counter (Option C) to resolve silent uint64 overflow bugs.
…etch - Created standalone Scripts/replay_blocks_and_export.go for computing TxCount - Created standalone Scripts/patch_accounts.go for updating Nonces and standardizing timestamps - Replaced CheckNonceAndGetLatest in Security.go with O(1) GetAccount verification - Fixed new account generation in DB_OPs to initialize Nonce to 0 natively - Hardened DID registration to enforce UnixNano() timestamps
…couple TxCountSent - schema: added TxCountSent to Account schema to decouple analytical stats from cryptographic Nonce - db: added UpdateAccountSenderState to handle atomic balance and state updates - core: injected execution-time strict nonce validation (tx.Nonce < didDoc.Nonce) in deductFromSender to eliminate same-block double-execution loophole - core: fixed severe bug where sequencer failed to persist Nonce updates to DB after block execution - scripts: created patch_accounts_v2.go to safely migrate accountsdb by mapping TxCount -> TxCountSent and MaxNonce+1 -> Nonce
Moved Redis consumer group initialization into an asynchronous retry loop inside runWorker. The node now boots successfully even if Redis is completely offline, gracefully degrading bulk sync operations while preserving live transaction processing. Retry interval set to 30s to minimize log spam.
… and enhance Redis integration This commit refactors the account synchronization process by modifying the initialization of the account sync worker. The StartAccountSyncWorker function now creates a WorkerManager and installs it as the package-level queue, allowing for lazy worker startup. Additionally, the BatchRestoreAccounts function has been updated to accept a context parameter, improving its integration with the overall context management. The logging messages have been adjusted for clarity, and new methods for managing the worker lifecycle have been introduced to enhance performance and reliability.
…into fix/Fastsync
…7c and add redis/go-redis/v9 v9.19.0 dependency - Updated the JMDN-FastSync dependency version in go.mod and go.sum for compatibility with recent changes. - Added redis/go-redis/v9 v9.19.0 as a new dependency in go.mod and go.sum.
…eID generation
- Reverted Account options and restored pure 'GenerateStateID()' for locally originated accounts.
- Removed hacky deterministic 'StateID' origination from 'BlockProcessing'. Consensus now strictly requires accounts to exist before transacting.
- Added pre-flight hook in 'eth_getBalance' facade to natively originate and gossip missing accounts via P2P before consensus.
- Updated 'Account' adapter mappings to cross-wire Fastsync 'types.Account' safely ('StateID' -> 'Nonce', 'Nonce' -> 'TxNonce').
- Secured incoming P2P DID Gossips using 'StorePropagatedAccount' to perfectly preserve network StateIDs.
…ration-and-consensus # Conflicts: # DB_OPs/Nodeinfo/account_sync_worker.go
… and adding TxNonce
- Introduced a new helper function `enqueueRecordsChunked` to split account records into fixed-size messages, ensuring each message adheres to the `maxRecordsPerMessage` limit to prevent exceeding Redis's bulk string size. - Updated `WriteAccounts` and `BatchUpdateAccounts` methods to utilize the new chunking logic, enhancing reliability by allowing best-effort delivery across chunks. - Added a white-box test for the chunking logic to validate message boundaries and error handling. - Updated documentation to reflect changes in the enqueue process and message size constraints. - Added a replace directive in `go.mod` for local development with the JMDN-FastSync repository.
… replays - Security module now exclusively uses SecurityCache for TxNonce resolution, drastically reducing DB I/O during batch validation. - Cache enforces mutation upon successful verification (UpdateTxNonce), mitigating intra-block replay attack vectors. - Renamed cache methods to strictly differentiate between Ethereum TxNonce and Fastsync Nonce (StateID). - Fixed static off-by-one error (MaxNonce vs MaxNonce + 1) in V3 migration script.
…ration-and-consensus
…s' into review/fastsync-migration
- Replaced deprecated ioutil.ReadFile and ioutil.WriteFile with os.ReadFile and os.WriteFile in bls-sgin.go - Replaced deprecated reflect.Ptr with reflect.Pointer in seednode.go - Applied auto-formatting via make fmt to clean up unused imports These changes resolve linter errors introduced by standard library deprecations in modern Go versions while keeping the logic functionally identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Why
How
Testing
Checklist
make buildpassesmake lintpassesmake fmt-checkpasses