fix: fix empty `block-nodes.json` race in parallel deploy by JeffreyDallas · Pull Request #4647 · hiero-ledger/solo

JeffreyDallas · 2026-06-12T20:33:35Z

Summary

When parallelDeploy: true, the one-shot orchestrator runs Deploy block node and Deploy network node (including node setup) concurrently. node setup's updateBlockNodesJson() was reading blockNodeMap before block-node add's updateConsensusNodesInRemoteConfig() had populated it, so block-nodes.json landed on the consensus node as {"nodes":[]}. The node started without any block node connections.

Root cause (confirmed from solo.log):

block-node add helm-installs the block node (~20–30 s) and only calls updateConsensusNodesInRemoteConfig() after the pod is healthy
Concurrently, node setup starts, loads remoteConfig, and runs updateBlockNodesJson() ~8 s before the block node pod is even ready
block-node add skips re-copying block-nodes.json because ledgerPhase === UNINITIALIZED
Result: permanent empty block-nodes.json on the node

Fix — event-gate approach (no change to parallelDeploy):

New BlockNodeDeployed event type + BlockNodeDeployedEvent class
block-node add emits the event after handleConsensusNodeUpdating() finishes (blockNodeMap persisted)
The skip callback in the one-shot orchestrator emits the event immediately when block-node deploy is skipped (not needed / already deployed), so the gate resolves from history and there is zero extra latency on sequential or retry paths
withWaitCondition(SoloEventType.BlockNodeDeployed, 10 min) added to the "Setup consensus node" phase — updateBlockNodesJson() now only runs after the blockNodeMap is guaranteed populated

Also adds BlockNodesJsonEmptySoloError (SOLO-5074) as an early-fail guard in createAndCopyBlockNodeJsonFileForConsensusNode: if the generated nodes array is empty the function throws immediately with actionable troubleshooting steps instead of silently deploying broken config.

Test plan

Run task test-e2e-standard — existing one-shot test suite
Run one-shot with parallelDeploy: true and confirm block-nodes.json contains node entries on the consensus node
Run one-shot with parallelDeploy: false (sequential) and confirm no regression / no extra wait time
Retry scenario: run deploy → destroy → deploy again and confirm block-nodes.json is correct on second deploy

🤖 Generated with Claude Code

…deploy When parallelDeploy is true, the one-shot orchestrator runs 'Deploy block node' and 'Deploy network node' (including node setup) concurrently. node setup's updateBlockNodesJson() was reading blockNodeMap before block-node add's updateConsensusNodesInRemoteConfig() had a chance to populate it, resulting in block-nodes.json being written as {"nodes":[]}. Fix: - Add BlockNodeDeployed event type and BlockNodeDeployedEvent class - Emit BlockNodeDeployedEvent from block-node add after handleConsensusNodeUpdating() persists the updated blockNodeMap - Emit the event from the skip callback when block-node deploy is skipped (not needed or already deployed), so the wait never hangs - Add withWaitCondition(BlockNodeDeployed) to the "Setup consensus node" phase so updateBlockNodesJson() only runs after the blockNodeMap is populated; waitFor checks history so sequential mode has zero extra latency Also add BlockNodesJsonEmptySoloError (SOLO-5074) as an early-fail guard in createAndCopyBlockNodeJsonFileForConsensusNode so any future empty-nodes case surfaces as a clear error instead of silent bad config. Fixes #4644 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

trunk-io · 2026-06-12T20:33:40Z

❌ This pull request failed tests. It has been removed from the merge queue. PR #4737 was used for testing. See more details here.

Failed Required Status	Conclusion
Windows Deploy and Destroy	Failure

To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

github-actions · 2026-06-12T20:40:40Z

Unit Test Results - Linux

38 tests 38 ✅ 0s ⏱️
17 suites 0 💤
1 files 0 ❌

Results for commit a7967cc.

♻️ This comment has been updated with latest results.

github-actions · 2026-06-12T20:41:48Z

Unit Test Results - Windows

1 files 337 suites 10s ⏱️
1 070 tests 1 070 ✅ 0 💤 0 ❌
1 074 runs 1 074 ✅ 0 💤 0 ❌

Results for commit a7967cc.

♻️ This comment has been updated with latest results.

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

The early-fail guard in createAndCopyBlockNodeJsonFileForConsensusNode was too strict: it fired during block node destroy when rebuildBlockNodesJsonForConsensusNodes() intentionally writes an empty file after the last block node is removed from blockNodeMap. Add allowEmpty parameter (default false) so callers in the destroy path can opt out of the guard while all deploy/setup callers still get the fail-fast protection. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

github-actions · 2026-06-15T17:43:23Z

E2E Test Report

10 files ±0 94 suites ±0 1h 26m 14s ⏱️ + 1m 28s
301 tests ±0 301 ✅ ±0 0 💤 ±0 0 ❌ ±0
320 runs ±0 320 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 90b231a. ± Comparison against base commit 3487dc2.

github-actions · 2026-06-15T17:43:23Z

E2E Test Report

10 files 94 suites 1h 24m 24s ⏱️
302 tests 302 ✅ 0 💤 0 ❌
321 runs 321 ✅ 0 💤 0 ❌

Results for commit a7967cc.

♻️ This comment has been updated with latest results.

jan-milenkov

If this has negative effect on performance, inform managers before proceeding with merge

The "Copy block-nodes.json" step inside `consensus network deploy` reads blockNodeMap from the consensus node state. In parallel one-shot deploy, block-node add registers the component before it populates blockNodeMap, creating a window where consensus network deploy sees a non-empty blockNodeComponents list but an empty blockNodeMap — causing the guard to throw BlockNodesJsonEmptySoloError. Move the withWaitCondition(BlockNodeDeployed) gate from "Setup consensus node" to "Deploy consensus node" so consensus network deploy only starts after block-node add has fully persisted blockNodeMap. Since block-node add (~30s) completes well before consensus network deploy finishes, there is no real latency cost. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

JeffreyDallas · 2026-06-16T21:32:55Z

If this has negative effect on performance, inform managers before proceeding with merge

no negative effect on oneshot timing or performance memory foot print

…k-nodes-json-empty-parallel-deploy Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com> # Conflicts: # src/core/helpers.ts

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

…k-nodes-json-empty-parallel-deploy Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com> # Conflicts: # src/commands/block-node.ts # src/commands/one-shot/orchestrator/deploy/default-one-shot-deploy-orchestrator.ts # src/core/helpers.ts

The relay JSON-RPC pod running under 250Mi is prone to OOM under Windows/WSL2 CI resource pressure. An OOM kill triggers a restart that creates memory/CPU churn, which delays mirror-node postgres initialisation and causes the mirror-grpc readiness check to time out. Raise the request to 350Mi and the limit to 700Mi to prevent OOM and give the relay enough headroom to start cleanly alongside the other one-shot components. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

JeffreyDallas · 2026-06-23T20:36:18Z

Wait #4798 to fix migration test

JeffreyDallas requested a review from a team as a code owner June 12, 2026 20:33

JeffreyDallas requested review from boris-bonin and jeromy-cannon June 12, 2026 20:33

jan-milenkov assigned JeffreyDallas Jun 15, 2026

JeffreyDallas and others added 4 commits June 15, 2026 11:30

format

42d15ce

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

Merge branch 'main' into fix/block-nodes-json-empty-parallel-deploy

d471dbe

fix migration

90b231a

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

JeffreyDallas added PR: Needs Team Approval A pull request that needs review from a team member. P1-💎 Current Milestone & Goals and removed PR: Checks Failed A pull request where the checks have failed. labels Jun 15, 2026

JeffreyDallas changed the title ~~fix(one-shot): fix empty block-nodes.json race in parallel deploy~~ fix: fix empty block-nodes.json race in parallel deploy Jun 15, 2026

jan-milenkov previously approved these changes Jun 16, 2026

View reviewed changes

jan-milenkov added PR: Unresolved Comments A pull request where there are comments and they need to be resolved. and removed PR: Needs Team Approval A pull request that needs review from a team member. labels Jun 16, 2026

JeffreyDallas dismissed jan-milenkov’s stale review via 4ac66e8 June 16, 2026 19:03

JeffreyDallas removed the PR: Unresolved Comments A pull request where there are comments and they need to be resolved. label Jun 16, 2026

JeffreyDallas removed the PR: Checks Failed A pull request where the checks have failed. label Jun 17, 2026

jan-milenkov changed the title ~~fix: fix empty block-nodes.json race in parallel deploy~~ fix: fix empty block-nodes.json race in parallel deploy Jun 18, 2026

jan-milenkov previously approved these changes Jun 18, 2026

View reviewed changes

jan-milenkov added the PR: Ready to Merge A pull request that is ready to merge. label Jun 18, 2026

jeromy-cannon marked this pull request as draft June 19, 2026 15:49

jeromy-cannon added PR: Merge Conflicts A pull request that has merge conflicts that need to be resolved. and removed PR: Needs Team Approval A pull request that needs review from a team member. PR: Ready to Merge A pull request that is ready to merge. labels Jun 19, 2026

JeffreyDallas added 2 commits June 22, 2026 08:54

Merge commit '5e1d5225e4d393b3fa2590bbad1dd334a7756079' into fix/bloc…

eb82d3f

…k-nodes-json-empty-parallel-deploy Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com> # Conflicts: # src/core/helpers.ts

format

d8d0cad

Signed-off-by: Jeffrey Tang <jeffrey@swirldslabs.com>

JeffreyDallas dismissed jan-milenkov’s stale review via d8d0cad June 22, 2026 13:56

JeffreyDallas removed the PR: Merge Conflicts A pull request that has merge conflicts that need to be resolved. label Jun 22, 2026

JeffreyDallas marked this pull request as ready for review June 22, 2026 20:29

JeffreyDallas added the PR: Needs Team Approval A pull request that needs review from a team member. label Jun 23, 2026

Merge branch 'main' into fix/block-nodes-json-empty-parallel-deploy

798926d

jeromy-cannon marked this pull request as draft June 23, 2026 06:05

jeromy-cannon added PR: Merge Conflicts A pull request that has merge conflicts that need to be resolved. and removed PR: Needs Team Approval A pull request that needs review from a team member. labels Jun 23, 2026

JeffreyDallas and others added 2 commits June 23, 2026 08:54

Merge branch 'main' into fix/block-nodes-json-empty-parallel-deploy

c64340b

JeffreyDallas removed the PR: Merge Conflicts A pull request that has merge conflicts that need to be resolved. label Jun 23, 2026

JeffreyDallas marked this pull request as ready for review June 23, 2026 14:46

jeromy-cannon added the PR: Checks Failed A pull request where the checks have failed. label Jun 23, 2026

jeromy-cannon marked this pull request as draft June 23, 2026 15:29

JeffreyDallas marked this pull request as ready for review June 23, 2026 16:16

JeffreyDallas and others added 2 commits June 23, 2026 11:16

Merge branch 'main' into fix/block-nodes-json-empty-parallel-deploy

a176c09

JeffreyDallas added Blocked Further development work is blocked by other item and removed PR: Checks Failed A pull request where the checks have failed. labels Jun 23, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: fix empty `block-nodes.json` race in parallel deploy#4647

fix: fix empty `block-nodes.json` race in parallel deploy#4647
JeffreyDallas wants to merge 16 commits into
mainfrom
fix/block-nodes-json-empty-parallel-deploy

JeffreyDallas commented Jun 12, 2026

Uh oh!

trunk-io Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

github-actions Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

jan-milenkov left a comment

Uh oh!

JeffreyDallas commented Jun 16, 2026

Uh oh!

JeffreyDallas commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

JeffreyDallas commented Jun 12, 2026

Summary

Test plan

Uh oh!

trunk-io Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results - Linux

Uh oh!

github-actions Bot commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results - Windows

Uh oh!

github-actions Bot commented Jun 15, 2026

E2E Test Report

Uh oh!

github-actions Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Test Report

Uh oh!

jan-milenkov left a comment

Choose a reason for hiding this comment

Uh oh!

JeffreyDallas commented Jun 16, 2026

Uh oh!

JeffreyDallas commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

trunk-io Bot commented Jun 12, 2026 •

edited

Loading

github-actions Bot commented Jun 12, 2026 •

edited

Loading

github-actions Bot commented Jun 12, 2026 •

edited

Loading

github-actions Bot commented Jun 15, 2026 •

edited

Loading