Summary
Allow independent stacks to be applied concurrently during forge up / forge stack apply, reducing wall-clock time for environments with multiple non-dependent stacks.
Motivation
Forge applies stacks sequentially — each stack must complete before the next begins (src/stack/mod.rs:127-141). In a typical multi-stack environment like:
clusters:
- name: west
stacks: [metallb, operator, site, gateway]
stacks metallb and operator may be fully independent, yet operator waits for metallb to finish before it can start. As the number of stacks grows, the cumulative wait becomes significant.
Parallelising independent stacks would reduce total apply time to the length of the critical path rather than the sum of all stacks.
Current behaviour
- Stacks are applied in list order via a sequential
for loop (apply_stacks in src/stack/mod.rs).
- Steps within each stack are sequential (
execute_steps in src/stack/engine.rs:230-243).
- The entire codebase is synchronous — no async runtime, no threading (
README: "synchronous CLI with no async runtime").
- State is written under a file lock after all stacks complete.
Proposed solution
Introduce an opt-in mechanism for declaring parallelisable stacks, and a runtime that can execute them concurrently.
Configuration surface
A possible approach: allow grouping stacks into parallel batches in the cluster's stacks list. For example:
clusters:
- name: west
stacks:
- parallel: [metallb, operator] # run concurrently
- site # after both finish
- gateway # after site
Or alternatively, a dependency DAG:
stacks:
metallb:
steps: [...]
operator:
steps: [...]
site:
dependsOn: [metallb, operator]
steps: [...]
gateway:
dependsOn: [site]
steps: [...]
Both approaches preserve backwards compatibility — existing flat stacks: [a, b, c] lists continue to mean sequential.
Runtime changes
- Introduce a thread pool (e.g.
std::thread::scope or rayon) or an async runtime (e.g. tokio) to run parallel stacks.
- State updates must be synchronised: either per-stack locking, or collect results and write state after a parallel batch completes.
- Error handling: if one stack in a parallel group fails, the others in that group should either be cancelled or allowed to finish (user-configurable or opinionated default — to be decided).
Capture steps that produce values consumed by a later stack create an implicit dependency that should be validated at plan time.
Steps within a stack
Step-level parallelism is out of scope for this issue. Steps within a single stack remain sequential — their ordering is meaningful.
Acceptance criteria
Alternatives considered
- Intra-step parallelism only (e.g. parallel Helm installs within a stack): more granular but harder to reason about, and the user-facing benefit is smaller since most stacks are short.
- External orchestration (e.g.
make -j): avoids in-tool complexity but loses state tracking, error handling, and the single-config-file ergonomics.
Additional context
PR instructions
The PR description should include:
- How to review the changes (areas of interest and suggested review order)
- Steps showing how to manually confirm the feature works locally (e.g. create a forge.yaml with parallel stacks, run
forge up, observe concurrent output and correct state)
Summary
Allow independent stacks to be applied concurrently during
forge up/forge stack apply, reducing wall-clock time for environments with multiple non-dependent stacks.Motivation
Forge applies stacks sequentially — each stack must complete before the next begins (
src/stack/mod.rs:127-141). In a typical multi-stack environment like:stacks
metallbandoperatormay be fully independent, yetoperatorwaits formetallbto finish before it can start. As the number of stacks grows, the cumulative wait becomes significant.Parallelising independent stacks would reduce total apply time to the length of the critical path rather than the sum of all stacks.
Current behaviour
forloop (apply_stacksinsrc/stack/mod.rs).execute_stepsinsrc/stack/engine.rs:230-243).README: "synchronous CLI with no async runtime").Proposed solution
Introduce an opt-in mechanism for declaring parallelisable stacks, and a runtime that can execute them concurrently.
Configuration surface
A possible approach: allow grouping stacks into parallel batches in the cluster's
stackslist. For example:Or alternatively, a dependency DAG:
Both approaches preserve backwards compatibility — existing flat
stacks: [a, b, c]lists continue to mean sequential.Runtime changes
std::thread::scopeorrayon) or an async runtime (e.g.tokio) to run parallel stacks.Capturesteps that produce values consumed by a later stack create an implicit dependency that should be validated at plan time.Steps within a stack
Step-level parallelism is out of scope for this issue. Steps within a single stack remain sequential — their ordering is meaningful.
Acceptance criteria
Capturecross-stack dependencies are validated at plan timeforge statuscorrectly reports per-stack state for parallel runsAlternatives considered
make -j): avoids in-tool complexity but loses state tracking, error handling, and the single-config-file ergonomics.Additional context
PR instructions
The PR description should include:
forge up, observe concurrent output and correct state)