Skip to content

engine: deploy scheduler barriers on whole levels, stalling dependents behind unrelated slow deploys #436

Description

@pskry

Problem

runPlansConcurrent executes the cross-deploy graph level by level with a
barrier between levels
(runLevel calls g.Wait() before the next level
starts). A deploy in level N+1 therefore waits for the entire level N to
finish, even when it depends on only one node in that level.

deploy "vm"   promises  lxc:100      (fast)
deploy "dns"  independent            (slow)
deploy "app"  target inputs lxc:100  (depends only on vm)

Level 0: [vm, dns]  ── g.Wait() ──┐
Level 1: [app]  ◄── blocked on the whole level, including dns

app only needs lxc:100 from vm, but the level barrier makes it wait
behind dns too. The levels are always correct (no dependency is violated),
just pessimistic on latency: a slow independent deploy in an early tier stalls
dependents that never touch it.

Fix

Convert the deploy runner from level-barrier to dependency-driven: fire each
deploy the moment its own upstream deps settle, not when its whole level does.
app starts when vm finishes; dns runs off on its own and never gates
app.

The graph already carries what's needed: deployNode.deps holds the real
per-node edges (app.deps == [vm]), and kahnLevels is only one consumer of
them. This is the exact pattern the engine already uses one layer down: the op
scheduler (opNode.deps + a pending counter, scheduled on completion) and the
step scheduler. Lift the same shape to the deploy layer.

Must preserve: failure propagation (#275)

The level model also does failure propagation: "an upstream level produced
failures, skip downstream nodes rather than racing them into ops that depend on
resources the failed producer was supposed to create." A DAG runner has to keep
that per edge: if vm fails, app must be skipped/aborted (its producer
never created lxc:100), while dns and anything independent still run. Get
the skip-on-failed-producer semantics right before merging.

Notes

Suggested labels

kind/optimization, impact/medium, priority/low

Metadata

Metadata

Assignees

No one assigned

    Labels

    impact/mediumImproves a real workflowkind/optimizationPerformance or internal quality improvement — behavior unchangedpriority/lowThe priority is low

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions