benchmarking: spread actors over several worker pools - #1760
Open
Chenyi Wang (chw120) wants to merge 1 commit into
Open
Chenyi Wang (chw120) wants to merge 1 commit into
Chenyi Wang (chw120) wants to merge 1 commit into
Conversation
The benchmark stack creates a single WorkerPool, so every actor a run makes
is interchangeable with every other and the scheduler may place it on any
worker. That blocks two things a scale run needs.
A run cannot compare machine types. Measuring n4 against c4 means two runs,
which measures the machine types and the day's cluster weather together.
More importantly, an actor cannot safely move between machine types, and
nothing today stops it. Suspending an actor writes its guest memory to a
snapshot; resuming restores that memory on whichever worker the scheduler
picks. The snapshot carries the CPU features the guest observed, and no layer
masks them to a common baseline, so an actor that resumes on a different CPU
model can fail to restore. The single pool hides this only because every
worker in it is identical -- the moment a run spans machine types it is a live
bug, and the failure surfaces as an unexplained actor crash rather than as a
placement error.
So make the pool a first-class dimension of a run, and pin each actor to one:
benchmarking/deploy_locust.sh --deploy --worker-count 100 \
--worker-pools 'n4:1:cloud.google.com/machine-family=n4,c4:1:cloud.google.com/machine-family=c4'
That creates one WorkerPool per entry, labelled pool=<name> and optionally
constrained to a node shape, splits --worker-count between them by weight, and
sends each actor to one pool for its whole life.
The pinning rides on Actor.worker_selector, which the API already has and the
scheduler already ANDs with the ActorTemplate's own workerSelector. So the
template keeps bounding actors to the benchmark pools and the new selector only
narrows an actor to one of them; no scheduler or API change is needed. A
WorkerPool's labels propagate to its workers, which is what the scheduler
matches.
An actor draws its pool once, at creation, weighted by the pool weights -- per
actor rather than per VU, so a VU running several actors still spreads them.
Weights are usually the pools' vCPU counts, so each pool gets the share of
actors it can hold.
A pool name is a class of interchangeable workers rather than the name of one
WorkerPool: it is matched as a label value, so several WorkerPools may carry
it, and an actor assigned to it may run on any worker in any of them. This
script gives each entry its own value because each entry is a distinct node
shape, but a deployment that splits one machine type across several pools for
capacity can share a value, and pinning stays exactly as strict as correctness
needs. The key is `pool` and not something like `cpu-class` because CPU
compatibility is only today's reason to separate workers; a label key flag
renames it for a deployment that separates them on some other axis.
Everything defaults to the previous behaviour: no --worker-pools means one pool
named benchmark-ateom, no per-actor selector, and the same deployment name the
existing tooling waits on.
The one hazard is the two halves disagreeing: the pools that exist and the
pools the actors ask for are configured by different scripts, and an actor
naming a pool that was never created never schedules. deploy_locust.sh and
install-ate.sh therefore take one flag and forward it to both halves, dropping
the node selector on the way to the boomer workers, which have no use for it.
Testing: new table tests cover the pool parsing, the weighted draw and the
selector it produces, and new tests against the fake control client assert the
wiring end to end -- that a created actor carries the expected worker_selector,
that it carries none when no pools are configured, that an actor's pool never
changes across creates, and that every actor a VU starts lands on a configured
pool with all pools represented. Those tests were mutation-checked: removing
the selector assignment, redrawing the pool per create, and drawing once per VU
instead of per actor each make a test fail. The shell splitting and manifest
rendering were exercised outside a cluster over the weighted, rounding,
one-worker-minimum and malformed-spec cases.
Not yet run against a live multi-pool cluster. Two things still need verifying
there: that an actor is really placed on a worker of its pool, and that a
suspend/resume cycle keeps it there.
Julian Gutierrez Oschmann (juli4n)
requested a review
from Max Smythe (maxsmythe)
September 20, 2026 21:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Large-scale benchmarks that span multiple node pools / machine families (e.g.
n4,c4,c3) require two layers of placement constraints:spec.template.nodeSelector): eachWorkerPool's worker pods must be scheduled onto the matching node pool.Actor.worker_selector): a suspended micro-VM memory snapshot captures the CPU features observed by the guest and cannot be resumed on an incompatible CPU model. While the Substrate scheduler already supports per-actorActor.worker_selector, the benchmark harness previously only deployed a singleWorkerPooland leftActor.worker_selectorunset on every actor.This PR updates both sides of the benchmark harness:
deploy.sh): creates oneWorkerPoolper entry withmetadata.labels.pool=<name>and optionalspec.template.nodeSelector.boomer): assigns each actor a pool at creation (weighted by pool capacity) and populatesActor.worker_selector: {pool: <name>}so the actor stays within that pool across its entirecreate→suspend→resumelifecycle.Changes
internal/benchmarking/boomer/userclass/pools.go: AddPoolPickerto parsename:weight[,...]specs, draw a pool weighted by capacity (Pick()), and buildActor.worker_selector(SelectorFor()matchingpool=<name>). All methods are nil-safe so single-pool runs incur no branching.internal/benchmarking/boomer/glutton/{lifecycle,durdir}.go&cmd/benchmarking/boomer-worker/main.go: Add--worker-pools, draw one pool per actor when minted instartUser(), and attachActor.worker_selectoronCreateActor.benchmarking/workloads/deploy.sh&manifests/workerpool.yaml.tmpl: Accept--worker-pools name:weight[:nodeSelectorKey=value], split--worker-countacross pools proportionally by weight, and render oneWorkerPoolper entry withmetadata.labels.pool=<name>and optionalspec.template.nodeSelector.benchmarking/deploy_locust.sh,benchmarking/locust/*,hack/install-ate.sh: Forward--worker-poolsto both the workload deployer (full spec withnodeSelector) and the boomer workers (name:weightlist) from a single flag so the two halves cannot drift.Backwards Compatibility
When
--worker-poolsis omitted (default):PoolPickerisnilandActor.worker_selectorremains unset (nil).workloads/deploy.shdeploys a singleWorkerPoolnamedbenchmark-ateomwithreplicas: ${WORKER_COUNT}and nonodeSelector.Testing
Unit tests for pool parsing, validation, nil safety, and weighted distribution (
internal/benchmarking/boomer/userclass/pools_test.go).Wiring tests asserting the
CreateActorRequestprotobuf payloads forgluttonanddurdir(internal/benchmarking/boomer/glutton/pools_wiring_test.go).Validated rendered single-pool and multi-pool manifests against the
WorkerPoolCRD withkubectl apply --validate=strict.Tests pass
Appropriate changes to documentation are included in the PR