You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Adapt the actor resource-telemetry stack (#550: GetWorkloadStats / GetActiveWorkloadStats on both ateom runtimes, the atelet stats sweep with template-level metrics and per-actor usage events) to multi-actor workers (#1266).
The stack was built on the one-actor-per-worker state machine. Once workers host N actors, worker-pod-level metrics (metrics-server, kubectl ate top workers) stop being a usable proxy for any actor's usage. This pipeline becomes the only actor-granular source, so it must be multi-actor-clean before workers report capacity > 1.
GetActiveWorkloadStatsResponse is now repeated WorkloadStatsSample samples = 1;. The oneof and NoSampleReason are deleted outright, no reserved tombstones.
"Available" is the empty list.
A workload with no numbers yet (boot, restore, teardown, a lifecycle transition underneath the read) is a pending entry: full attribution with source = STATS_SOURCE_UNSPECIFIED, which the sample contract already defines as "not measured". A workload that dies during boot is attributable, not anonymous.
The atelet sweep folds every entry of every response. Pending entries contribute nothing: not to aggregates (sampled_actors keeps meaning "measured"), not to CPU baselines, no usage event.
The keyed GetWorkloadStats(actor_uid) is unchanged. It is uid-addressed, so its per-actor codes stay correct at any occupancy, and its cost advantage becomes real: one actor instead of the whole worker.
Version skew during a mixed rollout is accepted, not bridged. Singular and repeated message fields of one number share a wire encoding, so mixed versions degrade (an old atelet sees a multi-actor worker as one sample; a booting worker on an old ateom reads as available for a tick) and converge with the rollout.
Both runtimes: the handlers still read one slot
The plural response is filled from a singular source. GetActiveWorkloadStats on each runtime loads the one activeActor atomic, measures it, and returns a list of at most one entry. Nothing iterates. This is the core of the remaining work, and it is the same on both runtimes:
The epic's runtime PRs introduce a per-actor registry (what today is the single activeActor slot, plus running / guestStats on micro-VM). The stats handlers must iterate that registry, one sample per entry, pending or measured, and must read it without the lifecycle mutex (the handlers are polled on a timer while the mutex is held across whole boots and checkpoints). The registry must therefore offer a lock-free snapshot: a copy-on-write map behind an atomic.Pointer, or per-entry atomics. Stats is a consumer of whatever the epic picks.
The load / sample / re-check pattern becomes per entry: an actor checkpointed mid-sweep drops out of that sweep's response (or reports pending); the other entries are unaffected. The existing test seams (readSandboxCgroup on gVisor, fakeAgent.onCall on micro-VM) carry over per entry.
The keyed GetWorkloadStats(actor_uid) becomes a registry lookup by uid instead of a compare against the one slot. Its codes do not change.
Cgroup leaves are named <actorUID>-<container> relative to the pod scope. ocispec.GVisorCgroupLeaf supplies the name to both the OCI shaping and the stats read, so writer and reader cannot drift. The read stays on the -pause leaf, because gVisor runs every container of a sandbox inside one sentry process that lives in that leaf. The layout is flat, not nested, so there is no parent-directory aggregate; the pause leaf is the actor's number.
No version window exists here. ateom both writes the path (runsc.shapeSpec at every create and restore) and reads it, the leaves live in the worker pod's private cgroup namespace and die with the pod, and no snapshot carries the path.
What #1667 did not do: it did not make the handler iterate. With the leaf keyed by uid, measuring N actors on gVisor is N local file reads, so once the iteration above lands, gVisor needs no concurrency work. Its remaining cost is the iteration itself.
micro-VM: what multi-actor changes
This is where the remaining work is. Numbered so the phase-3 reviews can check each.
One guest client per actor. Today the stats read uses one guestStats target: one ttrpc client over one vsock to one VM. N actors means N VMs, N vsock sockets, N clients. guestStats must become per-actor state alongside the registry entry described above; the running map entry is the natural home.
Cost grows N times, and it is I/O bound. A gVisor read is a local file read. A micro-VM read is one vsock round trip per container, so one actor's worst case is maxActorContainers x statsCallTimeout = 25 x 2s = 50s. Measured in series, N actors cost N x 50s and one slow guest blocks the whole worker's sample. Requirement: a discovery call measures its actors concurrently under a small cap. Then in atelet, minActorStatsPollInterval (50s) and statsRPCTimeout (55s) become ceil(actors-capacity / cap) x the single-actor worst case, not a blind N x.
Cgroups need no telemetry work. The micro-VM stats read never touches the host cgroup: it reads the guest's cgroups through the kata-agent, and inside each VM containers live under /ateomchv/<container>. Each VM is its own kernel, so two actors cannot collide there. Host-side cgroup keying for N cloud-hypervisor processes is the epic's concern (sizing and isolation, the ateom microvm: drop privileged: true, using atelet device plugin for KVM #1254 lineage), not this issue's.
The peak over-count appears more often.memory_peak_bytes for a multi-container actor is the sum of per-container peaks, an upper bound, not the true peak. More actors per worker means more multi-container workloads, so this shows up more. The fix is an upstream kata sandbox-level stats RPC reading the /ateomchv parent, which is not started.
A dead guest hides as a pending entry. New gap. The discovery read reports "guest not answering" as a pending entry, the same shape as "booting". The poller skips both. Today a dead guest also fails readyz, so it does not stay hidden. With N actors, one dead guest sits behind healthy siblings and is invisible in metrics for as long as it stays dead. Requirement: atelet tracks how long an actor uid has been pending and emits a signal (a pending too long event, or a gauge of long-pending actors per template) past a threshold. Design open; must land with the micro-VM capacity flip.
Items 1, 2, and 5 must be done before any micro-VM pool reports capacity > 1. Item 4 is not a blocker. The handler iteration (previous section) is a blocker for both runtimes: without it, a capacity-2 gVisor worker reports one of its two actors.
Handler iteration on both runtimes, riding the epic's runtime PRs that introduce the per-actor registry. Review against: lock-free registry snapshot, per-entry transition re-check, attribution echoed inside each sample, keyed read as a uid lookup. This is the step that turns the plural wire shape into plural data.
micro-VM items 1 and 2, in the same PRs or stacked directly on them. Item 2 adds the concurrency-under-a-cap invariant to the review list.
atelet timeout formulas and the dead-guest signal (item 5), in one atelet PR, before the first micro-VM capacity flip.
Metric names and label sets do not change. Template-level aggregation is actor-count-agnostic, so no dashboard migration is expected.
Out of scope
kubectl ate top workers: needs no multi-actor adaptation. Its CPU/MEMORY are pod-level totals that stay true at any occupancy, and the plural display already landed on main. Positional names and ALLOCATED/CAPACITY columns from multi-actor worker API #1283 are ordinary CLI work, not this issue's.
Actor-level usage in the CLI, in any form (top actors, top workers --actors, and the live read-through path they would need: an atelet relay RPC and ateapi node-grouped fan-out). Excluded by decision. Per-actor usage remains available via the events channel and, for one target, the keyed GetWorkloadStats. Revisit only with a concrete consumer. The earlier design is recorded here so it is not re-derived: ateapi pages by worker, one DialForNode call per node, per-worker discovery reads, "metrics unavailable" rows on node timeout, no central sample cache.
Context
Adapt the actor resource-telemetry stack (#550:
GetWorkloadStats/GetActiveWorkloadStatson both ateom runtimes, the atelet stats sweep with template-level metrics and per-actor usage events) to multi-actor workers (#1266).The stack was built on the one-actor-per-worker state machine. Once workers host N actors, worker-pod-level metrics (metrics-server,
kubectl ate top workers) stop being a usable proxy for any actor's usage. This pipeline becomes the only actor-granular source, so it must be multi-actor-clean before workers report capacity > 1.Status
repeated WorkloadStatsSample samplesactiveActorslot, at most one sample)The wire shape (step 1, done in #1644)
GetActiveWorkloadStatsResponseis nowrepeated WorkloadStatsSample samples = 1;. TheoneofandNoSampleReasonare deleted outright, noreservedtombstones.source = STATS_SOURCE_UNSPECIFIED, which the sample contract already defines as "not measured". A workload that dies during boot is attributable, not anonymous.sampled_actorskeeps meaning "measured"), not to CPU baselines, no usage event.GetWorkloadStats(actor_uid)is unchanged. It is uid-addressed, so its per-actor codes stay correct at any occupancy, and its cost advantage becomes real: one actor instead of the whole worker.Version skew during a mixed rollout is accepted, not bridged. Singular and repeated message fields of one number share a wire encoding, so mixed versions degrade (an old atelet sees a multi-actor worker as one sample; a booting worker on an old ateom reads as available for a tick) and converge with the rollout.
Both runtimes: the handlers still read one slot
The plural response is filled from a singular source.
GetActiveWorkloadStatson each runtime loads the oneactiveActoratomic, measures it, and returns a list of at most one entry. Nothing iterates. This is the core of the remaining work, and it is the same on both runtimes:activeActorslot, plusrunning/guestStatson micro-VM). The stats handlers must iterate that registry, one sample per entry, pending or measured, and must read it without the lifecycle mutex (the handlers are polled on a timer while the mutex is held across whole boots and checkpoints). The registry must therefore offer a lock-free snapshot: a copy-on-write map behind anatomic.Pointer, or per-entry atomics. Stats is a consumer of whatever the epic picks.readSandboxCgroupon gVisor,fakeAgent.onCallon micro-VM) carry over per entry.GetWorkloadStats(actor_uid)becomes a registry lookup by uid instead of a compare against the one slot. Its codes do not change.gVisor measurement path (done in #1667)
Cgroup leaves are named
<actorUID>-<container>relative to the pod scope.ocispec.GVisorCgroupLeafsupplies the name to both the OCI shaping and the stats read, so writer and reader cannot drift. The read stays on the-pauseleaf, because gVisor runs every container of a sandbox inside one sentry process that lives in that leaf. The layout is flat, not nested, so there is no parent-directory aggregate; the pause leaf is the actor's number.No version window exists here. ateom both writes the path (
runsc.shapeSpecat every create and restore) and reads it, the leaves live in the worker pod's private cgroup namespace and die with the pod, and no snapshot carries the path.What #1667 did not do: it did not make the handler iterate. With the leaf keyed by uid, measuring N actors on gVisor is N local file reads, so once the iteration above lands, gVisor needs no concurrency work. Its remaining cost is the iteration itself.
micro-VM: what multi-actor changes
This is where the remaining work is. Numbered so the phase-3 reviews can check each.
One guest client per actor. Today the stats read uses one
guestStatstarget: one ttrpc client over one vsock to one VM. N actors means N VMs, N vsock sockets, N clients.guestStatsmust become per-actor state alongside the registry entry described above; therunningmap entry is the natural home.Cost grows N times, and it is I/O bound. A gVisor read is a local file read. A micro-VM read is one vsock round trip per container, so one actor's worst case is
maxActorContainers x statsCallTimeout= 25 x 2s = 50s. Measured in series, N actors cost N x 50s and one slow guest blocks the whole worker's sample. Requirement: a discovery call measures its actors concurrently under a small cap. Then in atelet,minActorStatsPollInterval(50s) andstatsRPCTimeout(55s) becomeceil(actors-capacity / cap) xthe single-actor worst case, not a blind N x.Cgroups need no telemetry work. The micro-VM stats read never touches the host cgroup: it reads the guest's cgroups through the kata-agent, and inside each VM containers live under
/ateomchv/<container>. Each VM is its own kernel, so two actors cannot collide there. Host-side cgroup keying for N cloud-hypervisor processes is the epic's concern (sizing and isolation, the ateom microvm: dropprivileged: true, using atelet device plugin for KVM #1254 lineage), not this issue's.The peak over-count appears more often.
memory_peak_bytesfor a multi-container actor is the sum of per-container peaks, an upper bound, not the true peak. More actors per worker means more multi-container workloads, so this shows up more. The fix is an upstream kata sandbox-level stats RPC reading the/ateomchvparent, which is not started.A dead guest hides as a pending entry. New gap. The discovery read reports "guest not answering" as a pending entry, the same shape as "booting". The poller skips both. Today a dead guest also fails readyz, so it does not stay hidden. With N actors, one dead guest sits behind healthy siblings and is invisible in metrics for as long as it stays dead. Requirement: atelet tracks how long an actor uid has been pending and emits a signal (a
pending too longevent, or a gauge of long-pending actors per template) past a threshold. Design open; must land with the micro-VM capacity flip.Items 1, 2, and 5 must be done before any micro-VM pool reports capacity > 1. Item 4 is not a blocker. The handler iteration (previous section) is a blocker for both runtimes: without it, a capacity-2 gVisor worker reports one of its two actors.
Plan
Metric names and label sets do not change. Template-level aggregation is actor-count-agnostic, so no dashboard migration is expected.
Out of scope
kubectl ate top workers: needs no multi-actor adaptation. Its CPU/MEMORY are pod-level totals that stay true at any occupancy, and the plural display already landed on main. Positional names andALLOCATED/CAPACITYcolumns from multi-actor worker API #1283 are ordinary CLI work, not this issue's.top actors,top workers --actors, and the live read-through path they would need: an atelet relay RPC and ateapi node-grouped fan-out). Excluded by decision. Per-actor usage remains available via the events channel and, for one target, the keyedGetWorkloadStats. Revisit only with a concrete consumer. The earlier design is recorded here so it is not re-derived: ateapi pages by worker, oneDialForNodecall per node, per-worker discovery reads, "metrics unavailable" rows on node timeout, no central sample cache.privileged: true, using atelet device plugin for KVM #1254 lineage), not telemetry.Related
#550 (parent) · #594 (phase 0, complete: #667 #739 #832) · #896 (phase 1: #899 #961 #1206 merged; open for the lifecycle-bracket follow-up) · #1266 (the epic; API in #1283, gVisor leaves in #1667, uVM cgroup prep in #1254) · #1644 (step 1)