feat(meta): the WorkerPool's CPU generation pin — karpenter.k8s.aws/instance-generation next to the vendor — documented, render-guarded and verified by make verify-workerpool (#457) - #469
Merged
Conversation
…nstance-generation next to the vendor — documented, render-guarded (a non-string nodeSelector value fails the render naming the key) and verified by make verify-workerpool (#457) One pool runs one CPU feature set; #429 pinned the vendor and left the generation open. On gazelle (2026-09-14) goldens taken on m7a (gen 7) workers failed to restore on c5ad (gen 5) workers with `incompatible FeatureSet: missing features: … avx512f …`, twice per cold turn, then the turn failed. The pin is a second label in the same map, set per installation by the fleet template; the value is a string ("6"), so the meta chart refuses a bare number at the render instead of letting helm-controller fail the kagent release on apply. verify-workerpool asserts the default, the verbatim forwarding, the guard and the rendered WorkerPool of the kagent chart the range resolves to.
Published Helm chart
|
| Chart | agent-platform |
| Version | 4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd |
| Digest | sha256:35a3d48c405631fb90b3d6593bf1c5a272409469a75ece783c322443bd7d44a9 |
| Registry | public — gsoci.azurecr.io |
| Git catalog | giantswarm-test-catalog (index) |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecdPosted by architect-orb · build 8048 · commit 4266ecd · updated in place on every push
Published Helm chart
|
| Chart | agent-platform-connectivity |
| Version | 4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd |
| Digest | sha256:4d8a0fced51ca3e4b947a008ab9662dfd4d472d612e52b15d6e6403891eb46db |
| Registry | public — gsoci.azurecr.io |
| Git catalog | giantswarm-test-catalog (index) |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecdPosted by architect-orb · build 8050 · commit 4266ecd · updated in place on every push
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #457 (sub-issue of giantswarm/giantswarm#37787). The chart half; the fleet template (shared-configs) gains the matching knob and gazelle sets it in its own PRs.
Problem
One Substrate WorkerPool runs one CPU feature set: an actor's golden snapshot is a gVisor checkpoint that restores only on a host whose CPU offers every feature it recorded. #429 pinned the vendor (
karpenter.k8s.aws/instance-cpu-manufacturer: amd) and #430 recorded the caveat that a newer generation's snapshot does not restore on an older one of the same vendor. That caveat hit gazelle on 2026-09-14: Karpenter had twokagent-defaultworkers on anm7a.2xlarge(generation 7, AVX-512) and two onc5ad.xlarge(generation 5); the goldens ofsre-agentandtest-agentwere taken on the m7a workers at 16:24Z, and a new thread placed on a c5ad worker at 19:29Z failedtwice (30 s each), then
SendStreamingMessage Internalafter 60 s,dispatch error: actor "ai-…" request timed outin klaus-gateway, "the turn failed; please try again" in Slack. The next morning the pool was c6a, c5a ×2, m6a (generations 6, 5, 5, 6) and every cold turn restored — because the goldens had been re-taken on a gen-5 worker at 06:39Z, not by construction. Every spot replacement can change the mix.Change
karpenter.k8s.aws/instance-generation: "6", quoted. Which generation: one that is one CPU model across its families — for AMD on AWS 6 (c6a/m6a/r6a, EPYC 7R13 Milan) or 7 (c7a/m7a/r7a, EPYC 9R14 Genoa); 5 mixes Naples (m5a/r5a, EPYC 7571) with Rome (c5a/c5ad, EPYC 7R32) and is no feature-set pin — and one the installation's goldens restore on (an older generation's golden restores on a newer one, never the reverse). And the capacity trade-off: the pin narrows the spot pool to the generation's families × the NodePool's sizes; readkubectl get nodes -L node.kubernetes.io/instance-type,karpenter.k8s.aws/instance-generationand the NodePool's requirements first; three families is the floor. The family label stays as the narrower option.agent-platform.validateWorkerPool): akagent.substrateWorkerPool.template.nodeSelectorvalue that is not a string fails the render naming the key. The kagent chart forwards the template verbatim (toYaml) intoWorkerPool.spec.template, whosenodeSelectorismap[string]string; a bare6rendered, passed this chart's openkagentschema and would have failed only when helm-controller applied the kagent release — on every installation carrying it. No rendered object changes for an installation that sets no generation.make verify-workerpool(in CI,tests/verify-workerpool.py): the default forwards the architecture alone; an installation's three-label pin reaches the kagent release verbatim, every value a string, nothing else in the map; the unquoted generation fails the render naming the key; the kagent chart the range resolves to (0.11.0-gs.15today) renders the pin unchanged into the oneWorkerPool kagent-default'sspec.template.nodeSelector, next to the forwarded resources. The lab (kind) has no Karpenter labels — a pinned pool would simply not schedule there — so this render is where the pin is proven; the live proof is gazelle's.Not here: a restore that fails with
incompatible FeatureSetis retried on the same worker; placing it on another worker, and the durable fix (workers advertise their feature set, snapshots record it, the scheduler requires a superset — giantswarm/substrate#29, giantswarm/giantswarm#37742 row 12), are the Substrate line's.The placement decision for gazelle (recommended: generation 6)
The current goldens:
sre-agent(Swarmgeist's default agent),test-agent,issue-tracker,kyverno-policy-writerandfactory-analystwere checkpointed 06:39Z on the twoc5a.xlargeworkers (generation 5, Rome);arguson anm6a.xlarge(generation 6). The NodePoolgazelle-karpenteradmits 4/8/16/32 vCPU, nitro, spot and on-demand, amd64, any family but t2/t3/t3a, across eu-central-1a/b/c.Generation 6 is the lowest homogeneous AMD generation, three families clear the floor, and no golden has to be re-taken for compatibility: the next Harness change re-takes them on gen-6 workers by itself. Set in gazelle's
installations/gazelle/config.yaml.patch(agentPlatform.workerPoolCpuGeneration: "6") once the shared-configs knob is merged.Verification
make verify-workerpool— the four assertions above, against the publishedkagent0.11.0-gs.15.make verify-kagent-harness verify-components verify-modes verify-disruption— unchanged, green.pre-commit run --all-files— green (helm-docs regenerated the chart README; the schema is unchanged, thekagentblock is open).WorkerPool kagent-defaultcarries the three labels, all four workers land on generation-6 nodes, every AgentTemplate Ready, twenty consecutive new Slack threads on the default agent with zeroincompatible FeatureSetand zerodispatch error— recorded on kagent WorkerPool spans AMD generations (m7a and c5ad): golden snapshots taken on gen 7 cannot restore on gen 5 workers, half of the default agent's cold turns fail after 60 s #457.