Skip to content

feat(meta): the WorkerPool's CPU generation pin — karpenter.k8s.aws/instance-generation next to the vendor — documented, render-guarded and verified by make verify-workerpool (#457) - #469

Merged
teemow merged 1 commit into
mainfrom
workerpool-cpu-generation-pin
Sep 15, 2026
Merged

teemow merged 1 commit into
mainfrom
workerpool-cpu-generation-pin

Conversation

@teemow

@teemow teemow commented Sep 15, 2026

Copy link
Copy Markdown
Member

Closes #457 (sub-issue of giantswarm/giantswarm#37787). The chart half; the fleet template (shared-configs) gains the matching knob and gazelle sets it in its own PRs.

Problem

One Substrate WorkerPool runs one CPU feature set: an actor's golden snapshot is a gVisor checkpoint that restores only on a host whose CPU offers every feature it recorded. #429 pinned the vendor (karpenter.k8s.aws/instance-cpu-manufacturer: amd) and #430 recorded the caveat that a newer generation's snapshot does not restore on an older one of the same vendor. That caveat hit gazelle on 2026-09-14: Karpenter had two kagent-default workers on an m7a.2xlarge (generation 7, AVX-512) and two on c5ad.xlarge (generation 5); the goldens of sre-agent and test-agent were taken on the m7a workers at 16:24Z, and a new thread placed on a c5ad worker at 19:29Z failed

starting container: … failed to load kernel: incompatible FeatureSet: missing features: map[monitor pcid x2apic invpcid avx512f avx512dq avx512cd avx512bw avx512vl …]

twice (30 s each), then SendStreamingMessage Internal after 60 s, dispatch error: actor "ai-…" request timed out in klaus-gateway, "the turn failed; please try again" in Slack. The next morning the pool was c6a, c5a ×2, m6a (generations 6, 5, 5, 6) and every cold turn restored — because the goldens had been re-taken on a gen-5 worker at 06:39Z, not by construction. Every spot replacement can change the mix.

Change

  • Docs (values.yaml, the chart README, the root README's Substrate section, UPGRADE.md): the generation is the second label of the pin, next to the vendor — karpenter.k8s.aws/instance-generation: "6", quoted. Which generation: one that is one CPU model across its families — for AMD on AWS 6 (c6a/m6a/r6a, EPYC 7R13 Milan) or 7 (c7a/m7a/r7a, EPYC 9R14 Genoa); 5 mixes Naples (m5a/r5a, EPYC 7571) with Rome (c5a/c5ad, EPYC 7R32) and is no feature-set pin — and one the installation's goldens restore on (an older generation's golden restores on a newer one, never the reverse). And the capacity trade-off: the pin narrows the spot pool to the generation's families × the NodePool's sizes; read kubectl get nodes -L node.kubernetes.io/instance-type,karpenter.k8s.aws/instance-generation and the NodePool's requirements first; three families is the floor. The family label stays as the narrower option.
  • Guard (agent-platform.validateWorkerPool): a kagent.substrateWorkerPool.template.nodeSelector value that is not a string fails the render naming the key. The kagent chart forwards the template verbatim (toYaml) into WorkerPool.spec.template, whose nodeSelector is map[string]string; a bare 6 rendered, passed this chart's open kagent schema and would have failed only when helm-controller applied the kagent release — on every installation carrying it. No rendered object changes for an installation that sets no generation.
  • make verify-workerpool (in CI, tests/verify-workerpool.py): the default forwards the architecture alone; an installation's three-label pin reaches the kagent release verbatim, every value a string, nothing else in the map; the unquoted generation fails the render naming the key; the kagent chart the range resolves to (0.11.0-gs.15 today) renders the pin unchanged into the one WorkerPool kagent-default's spec.template.nodeSelector, next to the forwarded resources. The lab (kind) has no Karpenter labels — a pinned pool would simply not schedule there — so this render is where the pin is proven; the live proof is gazelle's.
  • The CHANGELOG entry, under Unreleased / Added.

Not here: a restore that fails with incompatible FeatureSet is retried on the same worker; placing it on another worker, and the durable fix (workers advertise their feature set, snapshots record it, the scheduler requires a superset — giantswarm/substrate#29, giantswarm/giantswarm#37742 row 12), are the Substrate line's.

The placement decision for gazelle (recommended: generation 6)

The current goldens: sre-agent (Swarmgeist's default agent), test-agent, issue-tracker, kyverno-policy-writer and factory-analyst were checkpointed 06:39Z on the two c5a.xlarge workers (generation 5, Rome); argus on an m6a.xlarge (generation 6). The NodePool gazelle-karpenter admits 4/8/16/32 vCPU, nitro, spot and on-demand, amd64, any family but t2/t3/t3a, across eu-central-1a/b/c.

pin instance types the NodePool admits spot pools (× 3 zones) one CPU model? current goldens restore?
generation 5 c5a, c5ad, m5a, m5ad, r5a, r5ad × 4 sizes = 24 72 no — Naples (m5a/r5a) next to Rome (c5a/c5ad) only where Rome's features exist
generation 6 c6a, m6a, r6a × 4 sizes = 12 36 yes — EPYC 7R13 Milan yes (gen-5 and gen-6 goldens ⊂ Milan; the 2026-09-15 re-test restored gen-5 goldens on gen-6 workers three times)
generation 7 c7a, m7a, r7a × 4 sizes = 12 36 yes — EPYC 9R14 Genoa yes, at a higher spot price for features no agent uses
family c6a c6a × 4 sizes = 4 12 yes yes — but one family is a thin spot pool

Generation 6 is the lowest homogeneous AMD generation, three families clear the floor, and no golden has to be re-taken for compatibility: the next Harness change re-takes them on gen-6 workers by itself. Set in gazelle's installations/gazelle/config.yaml.patch (agentPlatform.workerPoolCpuGeneration: "6") once the shared-configs knob is merged.

Verification

…nstance-generation next to the vendor — documented, render-guarded (a non-string nodeSelector value fails the render naming the key) and verified by make verify-workerpool (#457)

One pool runs one CPU feature set; #429 pinned the vendor and left the
generation open. On gazelle (2026-09-14) goldens taken on m7a (gen 7)
workers failed to restore on c5ad (gen 5) workers with `incompatible
FeatureSet: missing features: … avx512f …`, twice per cold turn, then the
turn failed. The pin is a second label in the same map, set per
installation by the fleet template; the value is a string ("6"), so the
meta chart refuses a bare number at the render instead of letting
helm-controller fail the kagent release on apply. verify-workerpool asserts
the default, the verbatim forwarding, the guard and the rendered WorkerPool
of the kagent chart the range resolves to.
@teemow
teemow requested a review from a team as a code owner September 15, 2026 08:30
@circleci-architect

Copy link
Copy Markdown

Published Helm chart agent-platform

4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd
Chart agent-platform
Version 4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd
Digest sha256:35a3d48c405631fb90b3d6593bf1c5a272409469a75ece783c322443bd7d44a9
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd

Posted by architect-orb · build 8048 · commit 4266ecd · updated in place on every push

@circleci-architect

Copy link
Copy Markdown

Published Helm chart agent-platform-connectivity

4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd
Chart agent-platform-connectivity
Version 4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd
Digest sha256:4d8a0fced51ca3e4b947a008ab9662dfd4d472d612e52b15d6e6403891eb46db
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.17.1-dev.workerpool--eration-pin.2026-09-15.08-28-22.h4266ecd

Posted by architect-orb · build 8050 · commit 4266ecd · updated in place on every push

@teemow
teemow merged commit d0f3931 into main Sep 15, 2026
13 checks passed
@teemow
teemow deleted the workerpool-cpu-generation-pin branch September 15, 2026 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant