Skip to content

Dedicated Substrate worker nodes for node-loss proofs (substrateNodes: N) #189

Description

@teemow

Problem

The lab's kind cluster has one node, so a proof that removes the node holding a paused actor's local snapshot (giantswarm/giantswarm#37795, giantswarm/substrate#34) cannot be run against a stock lab. Building one by hand costs two bring-ups worth of gotchas: the Dex CA has to exist before kind create (the apiserver's oidc-ca-file is mounted from certs/), kind keeps the control-plane NoSchedule taint on multi-node clusters so up times out on Dex, and the platform's pods have to be kept off the nodes that will be killed — substrate.atelet.tolerations plus kagent.substrateWorkerPool.template.{nodeSelector,tolerations} in a values overlay — so that stopping a node removes exactly what a spot interruption removes: that node's ateom workers and its atelet.

Proposed solution

A substrateNodes: N knob in agentlab.yaml (default 0 = today's single node). With N > 0 the rendered kind config carries N role: worker nodes labelled agentlab.giantswarm.io/substrate-worker=true and tainted substrate-workers=true:NoSchedule (a JoinConfiguration patch), the control-plane node keeps no taint, and the lab's rendered values pin the WorkerPool template and the atelet DaemonSet to those nodes with the matching toleration. agentlab certs stays part of up before the cluster is created, as today. Node loss is then docker stop <cluster>-worker && kubectl delete node <cluster>-worker.

Acceptance criteria

  • agentlab up with substrateNodes: 2 yields a Ready lab with the ateom workers and atelets on the two worker nodes and every other pod on the control-plane node; platform-test passes.
  • Stopping and deleting one worker node leaves the platform up; Substrate deregisters that node's workers and the pool rebuilds them on the remaining node(s).
  • The default (0) renders the single-node cluster unchanged.
  • README/docs: the knob and the node-loss recipe.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions