diff --git a/.github/workflows/helm-e2e.yaml b/.github/workflows/helm-e2e.yaml index ec7646477c..23b50e768a 100644 --- a/.github/workflows/helm-e2e.yaml +++ b/.github/workflows/helm-e2e.yaml @@ -33,6 +33,10 @@ jobs: go-version-file: go.mod - name: Setup Helm uses: azure/setup-helm@v4 + - name: Test Helm chart + run: | + helm plugin install https://github.com/helm-unittest/helm-unittest.git --version 1.0.3 --verify=false + make helm-test - name: Cache micro-VM assets uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 with: diff --git a/FORK.md b/FORK.md index 37479c677c..8c122a7d31 100644 --- a/FORK.md +++ b/FORK.md @@ -40,11 +40,14 @@ Everything on `giantswarm` that is not in the pin (`git log v0.0.26..giantswarm` | Grant atelet cluster-wide read access to sandbox configs | atelet's sandbox-asset prewarm degraded on the second test cluster without the RBAC ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 10) | `74b45f9e` (`git cherry-pick -x a7505e9c`) | [kagent-dev/substrate#33](https://github.com/kagent-dev/substrate/pull/33), merged 2026-09-08, not in v0.0.26 — falls away at the re-pin onto the first tag that contains it | | Let an actor's egress through while it resumes (ateom arms tunneled egress before the first container starts, atenet admits `RESUMING` actors, both hops log a refusal) | an actor whose workload fetches what it needs to become ready — kagent's Go ADK and Claude harnesses materialise git skills before readyz — never got its golden snapshot: atunnel dropped the fetch (`Broken pipe`), atenet would have refused a non-`RUNNING` actor, nothing was logged ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) rows 8 and 13; acceptance test `agentlab skills-test`, [agentlab#137](https://github.com/giantswarm/agentlab/issues/137)) | [#4](https://github.com/giantswarm/substrate/pull/4) (`181762747bb2`; first published as `0.0.27-dev.giantswarm.2026-09-10.22-37-39.h1817627`) | to file: the upstream-shaped patch is branch [`upstream/atenet-egress-during-resume`](https://github.com/giantswarm/substrate/tree/upstream/atenet-egress-during-resume) here (`3a95d7cf`, on the mirror `main`); a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it. **Complete with the egress dataplane this pull request pins:** the chart's `images.agentgateway` is the check in the request path (the egress config carries the `substrateEgress` policy and no `ext_proc`, so atenet's handler is not consulted); kagent-dev's `c0f5597c7cb8` (a pre-merge build of agentgateway#3237) authorized every CONNECT against ate-api itself — UID, then `RUNNING` — and refused the golden boot (`atunnel failed to open egress tunnel … 403 Forbidden: actor is not running`, agentlab 2026-09-11); upstream agentgateway v1.5.0 has no such check at all (its `substrateEgress` derives the actor from the SPIFFE id and checks nothing else — `agentlab skills-test` green on both halves with it swapped into `atenet-egress`, 2026-09-11, the run that proved the Substrate half). The line now runs the agentgateway line's `v1.5.1-gs.2`, which keeps #3237's UID and state check and admits `RESUMING` (giantswarm/agentgateway-upstream#4; upstream-facing branch [`upstream/substrate-egress-resuming`](https://github.com/giantswarm/agentgateway-upstream/tree/upstream/substrate-egress-resuming), #37742 row 8). Acceptance test of the combined fix: `agentlab skills-test` on the first build of this merge, recorded on agentlab#137 | | Declare the egress actor authorization as a frontend policy (`frontendPolicies.substrateEgress` in the atenet-egress config, the route-level policy removed) and pin `images.agentgateway` to the agentgateway line's `v1.5.1-gs.2` | the line's dataplane carries agentgateway#3237, which moved the CONNECT-time actor check from a route policy to a frontend policy; with v0.0.26's route-level shape the gs.2 dataplane refuses its config (`unknown field substrateEgress`, atenet-egress CrashLoopBackOff, agentlab 2026-09-11) and the pin moves with the config because `v1.5.1-gs.1` (v1.5.0) rejects the frontend-level field and the pre-merge build `c0f5597c7cb8` the route-level one only | the `chart: authorize the egress actor as a frontend policy at CONNECT time` commit of pull request #9 | [kagent-dev/substrate#28](https://github.com/kagent-dev/substrate/pull/28) (merged 2026-09-10, on `main`) makes the same move for its `v0.0.0-alpha.988ac151` dataplane under the name agentgateway#3318 gave the policy, `substrateEgressActorResolution`; falls away at the re-pin onto the first tag containing #28 once the agentgateway line carries #3318 (until then the field name differs — resolve by keeping the line's). The e2e install manifests (`manifests/ate-install/components/agentgateway`) still run kagent-dev's `c0f5597c7cb8` with the route-level config, self-consistent; #28 moved them too | +| Read ate-api-server's PostgreSQL connection string from a Secret (`postgres.connectionStringSecretRef`; the `ate-api-server-envvars` ConfigMap then carries only the schema) | meta chart 4.0 puts Substrate's control-plane database on the platform's CNPG cluster and hands ate-api-server the DSN through a Secret, never a ConfigMap ([giantswarm/agent-platform#342](https://github.com/giantswarm/agent-platform/issues/342); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 24) | `42c0c6b8` (`git cherry-pick -x 1872249e`) and `392841e2` (`git cherry-pick -x 41097da7`, the `helm plugin install --verify=false` of the same pull request), [#8](https://github.com/giantswarm/substrate/pull/8) | [kagent-dev/substrate#32](https://github.com/kagent-dev/substrate/pull/32), open (2026-09-04), not ours — falls away at the re-pin onto the first release that carries it | +| The atelet DaemonSet takes `nodeSelector`, `tolerations` and `affinity` (`atelet.{nodeSelector,tolerations,affinity}`, empty by default) | the platform pins atelet to worker nodes / node pools; the chart had no scheduling knob ([giantswarm/agent-platform#342](https://github.com/giantswarm/agent-platform/issues/342); [#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 23) | `d9717d78`, [#8](https://github.com/giantswarm/substrate/pull/8) | to file: the upstream-shaped patch is branch [`upstream/atelet-scheduling`](https://github.com/giantswarm/substrate/tree/upstream/atelet-scheduling) here (`b34c1690`, on the mirror `main`); [kagent-dev/substrate#16](https://github.com/kagent-dev/substrate/pull/16) touches the same knob (`atelet.nodeSelector`, no tolerations or affinity) inside a fork-wide 92-file pull request that has conflicted since July — align with the maintainers there; a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it | | Fork infrastructure: this file, the README pointer, `CODEOWNERS`, `.github/workflows/publish.yaml`, `.github/workflows/sync-upstream.yaml`, `.trivyignore`, and the branch triggers of `pr-workflow.yaml`, `helm-e2e.yaml`, `govulncheck.yaml` (`main` → `giantswarm`, govulncheck also on pull requests) | the line's CI, publishing and sync | the `giantswarm` branch history | not for upstream | -One patch changes Substrate's behaviour ahead of upstream — egress for an actor while it resumes, without -which no skill-carrying agent of the platform boots; it is written for upstream and leaves at the first -release that carries it. Everything else is what upstream has already merged. Giant Swarm specific wiring +Two patches change Substrate ahead of upstream — egress for an actor while it resumes, without which no +skill-carrying agent of the platform boots, and the atelet scheduling knobs; both are written for upstream +and leave at the first release that carries them. The Postgres Secret patch is upstream's own open pull +request. Everything else is what upstream has already merged. Giant Swarm specific wiring lives elsewhere: the CA/JWT pool bootstrap (`kubectl-ate admin make-ca-pool`/`make-jwt-pool` and the `ate-api-authentication` ConfigMap) is created by [agentlab](https://github.com/giantswarm/agentlab) and by meta chart 4.0; the `WorkerPool` the platform's Harnesses run on comes with the kagent chart @@ -159,7 +162,7 @@ request and weekly. | Consumer | Where the pin lives | Selects | |---|---|---| | [agentlab](https://github.com/giantswarm/agentlab) | `internal/lab/substrate.go` (`substrateChartsRepo`, `substrateImageRegistry`, `substrateVersion`) | an exact dev version or release; installs `substrate-crds` + `substrate` and preloads the worker image | -| agent-platform meta chart, branch `poc/kagent-main` | `helm/agent-platform/values.yaml` `kagent.substrateWorkerPool.workerImage` | the `ateom-gvisor` image at an exact version (the `WorkerPool` the kagent chart renders) | +| agent-platform meta chart 4.0 (`components.substrate` / `components.substrate-crds`, the `substrate:` values block; [giantswarm/agent-platform#342](https://github.com/giantswarm/agent-platform/issues/342)) | `helm/agent-platform/values.yaml`: the two components' version pins and `kagent.substrateWorkerPool.workerImage` | the `substrate-crds` + `substrate` charts at an exact dev version or release and the `ateom-gvisor` image at the same version (the `WorkerPool` the kagent chart renders) | | [giantswarm/kagent-upstream](https://github.com/giantswarm/kagent-upstream) | `Makefile` `SUBSTRATE_REPO ?= oci://ghcr.io/giantswarm/substrate/helm`, `SUBSTRATE_VERSION` | the `substrate`/`substrate-crds` chart dependencies of the kagent charts (off in the platform, which installs Substrate as cluster infrastructure) | ## Assets that are not images diff --git a/Makefile b/Makefile index 9165abfe63..363476d16d 100644 --- a/Makefile +++ b/Makefile @@ -111,6 +111,10 @@ clean: helm-template: @./hack/render-manifests.sh +.PHONY: helm-test +helm-test: + @helm unittest charts/substrate + # Verify that manifests/ate-install/ matches the chart output. Used in CI. .PHONY: verify-helm-template verify-helm-template: diff --git a/charts/substrate/README.md b/charts/substrate/README.md index 0640c9e9a7..567b272e51 100644 --- a/charts/substrate/README.md +++ b/charts/substrate/README.md @@ -37,9 +37,13 @@ See `values.yaml` for the full set; the important keys: |-----|---------|-------| | `postgres.enabled` | `true` | Deploy the bundled PostgreSQL instance | | `postgres.connectionString` | `""` (in-cluster) | Override to use external PostgreSQL | +| `postgres.connectionStringSecretRef` | disabled | Read an external PostgreSQL connection string from a Secret; its name defaults to `-postgres-connection` when enabled | | `postgres.schema` | `public` | Store the Substrate tables in this PostgreSQL schema | | `postgres.storageSize` | `1Gi` | In-cluster PostgreSQL PVC size | | `rustfs.enabled` | `true` | Deploy an in-cluster S3-compatible RustFS bucket for snapshots | | `atelet.storageBackend` | `s3` | Default snapshot backend, wired to RustFS when `rustfs.enabled=true` | | `atelet.gcpAuthForImagePulls` | `false` | Enable only when using GCP registry auth | +| `atelet.nodeSelector` | `{}` | Nodes the atelet DaemonSet runs on; empty = every schedulable node | +| `atelet.tolerations` | `[]` | Tolerations of the atelet pods | +| `atelet.affinity` | `{}` | Affinity of the atelet pods | | `otel.endpoint` | `""` | Set to an OTLP endpoint to export traces/metrics | diff --git a/charts/substrate/templates/_helpers.tpl b/charts/substrate/templates/_helpers.tpl index 32ae087336..a7bdba8ce4 100644 --- a/charts/substrate/templates/_helpers.tpl +++ b/charts/substrate/templates/_helpers.tpl @@ -78,6 +78,12 @@ Plaintext HTTP URL that clients use to reach atenet-router. {{- printf "http://%s.%s.svc:80" (include "substrate.fullname" (list "atenet-router" .)) .Release.Namespace -}} {{- end -}} +{{/* PostgreSQL connection Secret, when configured. */}} +{{- define "substrate.postgres.connectionStringSecretEnabled" -}} +{{- $ref := .Values.postgres.connectionStringSecretRef | default dict -}} +{{- if or (get $ref "enabled") (get $ref "name") -}}true{{- end -}} +{{- end -}} + {{/* Build an image reference for a substrate component binary. diff --git a/charts/substrate/templates/ate-api-server-envvars.yaml b/charts/substrate/templates/ate-api-server-envvars.yaml index ca76ae3ef8..0d18a942af 100644 --- a/charts/substrate/templates/ate-api-server-envvars.yaml +++ b/charts/substrate/templates/ate-api-server-envvars.yaml @@ -14,8 +14,12 @@ See the License for the specific language governing permissions and limitations under the License. */}} -{{- if and (not .Values.postgres.enabled) (empty .Values.postgres.connectionString) }} -{{- fail "postgres.connectionString is required when postgres.enabled=false" }} +{{- $secretEnabled := include "substrate.postgres.connectionStringSecretEnabled" . -}} +{{- if and .Values.postgres.connectionString $secretEnabled }} +{{- fail "postgres.connectionString and postgres.connectionStringSecretRef are mutually exclusive" }} +{{- end }} +{{- if and (not .Values.postgres.enabled) (empty .Values.postgres.connectionString) (not $secretEnabled) }} +{{- fail "postgres.connectionString or postgres.connectionStringSecretRef is required when postgres.enabled=false" }} {{- end }} apiVersion: v1 kind: ConfigMap @@ -23,5 +27,7 @@ metadata: name: {{ .Values.ateApiServerEnvVarsConfigMap }} namespace: {{ .Release.Namespace }} data: + {{- if not $secretEnabled }} ATE_API_POSTGRES_CONNECTION_STRING: {{ .Values.postgres.connectionString | default (printf "postgresql://postgres@%s.%s.svc:5432/atepg?sslmode=verify-full&sslrootcert=/run/servicedns.podcert.ate.dev/trust-bundle.pem&sslcert=/run/podidentity.podcert.ate.dev/credential-bundle.pem&sslkey=/run/podidentity.podcert.ate.dev/credential-bundle.pem" (include "substrate.fullname" (list "postgres" .)) .Release.Namespace) | quote }} + {{- end }} ATE_API_POSTGRES_SCHEMA: {{ .Values.postgres.schema | quote }} diff --git a/charts/substrate/templates/ate-api-server.yaml b/charts/substrate/templates/ate-api-server.yaml index b6d1960901..f37ffe8aee 100644 --- a/charts/substrate/templates/ate-api-server.yaml +++ b/charts/substrate/templates/ate-api-server.yaml @@ -94,6 +94,14 @@ spec: - "--drain-delay=13s" - "--drain-timeout=15s" env: +{{- $connectionStringSecretRef := .Values.postgres.connectionStringSecretRef }} +{{- if include "substrate.postgres.connectionStringSecretEnabled" . }} + - name: ATE_API_POSTGRES_CONNECTION_STRING + valueFrom: + secretKeyRef: + name: {{ get $connectionStringSecretRef "name" | default (include "substrate.fullname" (list "postgres-connection" .)) | quote }} + key: {{ get $connectionStringSecretRef "key" | default "connectionString" | quote }} +{{- end }} - name: POD_NAME valueFrom: fieldRef: diff --git a/charts/substrate/templates/atelet.yaml b/charts/substrate/templates/atelet.yaml index e2b4b52fb1..c57e2333ae 100644 --- a/charts/substrate/templates/atelet.yaml +++ b/charts/substrate/templates/atelet.yaml @@ -101,6 +101,18 @@ spec: prometheus.io/port: "9090" spec: serviceAccountName: {{ include "substrate.fullname" (list "atelet" .) }} +{{- with .Values.atelet.nodeSelector }} + nodeSelector: +{{ toYaml . | indent 8 }} +{{- end }} +{{- with .Values.atelet.tolerations }} + tolerations: +{{ toYaml . | indent 6 }} +{{- end }} +{{- with .Values.atelet.affinity }} + affinity: +{{ toYaml . | indent 8 }} +{{- end }} containers: - name: atelet image: {{ include "substrate.componentImage" (list "atelet" .) }} diff --git a/charts/substrate/tests/atelet_test.yaml b/charts/substrate/tests/atelet_test.yaml new file mode 100644 index 0000000000..96843813a9 --- /dev/null +++ b/charts/substrate/tests/atelet_test.yaml @@ -0,0 +1,79 @@ +# Copyright 2026 The Agent Substrate Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +suite: atelet scheduling +templates: +- atelet.yaml +tests: +- it: schedules atelet on every node by default + documentSelector: + path: kind + value: DaemonSet + asserts: + - notExists: + path: spec.template.spec.nodeSelector + - notExists: + path: spec.template.spec.tolerations + - notExists: + path: spec.template.spec.affinity + +- it: renders the configured nodeSelector and tolerations + documentSelector: + path: kind + value: DaemonSet + set: + atelet: + nodeSelector: + node-role.kubernetes.io/worker: "" + tolerations: + - key: dedicated + operator: Equal + value: sandboxes + effect: NoSchedule + asserts: + - equal: + path: spec.template.spec.nodeSelector + value: + node-role.kubernetes.io/worker: "" + - equal: + path: spec.template.spec.tolerations + value: + - key: dedicated + operator: Equal + value: sandboxes + effect: NoSchedule + - notExists: + path: spec.template.spec.affinity + +- it: renders the configured affinity + documentSelector: + path: kind + value: DaemonSet + set: + atelet: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/arch + operator: In + values: [amd64] + asserts: + - equal: + path: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0] + value: + key: kubernetes.io/arch + operator: In + values: [amd64] diff --git a/charts/substrate/tests/postgres_test.yaml b/charts/substrate/tests/postgres_test.yaml new file mode 100644 index 0000000000..4edb859f97 --- /dev/null +++ b/charts/substrate/tests/postgres_test.yaml @@ -0,0 +1,125 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +suite: PostgreSQL configuration +templates: +- ate-api-server-envvars.yaml +- ate-api-server.yaml +- postgres.yaml +tests: +- it: configures the bundled PostgreSQL database by default + template: ate-api-server-envvars.yaml + asserts: + - isNotNull: + path: data.ATE_API_POSTGRES_CONNECTION_STRING + - equal: + path: data.ATE_API_POSTGRES_SCHEMA + value: public + +- it: configures an external PostgreSQL connection string and schema + template: ate-api-server-envvars.yaml + set: + postgres: + enabled: false + connectionString: postgresql://user:pass@database:5432/agents + schema: substrate + asserts: + - equal: + path: data.ATE_API_POSTGRES_CONNECTION_STRING + value: postgresql://user:pass@database:5432/agents + - equal: + path: data.ATE_API_POSTGRES_SCHEMA + value: substrate + +- it: omits bundled PostgreSQL for an external connection + template: postgres.yaml + set: + postgres: + enabled: false + connectionString: postgresql://user:pass@database:5432/agents + asserts: + - hasDocuments: + count: 0 + +- it: reads an explicitly named connection Secret + template: ate-api-server.yaml + documentIndex: 3 + set: + postgres: + enabled: false + connectionStringSecretRef: + name: shared-database + key: url + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: ATE_API_POSTGRES_CONNECTION_STRING + valueFrom: + secretKeyRef: + name: shared-database + key: url + +- it: derives a release-scoped Secret name when enabled + template: ate-api-server.yaml + documentIndex: 3 + set: + postgres: + enabled: false + connectionStringSecretRef: + enabled: true + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: ATE_API_POSTGRES_CONNECTION_STRING + valueFrom: + secretKeyRef: + name: RELEASE-NAME-postgres-connection + key: connectionString + +- it: keeps credentials out of the ConfigMap in Secret mode + template: ate-api-server-envvars.yaml + set: + postgres: + enabled: false + schema: substrate + connectionStringSecretRef: + name: shared-database + asserts: + - notExists: + path: data.ATE_API_POSTGRES_CONNECTION_STRING + - equal: + path: data.ATE_API_POSTGRES_SCHEMA + value: substrate + +- it: rejects disabling bundled PostgreSQL without a connection + template: ate-api-server-envvars.yaml + set: + postgres: + enabled: false + asserts: + - failedTemplate: + errorMessage: postgres.connectionString or postgres.connectionStringSecretRef is required when postgres.enabled=false + +- it: rejects configuring both a literal connection and a Secret + template: ate-api-server-envvars.yaml + set: + postgres: + connectionString: postgresql://user:pass@database:5432/agents + connectionStringSecretRef: + name: shared-database + asserts: + - failedTemplate: + errorMessage: postgres.connectionString and postgres.connectionStringSecretRef are mutually exclusive diff --git a/charts/substrate/values.yaml b/charts/substrate/values.yaml index 287ac9c6ed..d6bd327d9c 100644 --- a/charts/substrate/values.yaml +++ b/charts/substrate/values.yaml @@ -27,6 +27,12 @@ postgres: enabled: true storageSize: 1Gi connectionString: "" + # Read the connection string from a Secret instead of values/ConfigMap. + connectionStringSecretRef: + # When enabled with no name, use the release-scoped postgres-connection name. + enabled: false + name: "" + key: connectionString schema: public resources: requests: @@ -53,6 +59,11 @@ atelet: storageBackend: s3 extraArgs: [] extraEnv: [] + # Where atelet runs: the DaemonSet's nodeSelector, tolerations and affinity. + # Empty = every schedulable node. + nodeSelector: {} + tolerations: [] + affinity: {} # Name of a ConfigMap in the release namespace that supplies per-environment # overrides for ate-api-server (ATE_API_POSTGRES_CONNECTION_STRING, ...).