k8s-platform-lab

A Go web app that makes Kubernetes behaviour visible in a browser, plus the platform layer that runs it: Envoy Gateway for ingress, cert-manager for TLS, and Argo CD for GitOps. Built phase by phase against docs/build-guide.md.

The app is deliberately a toy — its point is that it can lie about its own health, burn CPU and memory on demand, and tell you which pod and node answered you. That makes readiness vs. liveness, endpoint churn, resource limits, and traffic routing observable instead of theoretical.

Status

Phase	What	State
1	App on minikube — Deployment, Service, ConfigMap, Secret, probes	done
2	Envoy Gateway — GatewayClass, Gateway, per-app HTTPRoutes	done
3	cert-manager — self-signed ClusterIssuer, HTTPS listener on 443	done
4	Argo CD — one Application per component	partial — see below
5	Karpenter on EKS	not started

Phase 4 has an Application per component (app-a, app-b, platform) with automated sync and self-heal. Still missing: the app-of-apps root Application and sync-wave annotations for ordering, so the CRDs/controllers still have to be installed by hand before Argo takes over.

Layout

main.go, handlers.go, burn.go    the app — stdlib only, no framework, no router
templates/, static/              HTML templates and vendored HTMX
Dockerfile                       multi-stage, distroless runtime

k8s/platform/                    GatewayClass, Gateway, ClusterIssuer  (shared infra)
k8s/app-a/                       Deployment, Service, ConfigMap, Secret, HTTPRoute
k8s/app-b/                       same, different colour and instance name
argocd/                          one Argo CD Application per folder above
helm/certManager.yaml            Helm values for cert-manager

docs/                            build guide and write-ups — see below

app-a and app-b run the same image. They differ only by ConfigMap (COLOR, INSTANCE_NAME) and replica count — app-a runs 3, app-b runs 1.

---

The app

Standard library only — no web framework, no router, no third-party modules. HTMX is vendored and served from the binary, so the app needs no outbound network access.

Run locally

go run .          # http://localhost:8080

Two visually distinct instances:

PORT=8080 INSTANCE_NAME=alpha COLOR="#2d6cdf" go run .
PORT=8081 INSTANCE_NAME=beta  COLOR="#d9534f" go run .

Configuration

Every variable has a default; bad values log a warning and fall back rather than failing the boot.

Variable	Default	Purpose
PORT	8080	Listen port
POD_NAME	local	Displayed in UI — set from the Downward API in-cluster
NODE_NAME	local	Displayed in UI — set from the Downward API in-cluster
POD_NAMESPACE	default	Displayed in UI
INSTANCE_NAME	dev	Logical instance label, e.g. alpha / beta
COLOR	#2d6cdf	CSS accent colour
VERSION	dev	Version string (overridden by a -ldflags build-time value)
API_KEY	""	Rendered masked — last 4 chars only, or (unset)
READY_DELAY_SECONDS	0	Report not-ready for this long after boot
SHUTDOWN_DELAY_SECONDS	0	On SIGTERM, report not-ready, wait this long, then drain

Endpoints

Endpoint	Purpose
GET /	HTML page; polls /whoami every 2s
GET /whoami	HTML fragment: pod, node, namespace, instance, version, masked key, uptime, state
GET /healthz	200 ok / 500 unhealthy — independent of readiness
GET /readyz	200 ready / 503 not ready
POST /toggle-ready	Flips ready, returns the fragment
POST /toggle-healthy	Flips healthy, returns the fragment
POST /burn	Starts load, returns the fragment immediately
GET /api/info	JSON version of the /whoami data

POST /burn takes a form body — cpu (default 1, capped at NumCPU*2), memory_mb (default 0, capped at 512), seconds (default 30, capped at 300):

curl -d "cpu=2&memory_mb=128&seconds=60" localhost:8080/burn

Memory is written to one byte per 4096 so the pages are actually resident and show up in docker stats / cgroup accounting; it is freed at the deadline.

The toggles only affect /healthz and /readyz. Every other route keeps returning 200 — the app is never actually broken, it just claims to be.

Container image

docker build -t k8sdemo:dev --build-arg VERSION=1.0.0 .
docker run --rm -p 8080:8080 -e INSTANCE_NAME=alpha k8sdemo:dev

Multi-arch:

docker buildx build --platform linux/amd64,linux/arm64 \
  --build-arg VERSION=1.0.0 -t k8sdemo:1.0.0 .

Runtime is gcr.io/distroless/static-debian12:nonroot: one file (the binary), no shell, runs as non-root, needs no writable filesystem.

The manifests currently deploy docker.io/tharshen2124/go-app:v2 with imagePullPolicy: IfNotPresent.

Shutdown behaviour

On SIGTERM/SIGINT the app sets ready = false immediately, sleeps SHUTDOWN_DELAY_SECONDS, then calls server.Shutdown with a 15s timeout and exits 0. Set SHUTDOWN_DELAY_SECONDS=15 to demonstrate the endpoint-propagation race: the pod stops advertising readiness well before it stops accepting connections.

---

The cluster

Target environment is minikube with the docker driver. That choice matters — see the hostname note below.

Prerequisites

minikube start --cpus=4 --memory=8192

# Gateway API CRDs — these define what a Gateway *is*
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/standard-install.yaml

# Envoy Gateway — the controller that actually reads the Gateway and routes
helm install envoy-gateway oci://docker.io/envoyproxy/gateway-helm \
  -n envoy-gateway-system --create-namespace

# cert-manager — Gateway API support is OFF by default, hence the values file
helm repo add jetstack https://charts.jetstack.io && helm repo update
helm install cert-manager jetstack/cert-manager \
  -n cert-manager --create-namespace -f helm/certManager.yaml

Leave this running in a second terminal for the whole session — without it the Gateway never gets an address and sits Pending forever:

minikube tunnel

Ingress

Resource	Name	Notes
GatewayClass	go-app-cluster-gateway-class	points at gateway.envoyproxy.io/gatewayclass-controller
Gateway	go-app-cluster-gateway-api	HTTP on 80, HTTPS on 443 (Terminate, secret go-app-tls)
ClusterIssuer	go-app-cluster-issuer	selfSigned — browsers will warn, that's expected
HTTPRoute	go-app-a-http-route	a.127.0.0.1.nip.io → go-app-a-service:80
HTTPRoute	go-app-b-http-route	b.127.0.0.1.nip.io → go-app-b-service:80

The Gateway's allowedRoutes.namespaces.from: Same means only routes in default can attach — fine, since everything lives there.

curl -k https://a.127.0.0.1.nip.io
curl -k https://b.127.0.0.1.nip.io

Why 127.0.0.1 and not the minikube node IP. With the docker driver there is no routable VM IP, so minikube tunnel binds the LoadBalancer's external IP to the host loopback. nip.io hostnames hardcode an IP into the DNS name, so a.192.168.49.2.nip.io resolves fine but nothing is listening there. Full write-up in docs/bugs.md.

GitOps

Three Argo CD Applications, all pointed at this repo, all with selfHeal: true:

Application	Path	Prune
go-app-a-argocd	k8s/app-a	yes
go-app-b-argocd	k8s/app-b	yes
go-app-platform-argocd	k8s/platform	no

Prune is deliberately off for the platform Application: a mistyped path there would otherwise delete the Gateway both apps route through.

kubectl apply -f argocd/

# UI
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d
kubectl port-forward svc/argocd-server -n argocd 8080:443

Note the Applications track HEAD of https://github.com/Tharshen2124/k8s-platform-lab.git — local changes have to be pushed before Argo will see them.

Things to poke at

# readiness: pod leaves the load balancer but keeps running
kubectl get endpointslices -w      # then click Toggle Ready

# liveness: container is restarted, pod name stays the same
kubectl get pods -w                # then click Toggle Healthy

# GitOps: Argo puts it back
kubectl delete deployment go-app-a-deployment

The difference between the first two is the whole point of the app.

---

Docs

Doc	What
docs/build-guide.md	The 5-phase plan this repo is built against
docs/health-checks-and-traffic.md	Liveness vs readiness tested against the live cluster; Envoy panic threshold, load concentration
docs/envoy-gateway-review.md	Why one HTTPRoute with two hostnames and two rules doesn't route hostname → backend
docs/bugs.md	General issues hit along the way, e.g. the minikube tunnel / nip.io mismatch

Known gaps

• No app-of-apps root Application, and no sync-wave annotations — CRDs and controllers must be installed manually before Argo CD can sync.
• Secrets are committed as plaintext stringData placeholders. Fine for a lab, not for anything real.
• Probes use Kubernetes defaults (10s period, 3 failures), so a toggle takes ~30s to take effect.
• Phase 5 (Karpenter on EKS) not started — it costs real money and needs its own session.