Demo workflow

Intro part

We welcome the player and ask him whether he/she is good and Kubernetes troubleshooting. We tell them that there is a simple Kubernetes cluster that runs a Postgresql database that contains user credentials.

We received reports recently that the credentials may have been leaked. More and more users are complaining about “hacked” accounts.

Your task is to verify the system from security perspective.

Troubleshooting part

We give the user a laptop, with kubectl and k9s installed. They have admin access. They will probably check the pods, the logs, but there is nothing suspicious.

We give them the following hints:

1. No one can access the database except for us.
2. The system is highly secured. An external firewall is in place. Nobody can access the database from the outside.
3. We have recently reinstalled the host operating system. There can be no virus there.
4. Before the incident, we performed a Postgresql version upgrade.

The user will unable to find the security issue.

Whisker part

We will present the user the Whisker dashboard. The user will play with it. With our help, they will find a suspicious outgoing connection:

NetworkPolicy part (Optional, TODO)

If the player is interested, we can help them to provision a network policy that prevents the communication to the public internet.

Preparations

These preparations help to give a demo in an unreliable-network environment.

Firewall

Disable the firewall on the host.

Dummy network interface

Create a dummy network interface with the name whisker.

sudo ip link add dev whisker type dummy
sudo ip addr add 3.14.137.65/28 dev whisker

Postgres image

Building the image

You can build the image with the following command.

docker build 2025/postgresql/ -t postgresql/postgresql:16

Save the image

Save the image.

docker save postgresql/postgresql:16 -o postgres_16.tar

Postgres manifest files

You can generate the Postgres deployment manifest with the following command.

helm template postgres \
  --repo https://groundhog2k.github.io/helm-charts \
  --namespace postgres \
  --create-namespace \
  -f postgres-values.yaml \
  --version 1.5.7 > postgres.yaml

Calico resources

Manifest files

Download the Calico manifest files:

curl -LO https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/operator-crds.yaml
curl -LO https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/tigera-operator.yaml
curl -LO https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/custom-resources.yaml

Then update custom-resources.yaml so that the spec.calicoNetwork.ipPools.[0].cidr field has the value 192.168.101.0 in the Installation resource.

Container images

VERSION="v3.30.3"
IMAGES=("typha" "kube-controllers" "node" "csi")

for IMAGE in ${IMAGES[@]}
do
  echo "Saving ${IMAGE}:${VERSION}"
  docker pull "docker.io/calico/${IMAGE}:${VERSION}"
  docker save "docker.io/calico/${IMAGE}:${VERSION}" -o "${IMAGE}.tar"
done

VERSION="v1.38.6"
echo "Saving operator:${VERSION}"
docker pull "quay.io/tigera/operator:${VERSION}"
docker save "quay.io/tigera/operator:${VERSION}" -o operator.tar

Create demo environment manually

Start the netcat server

sudo nc -kl 3.14.137.65 137 | pv > /dev/null

Kind config file

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
networking:
  disableDefaultCNI: true
  podSubnet: 192.168.101.0/24

Create the kind cluster

kind create cluster --name whisker-the-game --config kind-config.yaml

Then check the cluster state:

kubectl cluster-info

Upload the container images

IMAGES=("typha" "kube-controllers" "node" "csi" "operator" "postgres_16")

for IMAGE in ${IMAGES[@]}
do
  echo "Uploading ${IMAGE}"
  kind load image-archive "${IMAGE}.tar" \
       --name whisker-the-game           \
       --nodes whisker-the-game-control-plane,whisker-the-game-worker,whisker-the-game-worker2
done

Install Calico CRDs

kubectl create -f operator-crds.yaml

Install the Tigera Operator

kubectl create -f tigera-operator.yaml

Install the Calico resources

kubectl create ns calico-system
kubectl create -f custom-resources.yaml

Install postgres YAML manifests

kubectl create ns postgres --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f postgres.yaml

Cleanup

Delete kind cluster

kind delete cluster --name whisker-the-game