WorkloadPolicy might show the status as "FAILED" even though the policy is enforced on the workload as expected.

[venkateshjayagopal]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- OS: Ubuntu
- Architecture: amd
- Cluster: RKE2

Issue Description

• Expected behavior: WorkloadPolicy should show the actual status which is "PASSED" even when one of the node on the cluster which had a runtime-enforcer agent up and running is not reachable.
• Current behavior: When one of the runtime-enforcer agent is not reachable, then all workloadPolicy status is moved to "Failed" state.
• Steps to reproduce:
• Deploy runtime-enforcer on a multi node cluster.
• Create application pods on the cluster, probably 4 to 5 deployments.
• Bring down one of the node on the cluster abruptly.
• Check the workloadPolicy status for deployments which are running on the other working nodes.

ubuntu@master-node-01 15:12:49:~$ kubectl get wp
NAME                         MODE      STATUS   ACTIVE VIOLATIONS   TOTAL VIOLATIONS
deploy-ubuntu-deployment     protect   Failed   10                  283
deploy-ubuntu-deployment-1   protect   Failed   2                   76
deploy-ubuntu-deployment-2   protect   Failed   2                   32
deploy-ubuntu-deployment-3   protect   Failed   2                   14232
deploy-ubuntu-deployment-4   protect   Failed   2                   15560
deploy-ubuntu-deployment-5   protect   Failed   5                   14578
ubuntu@master-node-01 15:39:12:~$ kubectl get wp deploy-ubuntu-deployment-5 -o yaml
kind: WorkloadPolicy
metadata:
  creationTimestamp: "2026-07-20T17:21:28Z"
  generation: 2
  labels:
    ---/promoted-from: deploy-ubuntu-deployment-5
  name: deploy-ubuntu-deployment-5
  namespace: default
  resourceVersion: "43648450"
  uid: 5c96727a-c49a-4855-a6ea-7fdb878cfbce
spec:
  mode: protect
  rulesByContainer:
    ubuntu:
      executables:
        allowed:
        - /usr/bin/bash
        - /usr/lib/cargo/bin/coreutils/sleep
        - /usr/lib/cargo/bin/coreutils/ls
status:
  activeViolationCount: 5
  failedNodes: 1
  nodesWithIssues:
    susesecurity-worker-02:
      code: Missing
      message: failed to get policies status
  observedGeneration: 2
  phase: Failed
  successfulNodes: 2
  totalNodes: 3
  violationCount: 14578
  violations:
  - action: protect
    containerName: ubuntu
    executablePath: /usr/lib/cargo/bin/coreutils/date
    id: 14577
    nodeName: susesecurity-worker-01
    podName: ubuntu-deployment-5-6874d75d6d-8mg4t
    timestamp: "2026-07-21T15:12:42Z"
    workloadKind: Deployment
    workloadName: ubuntu-deployment-5
  - action: protect
    containerName: ubuntu
    executablePath: /usr/lib/cargo/bin/coreutils/whoami
    id: 14574
    nodeName: susesecurity-worker-01
    podName: ubuntu-deployment-5-6874d75d6d-8mg4t
    timestamp: "2026-07-21T15:11:18Z"
    workloadKind: Deployment
    workloadName: ubuntu-deployment-5
  - action: protect
    containerName: ubuntu
    executablePath: /usr/bin/ps
    id: 0
    nodeName: susesecurity-worker-01
    podName: ubuntu-deployment-5-6874d75d6d-8mg4t
    timestamp: "2026-07-21T15:06:50Z"
    workloadKind: Deployment
    workloadName: ubuntu-deployment-5
  - action: protect
    containerName: ubuntu
    executablePath: /usr/lib/cargo/bin/coreutils/dircolors
    id: 14572
    nodeName: susesecurity-worker-01
    podName: ubuntu-deployment-5-6874d75d6d-8mg4t
    timestamp: "2026-07-21T15:06:26Z"
    workloadKind: Deployment
    workloadName: ubuntu-deployment-5
  - action: protect
    containerName: ubuntu
    executablePath: /runc
    id: 1
    nodeName: susesecurity-worker-01
    podName: ubuntu-deployment-5-6874d75d6d-8mg4t
    timestamp: "2026-07-21T15:06:26Z"
    workloadKind: Deployment
    workloadName: ubuntu-deployment-5
ubuntu@master-node-01 15:39:17:~$

ubuntu@master-node-01 15:15:59:~$ kubectl get pods -o wide
NAME                                   READY   STATUS    RESTARTS   AGE   IP            NODE                     NOMINATED NODE   READINESS GATES
ubuntu-deployment-5-6874d75d6d-8mg4t   1/1     Running   0          21h   10.42.1.252   susesecurity-worker-01   <none>           <none>
ubuntu@master-node-01 15:18:04:~$ kubectl get pods -n runtime-enforcer -o wide
NAME                                                   READY   STATUS        RESTARTS     AGE    IP            NODE                     NOMINATED NODE   READINESS GATES
runtime-enforcer-agent-6zdpz                           1/1     Running       0            23h    10.42.1.244   susesecurity-worker-01   <none>           <none>
runtime-enforcer-agent-d594f                           1/1     Running       0            23h    10.42.3.236   susesecurity-worker-02   <none>           <none>
runtime-enforcer-agent-xfdkz                           1/1     Running       1 (8h ago)   23h    10.42.0.101   master-node-01           <none>           <none>
runtime-enforcer-controller-manager-599786f94f-8hkv2   1/1     Terminating   1 (8h ago)   22h    10.42.3.238   susesecurity-worker-02   <none>           <none>
runtime-enforcer-controller-manager-599786f94f-zfnz9   1/1     Running       0            168m   10.42.1.2     susesecurity-worker-01   <none>           <none>
runtime-enforcer-otel-collector-7c5dff7d85-2z5rw       1/1     Running       0            168m   10.42.1.3     susesecurity-worker-01   <none>           <none>
runtime-enforcer-otel-collector-7c5dff7d85-bsz67       1/1     Terminating   0            23h    10.42.3.235   susesecurity-worker-02   <none>           <none>
ubuntu@master-node-01 15:21:30:~$ kubectl get nodes -o wide
NAME                     STATUS     ROLES                AGE    VERSION          INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
master-node-01           Ready      control-plane,etcd   129d   v1.35.2+rke2r1   10.115.50.60   <none>        Ubuntu 22.04.5 LTS   5.15.0-185-generic   containerd://2.1.5-k3s1
susesecurity-worker-01   Ready      <none>               129d   v1.35.2+rke2r1   10.115.54.56   <none>        Ubuntu 22.04.5 LTS   5.15.0-176-generic   containerd://2.1.5-k3s1
susesecurity-worker-02   NotReady   <none>               129d   v1.35.2+rke2r1   10.115.54.54   <none>        Ubuntu 22.04.5 LTS   5.15.0-176-generic   containerd://2.1.5-k3s1
ubuntu@master-node-01 15:21:38:~$
ubuntu@master-node-01 15:02:54:~$ kubectl exec -ti ubuntu-deployment-5-6874d75d6d-8mg4t -- bash
bash: /usr/bin/dircolors: Operation not permitted
root@ubuntu-deployment-5-6874d75d6d-8mg4t:/# pwd
/
root@ubuntu-deployment-5-6874d75d6d-8mg4t:/# ps
bash: /usr/bin/ps: Operation not permitted
root@ubuntu-deployment-5-6874d75d6d-8mg4t:/# ps
bash: /usr/bin/ps: Operation not permitted
root@ubuntu-deployment-5-6874d75d6d-8mg4t:/# pwd
/
root@ubuntu-deployment-5-6874d75d6d-8mg4t:/# whoami
bash: /usr/bin/whoami: Operation not permitted
root@ubuntu-deployment-5-6874d75d6d-8mg4t:/# whoami
bash: /usr/bin/whoami: Operation not permitted
root@ubuntu-deployment-5-6874d75d6d-8mg4t:/# pwd
/
root@ubuntu-deployment-5-6874d75d6d-8mg4t:/# pwd
/
root@ubuntu-deployment-5-6874d75d6d-8mg4t:/# whoami

[venkateshjayagopal]

We could probably use the keyword like "Tainted" instead of "Failed". When seeing "Failed" user might think the policies are not enforced on the workload when we are doing so.