Metrics container security context prevents access to WireGuard metrics

[Blarc]

The metrics container appears to be unable to access the necessary data to provide metrics.

When I k port-forward svc/wireguard-metrics-svc 9586:9586 and open http://localhost:9586/metrics I get only:

# HELP wireguard_sent_bytes_total Bytes sent to the peer
# TYPE wireguard_sent_bytes_total counter

# HELP wireguard_received_bytes_total Bytes received from the peer
# TYPE wireguard_received_bytes_total counter

# HELP wireguard_latest_handshake_seconds UNIX timestamp seconds of the last handshake
# TYPE wireguard_latest_handshake_seconds gauge

So I set --verbose "true" arg on metric container and got:

[2025-01-09T19:31:46Z TRACE prometheus_wireguard_exporter] wg show all dump stderr == Unable to access interface wg0: Operation not permitted

After a bit more of debugging, I figured out that the metric and agent containers have different security context configuration and run with different users:

# metrics
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    add:
    - NET_ADMIN
    drop:
    - ALL
  readOnlyRootFilesystem: true
  runAsGroup: 65534
  runAsNonRoot: true
  runAsUser: 65534
  
# agent
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    add:
    - NET_ADMIN
  readOnlyRootFilesystem: true

After manually editing the deployment and removing the following part from metrics container configuration, the metrics start working:

readOnlyRootFilesystem: true
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534

This issue seems related with #170.
(I have first opened a discussion about this: #222.)

[kryoseu]

Hey, looking at the manifest I don't see the configuration you mentioned:

readOnlyRootFilesystem: true
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534

Both containers from the wireguard-controller-manager deployment have:

readOnlyRootFilesystem: true
runAsGroup: 65534
runAsNonRoot: false
runAsUser: 65534

After manually editing the deployment and removing the following part from metrics container configuration, the metrics start working

Can you point me to what exactly you manually edited?

[Blarc]

Hey @kryoseu,

it's been a while since I opened the issue, so I am not 100% sure, but I believe I had to remove:

readOnlyRootFilesystem: true
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534

from the metrics deployment configuration.

[kryoseu]

Thanks for the response @Blarc !

I ended up downgrading wireguard as I couldn't find any deployment configured with runAsNonRoot: true in the latest manifest v2.7.0/release.yaml.

The controller-manager deployment is defined as:

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
    - ALL
  readOnlyRootFilesystem: true
  runAsGroup: 65534
  runAsNonRoot: false
  runAsUser: 65534
securityContext:
  fsGroup: 65534
  runAsNonRoot: true

The second securityContext with runAsNonRoot: true seems to be at the deployment level, even though the containers have:

runAsGroup: 65534
runAsNonRoot: false
runAsUser: 65534

So maybe changing the deployment-level one might do the trick, will give that a try.