Skip to content

feat(monitoring): measure container /dev/shm, and alert before authentik's fills - #416

Merged
PBernaerts merged 1 commit into
mainfrom
feat/container-shm-metric
Aug 21, 2026
Merged

feat(monitoring): measure container /dev/shm, and alert before authentik's fills#416
PBernaerts merged 1 commit into
mainfrom
feat/container-shm-metric

Conversation

@PBernaerts

Copy link
Copy Markdown
Owner

Implements mon-2, which #414 put in place of 6f evidence item 6.

authentik's proxy outpost keeps every gate session in /dev/shm
(goauthentik/authentik #25248). Full means it stops writing sessions and
members stop getting through the apex, mtg and music gates, which after the
Authelia retirement has nothing left to fall back on.

Item 6 asked for two df readings a day apart. That measures the day somebody
looked and no day after it.

What this adds

  • docker_container_shm_size_bytes{name=...} and
    docker_container_shm_used_bytes{name=...} from the existing docker-stats
    collector.
  • AuthentikShmFilling, warning at 60% for 15m.

Two design choices worth the words

No new timer. The collector already loops running containers for the
anon-memory metric, so shm rides along. Adding a timer is a step this repo has
got wrong before, and each new one needs an explicit enable --now.

No docker exec. One exec per container per run is far more expensive and
fails outright on containers with no shell. A plain stat -f through
/proc/<pid>/root crosses into the container's mount namespace without
entering it.

The trap, because the obvious approach silently does nothing

The documented per-container path,
/var/lib/docker/containers/<id>/mounts/shm, does not exist on this host.
Docker 29 with the overlayfs snapshotter puts it elsewhere, the same layout
change that makes cAdvisor unusable here. Tried it first and got
No such file or directory for a container that plainly has a 512MB shm.

Verified the replacement against a known-good docker exec df:
131072 blocks x 4096 = 512MB, matching authentik-server's configured
shm_size, and ~1.04M used against the 1.1M read by hand.

Why the alert is not generic

Scoped to authentik-server rather than every container, because postgres uses
/dev/shm heavily and legitimately for parallel query workers, so a generic
rule would page on authentik-db doing its job. Threshold 60% against a
measured baseline of 1.1M of 512M, which is 0.2%.

The assertions were proven before being trusted

Same discipline the 6f gate exists to enforce. Pointed at an unresolvable
PROC_BASE:

FAIL: grafana shm size emitted
FAIL: grafana shm used emitted
FAIL: sabnzbd shm size emitted
FAIL: sabnzbd shm used emitted
FAIL: shm size is an integer   expected [int] got [notint]
FAIL: shm used is an integer   expected [int] got [notint]
FAIL: shm used never exceeds shm size
passed=8 failed=7

All seven new assertions fail, and the collector's original eight still pass.
That second half matters: a shm read failure degrades to missing samples rather
than taking the whole textfile down, which would otherwise lose every
container metric at once.

PROC_BASE is overridable purely so this is testable offline.

Checks

tests/test-docker-stats.sh 15 passed. bash tests/run.sh green, zero failing
suites. promtool check rules SUCCESS, 3 rules. shellcheck -S error clean.

Not done here, and it is the part that matters

The alert has not been fired on purpose yet. Per #414 and the #401 lesson,
mon-2 does not count until it has. That needs the deploy first, then filling
the tmpfs deliberately (or temporarily lowering the threshold) and watching the
message land. Until then this is instrumentation, not evidence.

…tik's fills

authentik's proxy outpost keeps every gate session in /dev/shm
(goauthentik/authentik #25248). Full means it stops writing sessions and
members stop getting through the apex, mtg and music gates, which after the
Authelia retirement has nothing left to fall back on.

The 6f evidence list asked for two df readings a day apart. That measures the
day somebody looked and no day after it, so this replaces it with a metric and
an alert: continuous, and it warns long before the tmpfs matters rather than
after somebody stopped taking readings.

No new timer. The existing docker-stats collector already loops running
containers for the anon-memory metric, and adding a timer is the step this repo
has got wrong before. It also does not use docker exec: one exec per container
per run is far more expensive and fails on containers with no shell. Instead a
plain statfs through /proc/<pid>/root crosses into the container's mount
namespace without entering it.

Worth recording, because the obvious approach silently does not work here: the
documented per-container path, /var/lib/docker/containers/<id>/mounts/shm, does
NOT exist on this host. Docker 29 with the overlayfs snapshotter puts it
elsewhere, the same layout change that makes cAdvisor unusable. Verified
against a known-good docker exec df: 131072 blocks x 4096 = 512MB, matching
authentik-server's configured shm_size.

The alert is scoped to authentik-server rather than every container, because
postgres uses /dev/shm heavily and legitimately for parallel query workers and
a generic rule would page on authentik-db doing its job. Threshold 60% against
a measured baseline of 1.1M of 512M.

PROC_BASE is overridable purely so the block is testable offline. The new
assertions were proven to fail before being trusted: pointed at an unresolvable
path, all seven fail while the collector's original eight still pass, which
also demonstrates a shm read failure degrades to missing samples instead of
taking the whole textfile down with it.
@PBernaerts
PBernaerts merged commit e33ad50 into main Aug 21, 2026
9 checks passed
@PBernaerts
PBernaerts deleted the feat/container-shm-metric branch August 21, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant