feat(monitoring): measure container /dev/shm, and alert before authentik's fills - #416
Merged
Conversation
…tik's fills authentik's proxy outpost keeps every gate session in /dev/shm (goauthentik/authentik #25248). Full means it stops writing sessions and members stop getting through the apex, mtg and music gates, which after the Authelia retirement has nothing left to fall back on. The 6f evidence list asked for two df readings a day apart. That measures the day somebody looked and no day after it, so this replaces it with a metric and an alert: continuous, and it warns long before the tmpfs matters rather than after somebody stopped taking readings. No new timer. The existing docker-stats collector already loops running containers for the anon-memory metric, and adding a timer is the step this repo has got wrong before. It also does not use docker exec: one exec per container per run is far more expensive and fails on containers with no shell. Instead a plain statfs through /proc/<pid>/root crosses into the container's mount namespace without entering it. Worth recording, because the obvious approach silently does not work here: the documented per-container path, /var/lib/docker/containers/<id>/mounts/shm, does NOT exist on this host. Docker 29 with the overlayfs snapshotter puts it elsewhere, the same layout change that makes cAdvisor unusable. Verified against a known-good docker exec df: 131072 blocks x 4096 = 512MB, matching authentik-server's configured shm_size. The alert is scoped to authentik-server rather than every container, because postgres uses /dev/shm heavily and legitimately for parallel query workers and a generic rule would page on authentik-db doing its job. Threshold 60% against a measured baseline of 1.1M of 512M. PROC_BASE is overridable purely so the block is testable offline. The new assertions were proven to fail before being trusted: pointed at an unresolvable path, all seven fail while the collector's original eight still pass, which also demonstrates a shm read failure degrades to missing samples instead of taking the whole textfile down with it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
mon-2, which #414 put in place of 6f evidence item 6.authentik's proxy outpost keeps every gate session in
/dev/shm(goauthentik/authentik #25248). Full means it stops writing sessions and
members stop getting through the apex,
mtgandmusicgates, which after theAuthelia retirement has nothing left to fall back on.
Item 6 asked for two
dfreadings a day apart. That measures the day somebodylooked and no day after it.
What this adds
docker_container_shm_size_bytes{name=...}anddocker_container_shm_used_bytes{name=...}from the existing docker-statscollector.
AuthentikShmFilling, warning at 60% for 15m.Two design choices worth the words
No new timer. The collector already loops running containers for the
anon-memory metric, so shm rides along. Adding a timer is a step this repo has
got wrong before, and each new one needs an explicit
enable --now.No
docker exec. One exec per container per run is far more expensive andfails outright on containers with no shell. A plain
stat -fthrough/proc/<pid>/rootcrosses into the container's mount namespace withoutentering it.
The trap, because the obvious approach silently does nothing
The documented per-container path,
/var/lib/docker/containers/<id>/mounts/shm, does not exist on this host.Docker 29 with the overlayfs snapshotter puts it elsewhere, the same layout
change that makes cAdvisor unusable here. Tried it first and got
No such file or directoryfor a container that plainly has a 512MB shm.Verified the replacement against a known-good
docker exec df:131072 blocks x 4096 = 512MB, matchingauthentik-server's configuredshm_size, and~1.04Mused against the1.1Mread by hand.Why the alert is not generic
Scoped to
authentik-serverrather than every container, because postgres uses/dev/shmheavily and legitimately for parallel query workers, so a genericrule would page on
authentik-dbdoing its job. Threshold 60% against ameasured baseline of 1.1M of 512M, which is 0.2%.
The assertions were proven before being trusted
Same discipline the 6f gate exists to enforce. Pointed at an unresolvable
PROC_BASE:All seven new assertions fail, and the collector's original eight still pass.
That second half matters: a shm read failure degrades to missing samples rather
than taking the whole textfile down, which would otherwise lose every
container metric at once.
PROC_BASEis overridable purely so this is testable offline.Checks
tests/test-docker-stats.sh15 passed.bash tests/run.shgreen, zero failingsuites.
promtool check rulesSUCCESS, 3 rules.shellcheck -S errorclean.Not done here, and it is the part that matters
The alert has not been fired on purpose yet. Per #414 and the #401 lesson,
mon-2does not count until it has. That needs the deploy first, then fillingthe tmpfs deliberately (or temporarily lowering the threshold) and watching the
message land. Until then this is instrumentation, not evidence.