Skip to content

Add karma as a dashboard over the alert bus - #507

Open
ChrisonSimtian wants to merge 1 commit into
mainfrom
feat/karma
Open

Add karma as a dashboard over the alert bus#507
ChrisonSimtian wants to merge 1 commit into
mainfrom
feat/karma

Conversation

@ChrisonSimtian

Copy link
Copy Markdown
Collaborator

A nicer read-only view onto Alertmanager. One stateless container, no secrets, no data dir. Deployed and verified.

Alertmanager's own UI at :9093 stays the authority — its Status tab shows the routing tree and inhibition rules as Alertmanager actually parsed them, which is what you want when an alert didn't arrive and you need to know which route caught it. karma is the better viewer on the same data: grouped cards, dark mode, readable on a phone.

Where to look now

:8080 karma — the nice one. Read-only for alerts, but it can create silences.
:9093 Alertmanager — the authority; parsed config and routing tree
:3000 Grafana → Alerting — set the selector to Alertmanager, not "Grafana built-in", or the page looks empty (the rules are native Prometheus rules, so Grafana's own list is legitimately zero)

Two decisions worth reviewing

  • No UserNS, and that's per-image rather than house style. karma's image declares User: 0, so the default rootless mapping sends container-root to the podman user — same as snmp_exporter and pulse. Alertmanager needs keep-id because it runs as 65534, outside this LXC's uid map; copying either neighbour blindly gets you a crun: setgroups: Invalid argument crash loop, which is how I found that out on Monitor the UniFi stack, with Alertmanager as the alert bus #501. Checked the image.
  • No FILTERS_DEFAULT on purpose. karma already leads with active alerts, and pinning @state=active would hide silenced ones — precisely the alert you most want to notice, because a forgotten silence is invisible failure rather than noise.

It reaches Alertmanager by container name over the shared quadlet network, so it doesn't depend on the published :9093 and keeps working if that's ever unpublished.

Caught before shipping

Two env vars were wrong on my first pass and would have been accepted and silently ignored — karma takes PORT (not LISTEN_PORT) and FILTERS_DEFAULT (not KARMA_DEFAULT_FILTERS). Verified against karma's configuration docs rather than assumed. Same failure family as the Environment= whitespace splitting on #500: config that looks right in the file and isn't what the process gets.

Verified

  • karma.service active; UI returns 200; /healthPong
  • Connected to the bus: Upstream version 0.34.0 alertmanager=homelab
  • End-to-end — posted a synthetic alert to /api/v2/alerts, karma went groups=0groups=1, then back to 0 after resolving it. Alertmanager confirms 0 active alerts now.
  • 27/27 shapes valid, Apply summary — 1 applied, 0 failed

Note

The smoke test routed through the home-assistant receiver, so there'll be a stray "karma dashboard smoke test" notification (and its resolve) in HA's notification list. Harmless, dismissable.

Alertmanager's own UI stays the authority — its Status tab shows the routing
tree and inhibition rules as Alertmanager actually parsed them, which is what
you want when an alert did not arrive. karma is the nicer viewer on the same
data: grouped cards, dark mode, readable on a phone.

Stateless and uid 0, so no Volume and no UserNS. That last part is per-image,
not house style: alertmanager needs keep-id because it runs as 65534, outside
this LXC's uid map. Checked karma's image rather than copying either
neighbour.

Reaches alertmanager by container name over the shared network, so it does not
depend on the published :9093 and keeps working if that is ever unpublished.

No FILTERS_DEFAULT on purpose. karma already leads with active alerts, and
pinning `@state=active` would hide SILENCED ones — the alert you most want to
notice, because a forgotten silence is invisible failure rather than noise.

Two env vars were wrong on the first pass and would have been accepted and
ignored: the port is PORT (not LISTEN_PORT) and filters are FILTERS_DEFAULT.
Verified against karma's own docs before shipping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ChrisonSimtian ChrisonSimtian added the enhancement New feature or request label Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant