Problem
Several metrics use a per-job UUID as a label value, e.g.:
AddInstanceRetryTotal.WithLabelValues(job.UUID.String()).Inc() // pkg/starter/starter.go:167
AddInstanceBackoffDuration.WithLabelValues(job.UUID.String()).Observe(...) // pkg/starter/starter.go:168
and similar in pkg/runner/runner_delete.go:123-124. Every job creates new time series that are never deleted (DeleteLabelValues is never called), so a long-running myshoes with steady job traffic grows Prometheus cardinality without bound — this degrades both the local registry and the scraping Prometheus.
Suggested fix
- Drop the UUID label: a plain counter (
retry_total) and histogram (backoff_duration_seconds) answer the same operational questions (how often do we retry, how long do we back off) without per-job series.
- If per-job visibility is needed, log it (the UUID is already in the log lines) rather than labeling metrics with it.
Problem
Several metrics use a per-job UUID as a label value, e.g.:
and similar in
pkg/runner/runner_delete.go:123-124. Every job creates new time series that are never deleted (DeleteLabelValuesis never called), so a long-running myshoes with steady job traffic grows Prometheus cardinality without bound — this degrades both the local registry and the scraping Prometheus.Suggested fix
retry_total) and histogram (backoff_duration_seconds) answer the same operational questions (how often do we retry, how long do we back off) without per-job series.