Allow Prometheus labels prefix configuration for kube_*_labels and kube_*_annotations metrics #2818
Description
Kube-State-Metrics CLI flags --metric-labels-allowlist and --metric-annotations-allowlist allows for labels and annotations of Kubernetes resources to be extracted and added to kube_*_labels and kube_*_annotations Prometheus metrics labels. Example of such exported metric:
kind: Deployment
metadata:
name: some-app
labels:
team: some-team
...
---
kube_deployment_labels{deployment="some-app", label_team="some-team"}
Each of the configured labels and annotations turned to Prometheus labels are prefixed with the label_ and annotation_ string.
Each Kubernetes resource supported by KSM exporting such metrics calls the internal/store/utils.go:createPrometheusLabelKeysValues method with an hardcoded prefix string parameter set to either "label" or "annotation". See search results
See example implementation:
What would you like to be added:
I would like to be able to prevent these prefixes from being emitted by KSM.
- kube_deployment_labels{deployment="some-app", label_team="some-team"}
+ kube_deployment_labels{deployment="some-app", team="some-team"}Why is this needed:
These labels generate complexity and duplications in our Prometheus rules. Some metrics get a team label and sometimes need to be correlated to label_team from KSM. This also impacts alert routing. I would like to standardize my Prometheus labels around the team label, not label_team.
I also don't really see the point of these prefixes either, especially since they are only added on an already explicit metric.
Describe the solution you'd like:
Introducing new CLI flags --metric-labels-prefix and --metric-annotations-prefix. These flags would keep a default value (annotation and label) to avoid introducing breaking changes, but could be explicitly nulled (--metric-labels-prefix="") to edit the exported Prometheus labels of this feature.