Repository navigation
dcgm exporter conflict with self-installed dcgm exporter #525
Description
Activity
Thank you for reporting your feedback to us!
The internal ticket has been created: https://warthogs.atlassian.net/browse/SOLENG-2106.
This message was autogenerated
Hi @gabrielcocenza , I remember you handle this issue before?
Reacted by Gabriel CocenzaThere are some possible ways to handle this:
- Do as you said:
- stop the snap installed dcgm exporter from hardware observer
- keep the relation between hardware observer with grafana-agent-k8s for the scrape to work.
- enable hostNetwork in helm installed dcgm exporter for scraping.
This solution was discussed some time ago, but didn't got traction. See the docs
An alternative to not use
hostNetwork(preferable because of security) is to use the nvidia-gpu-operator-tf using theprometheus-scrape-target-k8sI guess in both ways, we need to have a config to not install the dcgm snap when the helm deployment because they cannot work together.
@jneo8 Maybe we can add a new possible value for dcgm-snap-channel as
disableor something similar to not install or remove the dcgm snap
Description
we are using helm installed gpu-operator which installed another set of dcgm exporter in the kubernetes cluster.
the dcgm exporter installed by hardware observer is conflicting with helm installed dcgm exporter with hostNetwork.
We need to have a way to scrape the helm installed dcgm exporter properly.
Charm revision
755
Juju version
3.6.21
Cloud
Kubernetes
Expected behaviour
COS is able to scraping helm installed dcgm exporter.
Reproduce / Test
When we want to scrape the metrics from dcgm exporter, we need to
Notes & References
No response