Error in updating neuvector-monitor to 2.8.5

[DuxVictorum]

We upgraded 'neuvector-core' from 2.8.4 to 2.8.5 just fine, but when we then subsequently tried to upgrade 'neuvector-monitor' from 2.8.4 to 2.8.5 the neuvector-prometheus-exporter-pod-* logs immediately showed the following:

│ Login to controller ...
│ Start exporter server ...                                                                                                                                                                                                                  
│ Register collector ...                                                                                                                                                                                                                     
│ Traceback (most recent call last):                                                                                                                                                                                                         
│   File "/usr/local/bin/nv_exporter.py", line 650, in <module>                                                                                                                                                                              
│     REGISTRY.register(COLLECTOR)                                                                                                                                                                                                           
│     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^                                                                                                                                                                                                           
│   File "/.venv/lib64/python3.13/site-packages/prometheus_client/registry.py", line 40, in register                                                                                                                                         
│     names = self._get_names(collector)                                                                                                                                                                                                     
│   File "/.venv/lib64/python3.13/site-packages/prometheus_client/registry.py", line 80, in _get_names                                                                                                                                       
│     for metric in desc_func():                                                                                                                                                                                                             
│                   ~~~~~~~~~^^                                                                                                                                                                                                              
│   File "/usr/local/bin/nv_exporter.py", line 455, in collect                                                                                                                                                                               
│     iwnslist.append(c['workload_domain'])                                                                                                                                                                                                  
│                     ~^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                   
│ KeyError: 'workload_domain'            

I have not run into this issue before. I can see some rearrangement of some potentially relevant code in nv_exporter in the bump from 1.0.2 to 1.0.3 of prometheus-exporter, but I can't figure out why I'm getting that KeyError. When I redeploy the helm chart v2.8.4, it starts working again with no errors.

We are running our deployments in EKS. Everything is managed by helm.

Any help appreciated!

[holyspectral]

Hi @DuxVictorum, thanks for reporting. The issue has been addressed in neuvector/prometheus-exporter#48 Are you using the helm repo located in https://neuvector.github.io/neuvector-helm ?

[DuxVictorum]

@holyspectral Yes, our code specifies that exact helm repo URL. I tried redeploying and am getting the same error, but I suppose that's because while you've merged the change into main it's not yet part of the helm chart v2.8.5 (which contains prometheus-exporter v1.0.3), which is what helm is referencing. Is there a way to pull in your patch fix now, or do I need to wait until v2.8.6?

[holyspectral]

@DuxVictorum yes that's my understanding too. Let me check with team to see if we can have a fix released soon in helm charts. In the mean time, could you see if adding --set exporter.image.tag=1.0.4 in monitor chart would work?

[DuxVictorum]

@holyspectral I am content with waiting until the next update drops. By the way, I have to give you all a HUGE SHOUT OUT because you guys/gals acknowledged the problem and put out a fix in like a couple hours. It takes us days sometimes to get traction with some support teams where we pay out the nose for enterprise support, yet you folks are over here solving problems and being awesome. Thanks again.

[holyspectral]

No problem at all and sorry for the inconvenience it caused. For a simple fix like this, GitHub issue works best because it comes to our attention immediately. On the other hand, going through support will usually get a better/quick outcome because it gets prioritized, especially for more complicated cases.

Sorry for the inconvenience it caused again!