Drop filename label from grafana logs (worker-nodes logs)

[samidbb]

Running log analysis shows that we can reduce number of log streams by dropping labels (reducing cardinality)

Label Name              Unique Values  Found In Streams
pod                     14446          51201
unit                    3775           10728
job                     3756           89362
service_name            3595           89362
container               3585           51201
filename                1873           26083
namespace               151            51322
app_kubernetes_io_name  99             7737
instance                83             38161
__time_shard__          49             55674
__stream_shard__        3              1687
stream                  2              51201
cluster                 1              89362
flags                   1              51201

This PR will remove filename as a label from log lines produced by the worker-nodes job.

The result will be less log streams produced because of lower cardinality and avoid issues hitting limit on number of streams in our Grafana instances which in turn caused loss in logs as mentioned in this incident https://dfdsit.statuspage.io/incidents/cq7qmzhv4j4v?utm_source=slack

The effect in this PR tested using sandbox and Grafana test instance https://dfdsqa.grafana.net/explore?schemaVersion=1&panes=%7B%22lmf%22:%7B%22datasource%22:%22grafanacloud-logs%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bcluster%3D%5C%22cargo%5C%22,%20job%3D%5C%22integrations%2Fkubernetes%2Fworker-node%5C%22%7D%20%7C%3D%20%60%60%20%7C%20unpack%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22grafanacloud-logs%22%7D,%22editorMode%22:%22code%22,%22direction%22:%22backward%22%7D%5D,%22range%22:%7B%22from%22:%22now-15m%22,%22to%22:%22now%22%7D,%22panelsState%22:%7B%22logs%22:%7B%22sortOrder%22:%22Descending%22,%22visualisationType%22:%22table%22,%22displayedFields%22:%5B%22Time%22,%22detected_level%22,%22___LOG_LINE_BODY___%22,%22job%22,%22instance%22,%22filename%22%5D%7D%7D,%22compact%22:false%7D%7D

[wcarlsen]

This is awesome. Be aware that moving labels into to the log payload breaks things like structured logging among other things and impact the way you would write queries. I think we should consider if filename information should be kept at all?

[samidbb]

@wcarlsen I don't use filename as a relevant information myself so I am for removing this label

[wcarlsen]

@wcarlsen I don't use filename as a relevant information myself so I am for removing this label

Great, me neither and we on plus side avoid changing things on the users side of things

[wcarlsen]

This is the diff of the rendition

The resources is probably because this branch was created before I merged my resource changes.

[samidbb]

a lot has slipped into the PR because of the fix to the Update CRDs and render helm templates workflow permission 🫠

[wcarlsen]

a lot has slipped into the PR because of the fix to the Update CRDs and render helm templates workflow permission 🫠

I think you can manage to figure this out