tracing: Jaeger/Opentracing are deprecated, use open-telemetry for instrumentation instead

[kavirajk]

Currently Loki is instrumented with opentracing/jaeger client libraries for tracing.

This instrumentation comes from weaveworks/common package and dskit's spanlogger package.

Those client libraries(opentracing, jaeger) are deprecated in the favor of opentelemetry client sdk. It's better to migrate.

I hope, It should be completely possible to migrate underlying dependencies without changing any API of those packages.

Example of using otel tracing client libraries for instrumentation in Go is here.

[mateuszdrab]

Looking forward to see this implemented, I realized that Grafana sends trace headers in w3c format to Loki which it can't support due to the feature not being implemented in the jaeger libraries. OCD was triggered as I wanted Grafana, Loki and Thanos to all support tracing.

[davidspek]

Just wanted to note that weaveworks/common isn't being used anymore and that grafana/dskit#385 is the PR needed to migrate to OTEL.

[mateuszdrab]

2 years later I'm still doing a nasty regex rewrite of traceparent to uber-trace-id in Loki's gateway nginx configuration.
Is there no better way to enable Grafana and Loki tracing?

[mateuszdrab]

For anyone interested how I do it, this is the snippet from my values

I had to resort to some tricky way of injecting proxy_set_header in each location block since the chart does not provide a locationSnippet option and it appears that the proxy_set_header from the http block won't be inherited into the location block if there is already a proxy_set_header there.

    gateway:
      ingress:
        annotations:
          nginx.ingress.kubernetes.io/limit-rps: "0"
          nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
          nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
          nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
          nginx.ingress.kubernetes.io/client-header-timeout: "600"
          nginx.ingress.kubernetes.io/client-body-timeout: "600"
      nginxConfig:
        # Hack to allow for custom nginx config from https://github.com/grafana/loki/issues/10521
        file: | 
          {{- tpl .Values.gateway.nginxConfig.custom_nginx_configfile_template . | indent 2 }} 
          
        custom_nginx_configfile_template: |

          load_module /usr/lib/nginx/modules/ngx_otel_module.so;

          {{- $nginxFile := include "loki.nginxFile" . }}
          {{- $traceHeader := "proxy_set_header uber-trace-id \"$${otel_trace_id}:$${otel_span_id}:$${otel_parent_id}:$${otel_parent_sampled}\";" }}
          {{- $patchedNginx := regexReplaceAll "(?m)^([ \t]*)proxy_pass" $nginxFile (printf "${1}%s\n${1}proxy_pass" $traceHeader) }}
          {{- $patchedNginx | indent 2 }}

        httpSnippet: |-
          client_body_timeout 600s;
          client_header_timeout 600s;
        
          otel_exporter {
            endpoint telemetry.monitoring-system.svc:4317;
          }

          otel_trace on;
          otel_trace_context propagate;
          otel_service_name loki-gateway;
      image:
        repository: nginx
        tag: alpine-otel

I then have the below extraEnvs for each of the components of Loki

- name: JAEGER_ENDPOINT
  value: "http://telemetry.monitoring-system.svc:14268/api/traces"
- name: JAEGER_SAMPLER_TYPE
  value: const
- name: JAEGER_SAMPLER_PARAM
  value: "1"
- name: JAEGER_REPORTER_LOG_SPANS
  value: "true"
- name: JAEGER_PROPAGATION
  value: w3c
- name: JAEGER_TAGS
  value: app=loki

This seems to work for me and allows Grafana queries to propagate the context properly into Loki