Migrate Mimir tracing to opentelemetry

[pracucci]

Is your feature request related to a problem? Please describe.

Mimir tracing is based on jaeger client and opentracing, which are deprecated.

Describe the solution you'd like

• Migrate to opentelemetry.
• Being consistent between Mimir, Loki and Tempo (hopefully sharing the same code)
  • Tempo introduced opentelemetry tracer support (optional) in http://github.com/grafana/tempo/pull/842
  • Loki hasn't introduced opentelemetry yet

Describe alternatives you've considered

Nothing else considered.

[bboreham]

• [Trace] Mimimum changes in dskit in order to migrate to opentelemetry dskit#385

[colega]

I am currently working on this issue.

Tempo has been fully migrated to opentelemetry libraries, making a breaking change in the exporter config, switching directly to OTel.

Loki (and obviously, Mimir) still use Opentracing. Pyroscope uses OpenTracing, I think, I didn't check, it doesn't matter (since Loki still does).

Both Loki and Mimir depend on dskit, and @bboreham pointed out in the previous comment that there were efforts to migrate dskit to OTel, however I don't have enough bandwidth to migrate both Mimir and Loki (and Pyroscope?) to OTel, so I'm going to go for a solution that would allow us a seamless transition.

Step 1: Decouple SpanLogger usages from OpenTracing.

One of the biggest offenders in the transition is spanlogger package, which is tightly tied to OpenTracing, embedding an opentracing.Span which is accessed by both Mimir and Loki (and some more private repos I found out later). I'm solving that by not embedding opentracing.Span in spanlogger.SpanLogger, but by exposing the methods used instead. I haven't put much effort into making the nicest API out there, honestly, I think we can do that once we fully transition Grafana products to OTel.

• : SpanLogger: don't embed Span dskit#679

Step 2: Make dskit compatible with both OpenTracing and OTel

Next step would be to add OTel support like in grafana/dskit#385 but without removing the OpenTracing support.

The approach would be the following:

• Wherever we take a span from the context, we're going to try to find an OTel one and an OpenTracing one, instantiate a wrapper and propagate the span with the library of the parent span.

• In the places where we start a new span like in server middlewares, we're going to configure opentracing if global opentracing.Tracer is registered (i.e., the app has configured it in main), otherwise we'll do OTel (without checking whether it's configured or not).

• In the places where span is propagated on to a request:

  • For HTTP, we're going to use the previous approach of using only one middleware
  • For gRPC, I'm still undecided, but I'll first try to just use both propagators (OpenTracing and OTel), assuming taht only one of them will find a Span to propagate.

• The explicit spanlogger.New and opentracing.StartSpanFromContext will have to be migrated by downstream libraries to OTel when they want to migrate to OTel.

  • spanlogger.NewOTel will create a new OTel span, we could avoid this but Tempo used a tracer per package which I want to simulate (because I assume they know what they're doing). We could have a single tracer for the spanlogger package and choose on the fly, but downstream projects still need to migrate their opentracing.StartSpanFromContext calls to tracer.Start, so it's just one extra call to migrate and we'll do everything properly.

• PR: Support for OTel tracing dskit#681

Step 3: Test that PR in Mimir, make sure that nothing about OpenTracing breaks.

Without migrating to OTel, we first want to check that we didn't break dskit for the existing OpenTracing, so we'll update Mimir to that dskit version and it should work as a drop-in replacement without any changes.

• Done

Step 4: Migrate Mimir to OTel using the Jaeger exporter

Next we'll remove all the opentracing-related calls from Mimir, and we'll use the OTel tracer provider implementation with a Jaeger exporter. The PR that Bryan mentioned above has all the code for that so we'd still use Jaeger config and it should work as a drop-in replacement for the users while all the code would use OTel.

This PR should also have an assertion that would check in runtime that opentracing.IsGlobalTracerRegistered is false, to make sure that no downstream project initializes opentracing, since Mimir code isn't supposed to support it. In reality, I think it could still work if the downstream project uses the same otel-opentracing custom bridge that Mimir has right now, but Mimir will also remove that package.

• Draft: Use OTel library from tracing while keeping Jaeger exporter config #11249

Step 5: Add OTel exporter

Since all the tracing is OTel, we should be able to switch the Jaeger exporter to OTel expoter. We could use env vars to figure out what the user wants.

• PR Pending