Link the `trace_id` across retries

[p-j-smith]

Motivation

We currently have a single trace that covers the end-to-end processing of a single message. However, if at any point the processing fails, when we retry the same study it will be given a new trace_id. It would be useful to be able to link traces across retries.

Pitch

We should ensure the publish_message span has an OTel span link back to an original attempt (if it exists). We should store the original traceparent on the corresponding Image in the PIXL db, so all retries can be linked to the first attempt.

• Add a nullable traceparent column to Image
• Store the Image.traceparent value immediately after first publishing a message for a given study (this can be captured by injecting the current context into a dict)
• In the CLI, check if Image.traceparent is not null. If there's a value, include it in the message.
• Update PixlProducer.publish to add a Link to the traceparent (if it exists)

Alternatives

Link each new retry to the most recent retry, rather than the first attempt. Updating the Image.traceparent value on each retry would be simple enough, but I think it would make viewing all retries in the Grafana UI more difficult (as there's no single trace_id they can all be linked to).

Additional context

The OTel Python cookbook's "Manually setting span context" gives an example of injecting/extracting the traceparent and using it to link between spans

[linear-code]

PIX-72