feat: OpenTelemetry distributed tracing across API and worker

[dotunv]

Overview

Add request-level tracing using OpenTelemetry that propagates trace context across the API, worker, and Stellar Horizon calls for end-to-end visibility.

Motivation

When a transfer fails mid-settlement, there is no way to trace the full lifecycle of that request across the API handler, queue dispatch, worker processing, and Horizon submission. Distributed tracing enables operators to pinpoint exactly where failures occur.

Proposed Design

1. Instrument the HTTP server with OpenTelemetry for automatic span creation
2. Propagate trace context into Asynq job metadata so worker jobs inherit parent traces
3. Instrument Horizon client calls to create child spans
4. Export traces to configurable OTEL_EXPORTER_ENDPOINT (supports Jaeger, Grafana Tempo, etc.)
5. Add trace_id to all log entries for correlation
6. Make tracing opt-in via OTEL_ENABLED=true env var (disabled by default)

Acceptance Criteria

• HTTP requests create traces with unique trace IDs
• Worker jobs inherit trace context from the dispatching API request
• Horizon client calls appear as child spans within the request trace
• Traces are exported to configured OTEL endpoint
• Trace IDs appear in structured log entries
• Tracing is disabled by default (no overhead when OTEL_ENABLED is not set)
• No performance impact when tracing is disabled
• Tests verify span creation and context propagation

[Stephan-Thomas]

@Stephan-Thomas has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

hi, i would like to work on this.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Stephan-Thomas to this issue.

[johnmic0132]

@johnmic0132 has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hi Maintainer! I’d love to take on this issue and get it done efficiently. Kindly assign it to me.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @johnmic0132 to this issue.

[Daniel202251]

@Daniel202251 has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

I’m a smart contract and full-stack developer who turns complex ideas into secure, production-ready blockchain code.
With strong skills in Rust and Solidity, I build systems that are fast, safe, and built to last.
My experience with the Stellar SDK lets me ship real-world Web3 solutions that actually move value

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Daniel202251 to this issue.

[therealbibson]

@therealbibson has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hi Maintainer! I’m interested in contributing to this task and can deliver it on time with a clean implementation.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @therealbibson to this issue.

[chiomailekuba]

@chiomailekuba has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Gm

Please Can i work on this issue

i would love to be a contributor to this great project

ℹ️ Repo Maintainers: To accept this application, review their application or assign @chiomailekuba to this issue.