feat(backend): OpenTelemetry instrumentation with trace IDs in logs

[vovinacci]

Closes the D11 debt for the backend (ADR-0010): TracerProvider with no span processor -- spans are real and carry real trace IDs but nothing is exported until Tempo lands. FastAPI extracts W3C traceparent from HTTP headers; the gRPC server gets the aio_server_interceptor added explicitly to its interceptor list (the GrpcAioInstrumentorServer monkeypatch depends on import order and was rejected). Every log line now carries trace_id via a logging filter attached to the root handlers; the canary's journey trace IDs appear verbatim on backend log lines for the same requests, so log-trace correlation works in Loki today.

Health and metrics endpoints are excluded from span creation with end-anchored patterns -- a bare /metrics exclusion would also swallow the real POST /metrics/frontend web-vitals route (regression-tested). Log format stays plain text; the JSON conversion is separate D6 debt, noted in the README.

Summary by CodeRabbit

• New Features
  • Added end-to-end OpenTelemetry tracing for HTTP and gRPC requests, including trace-context propagation.
  • Added trace IDs to log entries to support request and log correlation.
  • Added automatic tracing for incoming requests while excluding health, readiness and metrics routes where appropriate.
• Documentation
  • Documented tracing behaviour, trace ID correlation and route exclusions.
• Tests
  • Added coverage for HTTP and gRPC propagation, generated trace IDs, excluded routes and log correlation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3ca8f463-acb0-44c7-8ce5-71e920a8c3b1

📥 Commits

Reviewing files that changed from the base of the PR and between 58fbd53 and c98d2d5.

📒 Files selected for processing (6)

• services/backend/README.md
• services/backend/app/grpc_server.py
• services/backend/app/main.py
• services/backend/app/otel.py
• services/backend/pyproject.toml
• services/backend/tests/test_otel.py

---

Walkthrough

Adds OpenTelemetry tracing to the backend’s FastAPI and gRPC servers, propagates trace context, correlates logs with trace IDs, excludes selected endpoints, and adds documentation and tests for the new behaviour.

Changes

OpenTelemetry tracing

Layer / File(s)	Summary
Tracing foundation services/backend/app/otel.py, services/backend/pyproject.toml	Configures a record-only tracer provider, provides gRPC context extraction, injects trace IDs into logs, and adds OpenTelemetry dependencies and MyPy overrides.
HTTP and gRPC integration services/backend/app/main.py, services/backend/app/grpc_server.py	Instruments FastAPI with excluded health and metrics routes, configures trace-aware logging, and adds the OpenTelemetry gRPC interceptor.
Tracing validation and documentation services/backend/tests/test_otel.py, services/backend/README.md	Tests HTTP/gRPC propagation, route exclusions, fresh trace IDs, and log filtering; documents the tracing configuration and correlation format.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• vovinacci/devops-demo#161: Related gRPC server setup and interceptor wiring.

Sequence Diagram(s)

sequenceDiagram
  participant HTTPClient
  participant FastAPIApp
  participant OpenTelemetry
  participant Logger
  HTTPClient->>FastAPIApp: HTTP request with traceparent
  FastAPIApp->>OpenTelemetry: Create or propagate server span
  OpenTelemetry->>Logger: Inject active trace_id
  Logger-->>FastAPIApp: Emit trace-correlated log

Loading

sequenceDiagram
  participant GRPCClient
  participant GRPCServer
  participant OTELInterceptor
  participant SpanExporter
  GRPCClient->>GRPCServer: RPC with traceparent metadata
  GRPCServer->>OTELInterceptor: Extract W3C trace context
  OTELInterceptor->>SpanExporter: Record server span
  SpanExporter-->>GRPCServer: No production export configured

Loading

Poem

A rabbit hops through traces bright,
From HTTP hops to gRPC flight.
Logs wear IDs in tidy rows,
While health checks leave no span trails.
No exporter carries the tune—
The spans still dance beneath the moon.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarises the main change: backend OpenTelemetry instrumentation and trace IDs in logs.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/backend-otel

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}