This document outlines the observability stack used in Traqora to monitor application performance, track logs, visualize metrics, and alert on critical issues.
The monitoring stack is composed of the following services (defined in monitoring/docker-compose.monitoring.yml):
- Prometheus (Port 9090): Time-series database for collecting and storing metrics.
- Grafana (Port 3000): Visualization platform for creating dashboards from Prometheus and Loki data.
- AlertManager (Port 9093): Handles alerts sent by Prometheus, routing them to Slack and PagerDuty.
- Loki (Port 3100): Log aggregation system.
- Promtail: Log shipper that reads logs and sends them to Loki.
- Jaeger (Ports 16686 UI, 4317 gRPC, 4318 HTTP): Distributed tracing platform to track requests across services.
- Node Exporter & Postgres Exporter: Exporters for system-level and database metrics.
- Grafana: http://localhost:3000 (Default Admin:
admin/admin) - Prometheus: http://localhost:9090
- AlertManager: http://localhost:9093
- Jaeger UI: http://localhost:16686
All Traqora backend endpoints automatically emit metrics via the metricsMiddleware. The backend exposes these metrics at /metrics for Prometheus to scrape.
Metrics include:
traqora_http_request_duration_secondstraqora_http_requests_totaltraqora_http_request_errors_total- Various business and blockchain specific metrics (see
packages/backend/src/services/metrics.ts).
Traqora uses winston for structured JSON logging.
Correlation IDs:
To track requests as they flow through the system, a correlationId is generated (or read from the x-correlation-id header) at the edge (in packages/backend/src/middleware/requestLogger.ts). This ID is stored in AsyncLocalStorage and automatically appended to all log entries generated by winston during the lifecycle of that request.
This makes debugging easier in Loki by filtering with {job="traqora-backend"} |= "your-correlation-id".
Every job in packages/backend/src/jobs emits structured JSON logs with a
stable field set so Loki queries can analyse failures across runs:
| Field | Meaning |
|---|---|
job |
Stable job name (e.g. cache-warming, flight-status-polling, notification-worker) |
jobId |
Unique id for one invocation — a uuid for cron runs, or the queue's job id for Bull/Redis-queue jobs (stable across retries) |
step |
Where in the lifecycle the event happened (start, a named step, complete, failed) |
durationMs |
Milliseconds elapsed since this run started |
outcome |
success or failure on terminal events |
error |
Human-readable error message on failures |
Promtail promotes job, step and outcome to Loki labels (under the
jobName label, so the scrape-level job label is untouched); jobId and
durationMs stay in the JSON body because they are high cardinality.
# All failures for a specific job in the last hour
{job="traqora-backend"} | json | jobName="cache-warming" | outcome="failure" | __error__=""
# Every step of a single run (group retries by jobId)
{job="traqora-backend"} | json | jobId="<uuid>" | __error__=""
# Slow job runs (any job event slower than 10s in the last 24h)
{job="traqora-backend"} | json | jobName=~"cache-warming|flight-status-polling" | durationMs > 10000 | __error__=""
# Failure rate per job over the last hour (label-based)
sum by (jobName) (rate({job="traqora-backend"} | json | jobName=~".*" | outcome="failure" | __error__="" [1h]))
# Promtail-derived metric: job events by job, step and outcome
sum by (jobName, outcome) (rate(traqora_loki_job_events_total[5m]))
The backend classifies every booking-funnel operation (search, booking, refund) against a latency SLO target and exposes them as Prometheus metrics:
| Metric | Meaning |
|---|---|
traqora_slo_events_total{operation, result} |
Counter of events classified good/bad against the latency target |
traqora_slo_observed_latency_seconds{operation} |
Histogram of observed latencies |
Latency targets (keep in sync with SLO_TARGETS in
packages/backend/src/services/metrics.ts):
| Operation | Target | Instrumented at |
|---|---|---|
search |
2s (p95, 95% of events) | services/flightSearchService.ts (whole-search latency incl. cache hits) |
booking |
300s (p95, 95% of events) | recordBookingConfirmed() in services/metrics.ts |
refund |
3600s (p95, 95% of events) | recordRefundProcessed() in services/metrics.ts |
Recording rules and burn-rate alerts live in
monitoring/prometheus/slo-rules.yml (loaded via prometheus.yml):
traqora:slo_bad_ratio:rate5m/rate30d— share of bad events per operationtraqora:slo_error_budget_remaining— fraction of the 5% (30-day) error budget still available (1 = full, 0 = spent)traqora:booking_funnel_search_to_created:ratio24hand...created_to_confirmed:ratio24h— funnel conversion rates used by the Grafana dashboard- Alerts:
SLOFastBurnRate(burn rate > 14.4x over 5m),SLOSlowBurnRate(> 6x over 30m) andSLOErrorBudgetExhausted, all linking the SLO burn runbook (https://github.com/traqora/runbooks/slo-burn.md)
The Grafana dashboard monitoring/grafana/dashboards/traqora-slo.json shows
Traqora - Booking Funnel SLO: the booking funnel (searches → created →
confirmed with conversion rates), p95 latency vs target per operation, and
remaining error budget per operation.
Traqora uses OpenTelemetry for distributed tracing. The SDK is initialized in packages/backend/src/tracing.ts before the application starts, providing auto-instrumentation for HTTP, Express, PostgreSQL, and other standard libraries.
Traces are exported to Jaeger using the OTLP HTTP exporter. You can view the full lifecycle of a request, including database query durations and external API calls, in the Jaeger UI.
Prometheus evaluates rules defined in monitoring/prometheus/alerts.yml every 15 seconds.
Key alerts include:
- ServiceDown: Backend is unreachable.
- HighErrorRate: The percentage of 4xx/5xx errors exceeds 5% of total traffic.
- CriticalAPILatency: The 95th percentile of request duration exceeds 5 seconds.
- DatabaseConnectionFailed: Health check failures for the database.
- LowWalletBalance: Operational wallet drops below safe XLM thresholds.
AlertManager routes these to configured Slack webhooks and PagerDuty.