help wanted: production observability pack for self-hosted hotels (metrics that GMs feel)

[telivity-otaip]

Why hotels care

Self-hosted HAIP (compose / bare metal) fails in ways GMs feel before engineers do: booking widget 500s, ARI push stuck, night audit cron missed, webhook backlog to fiscal/reporting. HAIP_BUILD_PLAN.md lists "Hardening & reliability (integration retries/timeouts, observability, scale)" as planned. Integrations catalog mentions Grafana — we need an honest first metrics surface, not a full APM rewrite.

Current state (code anchors)

• Health: /api/v1/health
• Webhook deliveries table as delivery observability
• Channel lastSyncStatus / errors
• Agent orchestration performance endpoint (product metrics, not infra)
• Harden CLI probes (HTTP correctness, not time-series)
• No first-class Prometheus /metrics (confirm; add if missing) for RED/hotel-ops signals

What we need (MVP)

1. Prometheus-compatible metrics endpoint (or documented OpenTelemetry export) covering at least:
   • HTTP request rate/error/latency by route group
   • Booking-engine create success/fail
   • Channel ARI push success/fail counters
   • Night audit run success/fail + last business date gauge
   • Webhook delivery failure counters
   • Queue/BullMQ depth if applicable
2. Example Grafana dashboard JSON under ops/harden/ or docs/
3. Alert rule examples: booking-engine error rate, ARI fail, audit missed window
4. Docs for self-hosters: scrape config, cardinality warnings, multi-tenant label policy (property_id cardinality!)

Open questions

• Label with property_id on all series, or only on business metrics with aggregation?
• Prefer Prometheus pull vs OTLP push for the reference compose?

Out of scope

• Replacing structured logs
• Full distributed tracing across OTAIP
• Vendor lock-in to a paid APM

Acceptance criteria

• Metrics endpoint scrapable in prod compose
• Example dashboard shows the signals above
• Docs warn about high-cardinality labels
• Basic test that metrics route is mounted (auth policy explicit: public vs protected)

How to contribute

SRE / platform engineers who have run hotel or booking stacks; also indie hoteliers who already scrape with Prometheus/Grafana and can say which panels they look at at 07:00.

Search keywords: Prometheus, Grafana, observability, self-hosted, night audit monitoring, ARI metrics, booking engine errors, hotel PMS ops