Skip to content

Export HTTP request rate, error rate and p95 latency — the monitoring plan's largest gap #148

Description

@Miracle656

Background

Lens already has real observability: src/metrics.ts exports seven Prometheus metrics via a public /metrics route, and /status reports lastProcessedAt. That part works and is not in question.

What is missing is the HTTP layer. Nothing anywhere in Lens exports:

  • request rate
  • error rate — nothing counts 5xx at all
  • p95 latency (db_query_duration_seconds covers the database, not the request)

price_requests_total counts price calls specifically, not requests overall, so it cannot answer "is the API healthy".

This is called out as the largest gap in veil's docs/MONITORING.md, which is an SCF #46 tranche-2 deliverable. Right now that document has to mark error rate and p95 as ❌ nowhere — because listing a metric nothing emits would be worse than admitting the hole.

The wraith half is tracked separately in wraith#39. This issue is Lens only.

Why it is not a five-line middleware

The naive version — one histogram labelled by req.url — produces unbounded label cardinality (every address, every pair, every cursor becomes its own time series) and will eventually take down the Prometheus scraping it. Getting the label set right is the work.

What to build

A Fastify hook exporting standard HTTP metrics on the existing registry.

Key files

  • src/metrics.ts — the registry and existing metrics
  • src/index.ts — plugin registration order and the /metrics route
  • src/api/rest.ts — routes, and where price_requests_total is incremented today

Suggested execution

  1. Add http_requests_total{method, route, status_class} and http_request_duration_seconds{method, route}.
  2. Label by route template (/price/:assetA/:assetB), never the resolved URL. Fastify exposes the matched route — use it, and fall back to a literal "unmatched" rather than the raw path for 404s.
  3. Keep status_class (2xx/4xx/5xx) rather than the exact code, to bound cardinality; add exact status only if a concrete need appears.
  4. Choose histogram buckets deliberately for this service and say why in a comment — the defaults are tuned for fast in-process calls, and these handlers make network and DB round-trips.
  5. Make sure the hook runs for error responses too. A 500 that escapes the counter is the exact case this exists to catch.
  6. Leave price_requests_total alone — it means something different, and dashboards may use it.

Acceptance criteria

  • http_requests_total and http_request_duration_seconds are exported on /metrics
  • Labels use the route template; a test asserts that 100 requests to distinct addresses produce one series, not 100
  • 5xx responses are counted — assert with a route that throws, not only with happy-path traffic
  • 4xx (validation failures, bad API key) are counted and distinguishable from 5xx
  • Unmatched routes do not create a series per path
  • Latency is observed for both successful and failed requests
  • Histogram buckets are justified in a comment
  • A short section in the repo README (or docs/) lists what is exported and what an alert on it should look like, so MONITORING.md can be updated to name real metrics

Notes

  • Registration order matters: the hook must sit outside the x402 and auth plugins, or payment-rejected and unauthorised requests go uncounted.
  • /metrics itself should not be counted, or the scraper inflates its own numbers.

Drips Wave · Complexity: Advanced · 200 points


Required: Before submitting, join the contributor Telegram so your work can be tracked and counted toward the Stellar Wave: https://t.me/+fxHXq8f1SwlkZDBk

Metadata

Metadata

Assignees

Labels

Stellar WaveIssues in the Stellar wave programarea:apiREST / GraphQL APIdifficulty:advancedMulti-day projecthelp wantedExtra attention is neededpoints:200Advanced tier — 200 Wave points

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions