[Telemetry] Observability: Metrics & Context Propagation

[Abishekcs]

Implementation Plan

Phase 1 Telemetry primitives (rage)

• Rage::Telemetry.every(ms), wraps Iodine.run_every handles boot timing internally (Iodine.running? and on_state(:on_start))
• Write Specs along with each PR.

Phase 2 Socket & connection metrics (rage & rage-iodine)

rage-iodine exposes no accept-queue depth. We add them, following the existing pattern (task_inc!/task_dec!/stopping? core fn in fio.c, declaration in fio.h and Ruby wrapper + registration in
iodine_defer.c).

Metric:

• Socket backlog (Accept Queue), new C method Iodine.socket_backlog in iodine: save the listener's fd when fio_listen runs, then read the kernel's accept-queue length (handshaked (ESTABLISHED) connections waiting for accept()) via getsockopt(TCP_INFO) tcpi_unacked.
• Rage::Telemetry.queued_connections, method-based reader exposing the Iodine server accept queue.
• Update existing spec or add new if required according to the changes.

Phase 3 OpenTelemetry metrics (opentelemetry-instrumentation-rage)

• Event loop lag via Telemetry.every
• GC pressure gauges via periodic GC.stat sampling
• Observable gauges for all Rage raw keys
• Test the metrics are correct using the SDK's , then confirm they arrive at a real Collector running locally

Phase 4 Grafana dashboard

• Connect to Grafana dashboard

Phase 5 Context propagation (rage)

• Make Current.user (and all CurrentAttributes) survive every fiber boundary — enqueue, task execution and Fiber.schedule

Phase 6 Benchmark & docs

• Benchmarks
• Complete spec coverage for the new metrics layer
• Docs + production monitoring guide

[rsamoilov]

Rage::Telemetry.raw, symbol-key read interface

It doesn’t really have to be a symbol-key read interface. Could regular methods work better here?

:backlog_size from Rage::Deferred::Queue#backlog_size (already tracked)

backlog_size may not be the best name, but what I meant by it was the same thing you described as Iodine.socket_backlog. Exposing the deferred backlog size can also be useful but is not a priority.

:active_fibers from FiberScheduler#fiber, increment on start, decrement in ensure

Not sure how useful it is.

Active connections , new fio_data counter

Same here — not sure how useful it is, especially considering users who need it can count active connections via the controller.action.process span.

[Abishekcs]

It doesn’t really have to be a symbol-key read interface. Could regular methods work better here?

Sure, I can change it.

backlog_size may not be the best name, but what I meant by it was the same thing you described as Iodine.socket_backlog. Exposing the deferred backlog size can also be useful but is not a priority.

oh i thought it also meant the deferred backlog size I will leave it out then.

Same for active connection and fiber won't consider them.

[Abishekcs]

REACTOR_LAG.md

The main reason here is that I'm trying to understand how to expose the "event loop lag" metric we've discussed previously natively from Iodine. And if so, it would be added to the same module.

@rsamoilov I suppose i should start looking into this ? I didn't thought of measuring natively.

I will try to let you know this week whatever I can find. Will use the doc you shared as a base.

[rsamoilov]

Sure, feel free to look into it.

We initially were discussing using Rage::Telemetry.every, similar to what you'd do with setTimeout in Node, but then it hinges on instrumentations to support it. A native metric (if it can be implemented with very low overhead) makes the data available right away, and we can probably make it more precise this way too.