Skip to content

telemetry pack: the channel table and the rollup, since the durable half already works #2575

Description

@oceanwaves630

Summary

khive has the event plane already: stream.append / stream.read / stream.batch are the durable
log, and brain.event_counts proves the read side is buildable — it returns windowed counts grouped
by kind, actor and verb. What is missing is a telemetry pack: the same shape over an arbitrary
stream rather than over brain's own plane, plus the one piece that is genuinely absent, a declared
assignment of event kinds to carriers.

This is deliberately a small ask. Most of it is already served; the argument is for naming the
service so consumers stop each building the aggregation half in process.

What a consumer has today, as one worked example

One runtime's telemetry package is about 1,100 lines across six modules. Its parts, and where each
would land:

part what it does khive today
typed envelope validates an event's payload against a per-kind schema before it is emitted nothing
channel table assigns each kind a carrier, a failure posture and a cursor kind nothing
durable append event-first append to the log, ahead of any subscriber stream.append
ephemeral ring bounded broadcast, never stored; a reconnect from before the window sees a gap nothing
counters per-run and per-job rollups so a reader answers "what happened" without replaying brain.event_counts, but only over brain's plane
cursors resume a reader from where it stopped stream.read

Two of those six are already khive. One (counters) exists for exactly one plane. Three do not exist.

The part that is a design decision, not a port

The channel table is configuration, not code. Each event kind is assigned three things:

  • carrierdurable (goes to the log, readers get everything since their cursor) or ephemeral
    (goes to a bounded ring, a reader that fell behind sees a gap).
  • failure posturestop (a refused append raises out of the emitting task) or gap (a dropped
    event surfaces as a gap at reconnect, never as an error).
  • cursor kindlog or ring, which follows from the carrier but is worth stating because it
    is what a reader's resume logic keys on.

The reason this belongs in a table rather than at the emit site is that the choice is an operator's,
not an emitter's: whether a streaming delta is worth durable storage depends on the deployment, and
an emitter that decides it has hard-coded one deployment's answer. A worked table looks like:

[telemetry]
stream = "telemetry"
default_carrier = "ephemeral"         # fail-cheap: an unclassified kind is not silently made durable

[[telemetry.channels]]
kinds = ["run.started", "run.completed", "run.failed", "turn.completed", "turn.failed",
         "tool.executed", "governance.denied", "governance.approved"]
carrier = "durable"
failure_posture = "stop"

[[telemetry.channels]]
kinds = ["turn.delta", "*.heartbeat"]
carrier = "ephemeral"
failure_posture = "gap"

[telemetry.ring]
capacity = 4096

One thing to decide deliberately: whether an unclassified kind defaults to durable or ephemeral,
or is refused. Defaulting to durable makes an unnoticed new kind expensive; defaulting to ephemeral
makes it silently unrecorded; refusing makes adding a kind a config change. The example above picks
ephemeral, and the choice should be khive's, stated in the verb's help, not inherited from whichever
default was easiest.

Proposed verbs

telemetry.emit(kind, payload, run_id?, actor?)  -> {carrier, seq?, dropped?, receipt_id}
telemetry.read(stream, since?, limit?, kinds?)  -> {events, next_cursor, gap?}
telemetry.counts(stream, window, group_by?, kinds?) -> {rows: [{key, count, ...}], window, total}
telemetry.channels()                            -> the effective table, as resolved from config
  • telemetry.emit returning the carrier it chose is the point: an emitter should be able to see
    that its event went to the ring rather than the log, without reading the config itself.
  • dropped is a field rather than an inference, for the same reason cached is on a web fetch.
  • gap on the read side is the contract that makes an ephemeral carrier honest: a reader that fell
    out of the window is told so and is not handed a shorter list that reads complete.
  • telemetry.channels() because a table that cannot be read back is a table nobody can debug.

How many backends

Three, and only one of them is new:

  1. The durable log — already khive streams. No new backend, stream.append under the covers.
  2. The aggregationtelemetry.counts. Can be computed at read time over the stream, the way
    brain.event_counts already does, which means no new storage. An incremental rollup table is an
    optimization to reach for when a measurement says read-time is too slow, not before.
  3. The ephemeral ring — the one genuinely new piece, and the one worth questioning. A ring is
    naturally per-process; a cross-process ring is a pub/sub with different operational properties.
    If khive would rather not own a broadcast bus, saying so is a fine answer, and the consequence is
    that carrier = "ephemeral" means "the daemon drops it" rather than "the daemon holds a window of
    it" — which is still a useful contract, just a weaker one. This is the fork worth ruling on first,
    because the rest of the design does not depend on which way it goes.

Ask

A telemetry pack with the four verbs above, the channel table as config, and an explicit ruling on
whether khive owns an ephemeral broadcast window or only a drop. The durable half already works
through stream.append today; what a consumer cannot get from khive is the classification and the
rollup, and those are the two that every consumer otherwise rebuilds.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions