Actor resource-usage events: emit from ateom over the OTLP relay

[baizhenyu]

Proposal

Move actor resource-usage event emission from atelet into ateom, and send the events to the OTel collector as OTLP log records through atelet's unix-socket relay. Two parts:

1. Periodic usage events emitted by ateom. Each ateom samples its own actors on a timer and emits one ate.actor.usage_sampled record per actor. Today atelet's sweep emits these records to stdout after it polls each ateom (atelet: emit per-actor usage events from the stats sweep #1206).
2. Lifecycle usage events, reinstated in ateom. A first sample when the workload becomes measurable, and a final sample taken inside CheckpointWorkload before the sandbox goes away. These were descoped from atelet: emit per-actor usage events from the stats sweep #1206 because atelet taking them added latency to the resume and suspend paths. In ateom they are local reads at points the runtime already passes through.

Both go out over the relay, not over the pod network.

Design: emission lives in ateom

• The data is local. ateom holds the attribution and does the measurement. atelet only relays a copy back to itself over gRPC to emit it. The round trip adds nothing.
• The lifecycle bracket needs ateom. The final sample must be taken before CheckpointWorkload tears the sandbox down. Only ateom is on that path. atelet: emit per-actor usage events from the stats sweep #1206 proved atelet cannot do it without touching resume and suspend latency.
• Multi-actor cost. Once workers host N actors ([EPIC] Multi-Actor Worker Support #1266), one atelet probe measures every actor on the worker, and the discovery RPC's worst case grows with N (Actor resource telemetry: adapt to multi-actor workers #1640 item 2). If ateom samples on its own timer, the RPC returns cached samples and the cost problem goes away.
• Pending state stops hiding. ateom knows how long each actor has been pending. A "pending too long" event is a local timer, which closes Actor resource telemetry: adapt to multi-actor workers #1640 item 5 at the source.

Design: transport is the relay

#809 gave the policy and the mechanism. ateom runs in the worker pod next to untrusted agent code, so it has no network path to the collector. It exports traces and metrics through a unix socket atelet serves and forwards. A unix socket cannot leave the node, N ateoms collapse into atelet's one collector connection, the traffic stays clear of the egress rules ateom installs for the actor, and atelet outlives the worker pod so records queued at teardown are not lost. The last point is exactly what the final sample needs.

The direct export to OTEL_EXPORTER_OTLP_ENDPOINT stays what it is today: a fallback when the relay socket is absent at startup, not the design.

Dependencies

• (feat): Actor lifecycle events over otlp #1658 (actor lifecycle events over OTLP): brings otel/log, otel/sdk/log, otlploggrpc, serverboot.InitLogging, the internal/actorevent builder, and the events.yaml registry. Our event is one more Event value and one more registry entry.
• InitLogging dials the network. It needs a conn variant, like InitMetricsPushOnlyVia, so ateom can point it at the relay socket. Small; can land with or after (feat): Actor lifecycle events over otlp #1658.
• The relay (internal/otlprelay) forwards TraceService and MetricsService only. It needs LogsService. No dependency; can go first.

Plan

1. Relay: add LogsService. Third copy of the existing verbatim-forward pattern. Test: a log record sent by an ateom arrives at the collector with the ateom's own resource attributes, not atelet's.
2. serverboot.InitLoggingVia(ctx, opts, conn). Uses otlploggrpc.WithGRPCConn when conn is non-nil. ateom calls it with the relay connection it already opens for metrics.
3. Event definition. ate.actor.usage_sampled in internal/actorevent and docs/metrics/registry/events.yaml. Attributes: the actor identity set, ate.sandbox.class, ate.stats.source, the four measurements, observed_at_unix_nano, and kind with values periodic, first, final. Severity Info.
4. Periodic emission in ateom. A per-ateom timer with a configurable interval (default matches atelet's sweep, 60s). One record per actor per tick. Pending actors emit a record with source = STATS_SOURCE_UNSPECIFIED and no measurements, so a pending actor is visible in the log stream.
5. Lifecycle emission in ateom. kind: first when the workload becomes measurable (the cgroup leaf exists on gVisor; the guest target is published on micro-VM). kind: final inside CheckpointWorkload, taken before teardown, and echoed in CheckpointWorkloadResponse so atelet can log it too. The final sample carries the epoch's last cumulative CPU value, which is how a consumer reconstructs lifetime CPU across restore epochs (Actor resource telemetry Phase 1: atelet reader and OTel emission #896).
6. atelet: stop emitting. Remove statsEventEmitter and its stdout records. atelet keeps the metrics channel: the template aggregation, the CPU delta baselines, and the OTel push. Nothing else in atelet changes.
7. Discovery RPC returns cached samples. Once ateom samples on its own timer, GetActiveWorkloadStats returns the last sample per actor instead of measuring on demand. The atelet timeout constants can then drop to a plain RPC timeout. This is the multi-actor cost fix from Actor resource telemetry: adapt to multi-actor workers #1640 item 2, and it is why this issue sequences before the micro-VM capacity flip.

Steps 1 and 2 can go now. Steps 3 to 7 wait for #1658.

Volume

Lifecycle events fire a few times per actor per day. Usage samples fire every 60s per actor: at 1,000 actors, about 1.4M records per day. The batch processor handles the rate, but the collector and its backend see a new sustained stream. The event gets its own gate, off by default, so an operator can send lifecycle events to OTLP without usage samples. Same OTEL_LOGS_EXPORTER mechanism for the provider; a second flag on ateom for the usage event.

Out of scope

• The template-level metrics. They stay in atelet, per Observability At-Scale #174: ateom must not write to the TSDB, and the CPU deltas need a cross-actor view.
• The kubectl ate top surface. Unchanged.
• Any change to the stdout copy of lifecycle events. (feat): Actor lifecycle events over otlp #1658 keeps both copies on purpose; usage events follow the same rule until a filelog agent exists.

Related

#550 (parent) · #896 (phase 1, closed; the lifecycle bracket was its open follow-up) · #1206 (atelet events channel, to be retired) · #809 (the relay) · #1658 (lifecycle events over OTLP) · #1640 (multi-actor telemetry; this closes its items 2 and 5) · #1266 (the epic)