Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Trajectory: Make node load telemetry honest and explicit

> **Status:** ✅ Completed
> **Confidence:** 95%
> **Started:** August 6, 2026 at 06:58 AM
> **Completed:** August 6, 2026 at 07:00 AM

---

## Summary

Made node load unavailable by default, require explicit measurement provenance, keep max_agents=0 consistently unlimited, guard future heartbeat timestamps, and update all SDK/API types with comprehensive tests.

**Approach:** Standard approach

---

## Key Decisions

### Require an explicit load_reported signal before trusting numeric node load
- **Chose:** Require an explicit load_reported signal before trusting numeric node load
- **Reasoning:** Every released provider hard-coded numeric placeholders, including finite-capacity providers, so neither legacy zero nor max_agents alone proves a measurement; explicit additive provenance preserves compatibility and honesty.

### Keep max_agents zero as unlimited and aggregate mixed providers as unlimited
- **Chose:** Keep max_agents zero as unlimited and aggregate mixed providers as unlimited
- **Reasoning:** Placement already treats zero as unlimited, and a single unlimited provider makes the node aggregate unbounded; additive zero previously produced a false finite cap.

### Reject negative heartbeat ages as fresh
- **Chose:** Reject negative heartbeat ages as fresh
- **Reasoning:** Node timestamps are server-stamped, but persisted future timestamps can still occur; an explicit lower bound prevents future data from bypassing the freshness TTL.

---

## Chapters

### 1. Work
*Agent: default*

- Require an explicit load_reported signal before trusting numeric node load: Require an explicit load_reported signal before trusting numeric node load
- Keep max_agents zero as unlimited and aggregate mixed providers as unlimited: Keep max_agents zero as unlimited and aggregate mixed providers as unlimited
- Reject negative heartbeat ages as fresh: Reject negative heartbeat ages as fresh
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
{
"id": "traj_ae0vmeog5qmo",
"version": 1,
"task": {
"title": "Make node load telemetry honest and explicit"
},
"status": "completed",
"startedAt": "2026-08-06T10:58:56.890Z",
"completedAt": "2026-08-06T11:00:17.298Z",
"agents": [
{
"name": "default",
"role": "lead",
"joinedAt": "2026-08-06T10:59:06.998Z"
}
],
"chapters": [
{
"id": "chap_x6t8c9x0k1a9",
"title": "Work",
"agentName": "default",
"startedAt": "2026-08-06T10:59:06.998Z",
"endedAt": "2026-08-06T11:00:17.298Z",
"events": [
{
"ts": 1786013947002,
"type": "decision",
"content": "Require an explicit load_reported signal before trusting numeric node load: Require an explicit load_reported signal before trusting numeric node load",
"raw": {
"question": "Require an explicit load_reported signal before trusting numeric node load",
"chosen": "Require an explicit load_reported signal before trusting numeric node load",
"alternatives": [],
"reasoning": "Every released provider hard-coded numeric placeholders, including finite-capacity providers, so neither legacy zero nor max_agents alone proves a measurement; explicit additive provenance preserves compatibility and honesty."
},
"significance": "high"
},
{
"ts": 1786013948396,
"type": "decision",
"content": "Keep max_agents zero as unlimited and aggregate mixed providers as unlimited: Keep max_agents zero as unlimited and aggregate mixed providers as unlimited",
"raw": {
"question": "Keep max_agents zero as unlimited and aggregate mixed providers as unlimited",
"chosen": "Keep max_agents zero as unlimited and aggregate mixed providers as unlimited",
"alternatives": [],
"reasoning": "Placement already treats zero as unlimited, and a single unlimited provider makes the node aggregate unbounded; additive zero previously produced a false finite cap."
},
"significance": "high"
},
{
"ts": 1786013949800,
"type": "decision",
"content": "Reject negative heartbeat ages as fresh: Reject negative heartbeat ages as fresh",
"raw": {
"question": "Reject negative heartbeat ages as fresh",
"chosen": "Reject negative heartbeat ages as fresh",
"alternatives": [],
"reasoning": "Node timestamps are server-stamped, but persisted future timestamps can still occur; an explicit lower bound prevents future data from bypassing the freshness TTL."
},
"significance": "high"
}
]
}
],
"retrospective": {
"summary": "Made node load unavailable by default, require explicit measurement provenance, keep max_agents=0 consistently unlimited, guard future heartbeat timestamps, and update all SDK/API types with comprehensive tests.",
"approach": "Standard approach",
"confidence": 0.95
},
"commits": [],
"filesChanged": [],
"projectId": "AgentWorkforce/relaycast",
"tags": [],
"_trace": {
"startRef": "f0b084bd7e8e849bfa122c097a23d60cec0a4399",
"endRef": "f0b084bd7e8e849bfa122c097a23d60cec0a4399"
}
}
7 changes: 6 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,12 @@ This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

Packages without a separate changelog are covered by the cross-package notes below.

## [Unreleased]
## [Unreleased - Major]

### Fixed

- Fleet node rosters now return `load: null` until a direct node, or every constituent provider of a broker node, explicitly reports a genuine normalized capacity-utilization measurement.
- Fleet capacity now treats `max_agents: 0` consistently as unlimited.

Comment thread
khaliqgant marked this conversation as resolved.
## [6.3.2] - 2026-08-02

Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,10 @@ ephemeral `node.online`, `node.heartbeat`, and `node.offline` events. Each
carries a `node` payload matching the `GET /nodes` roster entry (capabilities,
tags, `load`, `active_agents`/`max_agents`, `handlers_live`,
`last_heartbeat_at`), so a single event fully refreshes a node's row.
`load` is normalized managed-agent capacity utilization and remains `null`
unless a direct node, or every constituent provider of a broker node,
explicitly reports a genuine measurement;
`max_agents: 0` means unlimited capacity, not zero capacity.

Nodes are first-class delivery hosts and every agent has a node route. `kind`
describes transport (`ws`, `http_push`, or `poll`), `role` describes ownership
Expand Down
6 changes: 6 additions & 0 deletions openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -732,10 +732,16 @@ components:
type: boolean
load:
type: number
nullable: true
minimum: 0
maximum: 1
description: Normalized managed-agent capacity utilization. Null unless a direct node, or every constituent provider of a broker node, explicitly reports a genuine measurement.
active_agents:
type: integer
max_agents:
type: integer
minimum: 0
description: Maximum managed agents. Zero means unlimited capacity.
Comment thread
khaliqgant marked this conversation as resolved.
Comment thread
khaliqgant marked this conversation as resolved.
last_heartbeat_at:
type: string
format: date-time
Expand Down
10 changes: 9 additions & 1 deletion packages/engine/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,15 @@ See the [root changelog](../../CHANGELOG.md) for cross-package release highlight
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [Unreleased - Major]

### Fixed
Comment thread
khaliqgant marked this conversation as resolved.

- Node heartbeats now accept absent/null `load` and require `load_reported` before trusting a numeric measurement.
- Migration `0034` leaves historical placeholder load values unreported.
- `GET /v1/nodes` now returns null load until a direct node, or every constituent provider of a broker node, reports a genuine measurement.
- Future-dated heartbeats no longer count as fresh.
- Broker capacity remains unlimited when any constituent provider is unbounded.

Comment thread
khaliqgant marked this conversation as resolved.
## [6.3.2] - 2026-08-02

Expand Down
102 changes: 100 additions & 2 deletions packages/engine/src/__tests__/conformance/node.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -438,7 +438,7 @@ describe('node adapter conformance', () => {
id: string;
name: string;
capabilities: Array<ReturnType<typeof capability>>;
load?: number;
load?: number | null;
maxAgents?: number;
},
) {
Expand Down Expand Up @@ -497,13 +497,107 @@ describe('node adapter conformance', () => {
await handle.handleMessage(JSON.stringify({
v: 1,
type: 'node.heartbeat',
load: opts.load ?? 0,
...(typeof opts.load === 'number' ? { load: opts.load, load_reported: true } : {}),
active_agents: 0,
handlers_live: true,
}));
return { sock, handle };
}

it('reports placeholder load as unavailable until a finite node explicitly marks it measured', async () => {
const ws = await createWorkspace(stack.app, 'fleet-unreported-load-ws');
const unbounded = await enrollAndAttachNode(ws, {
id: 'node_unbounded',
name: 'unbounded',
capabilities: [capability('spawn:codex', 'spawn')],
maxAgents: 0,
load: null,
});

const roster = await stack.app.request('/v1/nodes?name=unbounded', {
headers: { authorization: `Bearer ${ws.workspaceKey}` },
});
expect(roster.status).toBe(200);
const body = await roster.json() as { data: Array<Record<string, unknown>> };
expect(body.data[0]).toMatchObject({
name: 'unbounded',
load: null,
active_agents: 0,
max_agents: 0,
});

const [stored] = await stack.runtime.handle.db
.select({ load: nodes.load, loadReported: nodes.loadReported })
.from(nodes)
.where(and(eq(nodes.workspaceId, ws.workspaceId), eq(nodes.id, 'node_unbounded')));
expect(stored).toEqual({ load: 0, loadReported: false });

// An unbounded node cannot report normalized utilization because it has
// no finite denominator, even when a client incorrectly claims it did.
await unbounded.handle.handleMessage(JSON.stringify({
v: 1,
type: 'node.heartbeat',
load: 0.75,
load_reported: true,
active_agents: 25,
handlers_live: true,
}));
const explicitlyReportedRoster = await stack.app.request('/v1/nodes?name=unbounded', {
headers: { authorization: `Bearer ${ws.workspaceKey}` },
});
const explicitlyReportedBody = await explicitlyReportedRoster.json() as { data: Array<Record<string, unknown>> };
expect(explicitlyReportedBody.data[0]).toMatchObject({ load: null, active_agents: 25, max_agents: 0 });

// Older brokers sent a literal zero for the same unbounded state. The
// engine knows the denominator is absent and must keep treating it as
Comment thread
khaliqgant marked this conversation as resolved.
// unreported during the rolling upgrade.
await unbounded.handle.handleMessage(JSON.stringify({
v: 1,
type: 'node.heartbeat',
load: 0,
active_agents: 25,
handlers_live: true,
}));
const legacyRoster = await stack.app.request('/v1/nodes?name=unbounded', {
headers: { authorization: `Bearer ${ws.workspaceKey}` },
});
const legacyBody = await legacyRoster.json() as { data: Array<Record<string, unknown>> };
expect(legacyBody.data[0]).toMatchObject({ load: null, active_agents: 25, max_agents: 0 });

const finite = await enrollAndAttachNode(ws, {
id: 'node_finite',
name: 'finite',
capabilities: [capability('spawn:claude', 'spawn')],
maxAgents: 4,
});
await finite.handle.handleMessage(JSON.stringify({
v: 1,
type: 'node.heartbeat',
load: 0,
active_agents: 0,
handlers_live: true,
}));
let finiteRoster = await stack.app.request('/v1/nodes?name=finite', {
headers: { authorization: `Bearer ${ws.workspaceKey}` },
});
let finiteBody = await finiteRoster.json() as { data: Array<Record<string, unknown>> };
expect(finiteBody.data[0]).toMatchObject({ load: null, max_agents: 4 });

await finite.handle.handleMessage(JSON.stringify({
v: 1,
type: 'node.heartbeat',
load: 0,
load_reported: true,
active_agents: 0,
handlers_live: true,
}));
finiteRoster = await stack.app.request('/v1/nodes?name=finite', {
headers: { authorization: `Bearer ${ws.workspaceKey}` },
});
finiteBody = await finiteRoster.json() as { data: Array<Record<string, unknown>> };
expect(finiteBody.data[0]).toMatchObject({ load: 0, max_agents: 4 });
});

it('drives node control directly without the websocket route wrapper', async () => {
const ws = await createWorkspace(stack.app, 'node-control-direct-dispatch');
const db = stack.runtime.handle.db;
Expand Down Expand Up @@ -586,6 +680,7 @@ describe('node adapter conformance', () => {
id: 'control-broker-heartbeat',
type: 'node.heartbeat',
load: 0.25,
load_reported: true,
active_agents: 1,
handlers_live: true,
node_id: 'node_control_broker',
Expand Down Expand Up @@ -671,6 +766,7 @@ describe('node adapter conformance', () => {
id: 'control-direct-heartbeat',
type: 'node.heartbeat',
load: 0.5,
load_reported: true,
active_agents: 7,
handlers_live: true,
node_id: 'node_control_direct',
Expand Down Expand Up @@ -1692,6 +1788,7 @@ describe('node adapter conformance', () => {
v: 1,
type: 'node.heartbeat',
load: 0.5,
load_reported: true,
active_agents: 1,
handlers_live: true,
node_id: 'node_alpha',
Expand Down Expand Up @@ -1749,6 +1846,7 @@ describe('node adapter conformance', () => {
v: 1,
type: 'node.heartbeat',
load: 0.1,
load_reported: true,
active_agents: 0,
handlers_live: true,
}));
Expand Down
19 changes: 19 additions & 0 deletions packages/engine/src/__tests__/conformance/nodeLiveness.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import { describe, expect, it } from 'vitest';
import { isProviderLive } from '../../engine/nodeProvider.js';
import { isNodeLive, NODE_LIVENESS_TTL_MS } from '../../engine/placement.js';

const NOW = 1_700_000_000_000;

describe('node heartbeat freshness', () => {
it('accepts a recent server timestamp and rejects stale or future timestamps', () => {
expect(isNodeLive({ status: 'online', lastHeartbeatAt: new Date(NOW - 1_000) }, NOW)).toBe(true);
expect(isNodeLive({ status: 'online', lastHeartbeatAt: new Date(NOW - NODE_LIVENESS_TTL_MS - 1) }, NOW)).toBe(false);
expect(isNodeLive({ status: 'online', lastHeartbeatAt: new Date(NOW + 1) }, NOW)).toBe(false);
});

it('applies the same explicit negative-age guard to provider freshness', () => {
const provider = { status: 'online', handlersLive: true, lastHeartbeatAt: new Date(NOW + 1) };
expect(isProviderLive(provider, NOW)).toBe(false);
expect(isProviderLive({ ...provider, lastHeartbeatAt: new Date(NOW) }, NOW)).toBe(true);
});
});
Loading
Loading