Skip to content

engine: nodes.active_agents has two independent writers that overwrite each other (binding reservations vs heartbeat recompute) #284

Description

@willwashburn

Severity: medium/major · packages/engine/src/engine/node.ts:768-797 (reserveNodeAgentSlot/releaseNodeAgentSlots) vs engine/nodeProvider.ts:309-333 (recomputeNodeAggregate)

Binding operations increment/decrement nodes.activeAgents directly to enforce capacity (activeAgents = activeAgents + 1 guarded on activeAgents < maxAgents), but every broker node.register/node.heartbeat overwrites the same column with the provider self-report sum:

const activeAgents = providers.reduce((sum, p) => sum + p.activeAgents, 0);
await db.update(nodes).set({ ...activeAgents, maxAgents, ... })

maxAgents is likewise recomputed while the binding path reads the persisted value. The two accounting schemes disagree and clobber each other.

Failure scenario: an agent registers via a node (slot reserved, activeAgents=1); before the broker's roster reflects it, a heartbeat reports active_agents=0recomputeNodeAggregate resets the column, dropping the reservation → a concurrent capacity check under-counts and over-subscribes the node. Conversely a stale high provider report wedges a node at false capacity. This is the same bug class documented as a past incident in relaycast-cloud's CLAUDE.md ("binding-only deactivation left nodes.active_agents overcounting").

Fix: pick a single source of truth for the column (derive from active agent_node_bindings rows, or make the heartbeat path reconcile instead of overwrite).


Found during a multi-agent reliability review of the relay tool/protocol.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions