Design scoped observer tokens for workspace stream and read-only REST

[willwashburn]

Summary

We should split Relaycast authentication into distinct principals instead of using the root workspace key for both administration and observation.

The immediate design question is: should /v1/ws?scope=workspace and read-only history/search REST APIs be accessible with the root rk_live_ workspace key, or should they use a separate observer credential?

Proposed answer: introduce scoped observer tokens, likely ot_live_, and make the workspace stream plus read-only REST surfaces use those tokens. Keep node credentials separate as nt_live_, agent credentials as at_live_, and reserve rk_live_ for workspace administration.

Motivation

Today the workspace stream is effectively a firehose. It is useful for dashboards, brokers, audit tools, backfill systems, and monitoring, but it is too broad to require handing those systems the root workspace key.

A root workspace key can administer the workspace. If that same key is used in browser dashboards, hosted monitors, brokers, or third-party observability tooling, a read-only integration becomes a full workspace compromise.

The new node-delivery direction also makes the boundary clearer:

• Agents should receive through their bound node route.
• Nodes should receive node-scoped delivery/context through node credentials.
• Workspace stream should be observer-only.
• Admin APIs should remain workspace-key-only.

So we should make observation a separate principal, not just another use of the workspace admin key.

Proposed Principal Model

rk_live_: workspace admin key

Root/admin credential for a workspace.

Allowed:

• Create, update, delete workspace configuration.
• Register/create agents.
• Create/rotate/revoke agent tokens.
• Create/register nodes.
• Create/rotate/revoke node tokens.
• Create/rotate/revoke observer tokens.
• Configure observer token scopes and filters.
• Administrative listing of tokens, agents, nodes, and workspace settings.

Not ideal long term:

• Opening /v1/ws?scope=workspace directly.
• Being embedded into dashboards or monitoring tools.
• Being used by brokers that only need observation.

Migration option:

• Temporarily continue accepting rk_live_ on the workspace stream for compatibility, but mark it deprecated and add a clear sunset path.
• For a major upgrade, make /v1/ws?scope=workspace require ot_live_ only.

at_live_: agent token

One agent identity.

Allowed:

• Act as exactly one agent.
• Send messages as that agent.
• Read APIs that are already available to that agent.
• Open the agent websocket/direct route.
• Acknowledge/fail/defer deliveries owned by that agent.

Important invariant:

• A directly connected agent is represented internally as an implicit direct_ws node-of-one, but it should not need a node token. The agent token authenticates the principal; the direct node is a routing/ledger abstraction.

Not allowed:

• Workspace firehose.
• Node control websocket.
• Admin token minting.

nt_live_: node token

One node identity.

Allowed:

• Open /v1/node/ws for that node.
• Receive node-scoped deliveries and context.
• Heartbeat/report node inventory.
• Spawn/register/deregister agents for broker-controlled nodes, subject to node contract/capabilities.
• Complete node-owned invocations.

Not allowed:

• Workspace firehose.
• Root workspace administration.
• Arbitrary read-only message search outside scoped node context.

HTTP push nuance:

• HTTP push delivery auth, such as HMAC/bearer/static header webhook credentials, is adapter auth, not a Relaycast principal.
• If an HTTP receiver needs to call back to Relaycast, it should use a scoped nt_live_ node token or a narrower one-shot delivery acknowledgement credential, not the root workspace key.

ot_live_: observer token

Read-only observer principal for realtime observation and read-only REST backfill/history/search.

Allowed, depending on scopes:

• Open /v1/ws?scope=workspace.
• Read channels, threads, messages, DMs, reactions, agents, nodes, deliveries/activity, and search endpoints according to scopes and filters.

Not allowed:

• Sending messages.
• Creating/updating/deleting channels.
• Registering or mutating agents.
• Creating/rotating node or agent tokens.
• Invoking actions.
• Acknowledging/failing/defering deliveries.
• Opening /v1/node/ws.
• Acting as an agent.

Observer Token Scopes

Suggested scope names:

• stream:read - open workspace observer websocket.
• messages:read - read channel message history.
• threads:read - read thread replies.
• dms:read - read direct/group DM history where allowed by filters.
• channels:read - list/read channels and members.
• search:read - search messages/content.
• agents:read - list/read agents and agent status.
• nodes:read - list/read node roster/status.
• deliveries:read - read delivery status/activity where appropriate.
• activity:read - read workspace activity feed.
• files:read - read file metadata and/or download files, depending on how files are authorized.
• reactions:read - read reactions, if we want finer granularity than messages:read.

Possible preset bundles:

Dashboard observer

For the full observer dashboard:

stream:read
messages:read
threads:read
dms:read
channels:read
search:read
agents:read
nodes:read
activity:read
files:read

Ops monitor

For node/agent health monitoring only:

stream:read
agents:read
nodes:read
activity:read

Search/backfill worker

For indexing/search only:

messages:read
threads:read
dms:read
channels:read
search:read
files:read

Realtime-only consumer

For a pure realtime subscriber:

stream:read

Observer Token Filters

Scopes answer "what type of API can this token use?" Filters answer "which workspace data can it see?"

Suggested optional filters:

• channel_ids: allow only specific channels.
• channel_names: allow only specific channel names.
• include_dms: boolean, default false for safer observer tokens.
• dm_conversation_ids: allow only specific DM/group DM conversations.
• agent_ids: limit agent/node/status visibility to conversations involving selected agents.
• event_types: limit workspace stream event types.
• created_after or retention override: optional backfill lower bound.

Filter behavior should be consistent across websocket and REST:

• If an observer token cannot read a channel via REST, it should not receive that channel's stream events.
• If it cannot read DMs, it should not receive dm.received or group_dm.received stream events.
• If it cannot read files, message events should either omit file download URLs or include only metadata, depending on the file model.

REST Behavior

ot_live_ should be accepted by read-only endpoints only when the token has the required scope.

Candidate endpoint mapping:

Endpoint family	Required observer scope
GET /v1/activity	activity:read
GET /v1/channels	channels:read
GET /v1/channels/:name	channels:read
GET /v1/channels/:name/members	channels:read
GET /v1/channels/:name/messages	messages:read plus channel filter
GET /v1/messages/:id/reactions	messages:read or reactions:read
GET /v1/messages/:id/thread / thread reads	threads:read plus channel/conversation filter
DM list/read endpoints	dms:read plus conversation filter
Search endpoints	search:read plus filters applied to result set
Agent directory/list/status reads	agents:read
Node roster/status reads	nodes:read
Delivery status/activity reads	deliveries:read or activity:read, depending on endpoint semantics
File metadata/download	files:read plus message/channel/conversation filter

All write endpoints should reject ot_live_ with 403 forbidden or 401 invalid_token depending on current auth semantics. Prefer 403 insufficient_scope when the token is valid but under-scoped.

WebSocket Behavior

/v1/ws?scope=workspace should eventually accept only ot_live_ with stream:read.

Behavior:

• Authenticate observer token.
• Resolve workspace.
• Apply event filters before publishing to that socket.
• Apply channel/conversation visibility filters before emitting message/DM/thread/file events.
• Stamp stream events the same way current workspace observer clients expect, unless we intentionally version the stream.

Open compatibility question:

• During migration, should rk_live_ still be accepted for /v1/ws?scope=workspace with a deprecation warning/event/header, or should the major upgrade remove it immediately?

Token Lifecycle API

Potential workspace-admin endpoints:

POST /v1/observer-tokens
GET /v1/observer-tokens
GET /v1/observer-tokens/:id
PATCH /v1/observer-tokens/:id
POST /v1/observer-tokens/:id/rotate
DELETE /v1/observer-tokens/:id

Create request shape:

{
  "name": "dashboard-prod",
  "description": "Production read-only dashboard",
  "scopes": ["stream:read", "messages:read", "search:read", "channels:read", "agents:read", "nodes:read"],
  "filters": {
    "channel_names": ["general", "ops"],
    "include_dms": false,
    "event_types": ["message.created", "agent.status.active", "agent.status.offline", "node.online", "node.offline"]
  },
  "expires_at": "2026-12-31T23:59:59Z"
}

Create response shape:

{
  "ok": true,
  "data": {
    "id": "ot_...",
    "name": "dashboard-prod",
    "token": "ot_live_...",
    "scopes": ["stream:read", "messages:read"],
    "filters": { "include_dms": false },
    "expires_at": "2026-12-31T23:59:59Z",
    "created_at": "..."
  }
}

List response should not return token material after creation/rotation.

Storage Model

Potential table: observer_tokens

Suggested columns:

• id
• workspace_id
• name
• description
• token_hash
• scopes JSON array
• filters JSON object
• status (active, revoked)
• expires_at
• created_by / created_by_type
• last_used_at
• created_at
• updated_at
• revoked_at

Indexes:

• unique token hash
• (workspace_id, status)
• (workspace_id, name) if names should be unique per workspace
• (expires_at) if we sweep expired tokens

Auth Middleware Shape

Add an observer auth result alongside current workspace/agent/node auth results.

Conceptually:

type AuthPrincipal =
  | { kind: 'workspace'; workspace }
  | { kind: 'agent'; workspace; agent }
  | { kind: 'node'; workspace; node }
  | { kind: 'observer'; workspace; observerToken; scopes; filters };

Then route middleware can use requirements like:

requireWorkspaceAdmin
requireAgentToken
requireNodeToken
requireObserverScope('messages:read')
requireWorkspaceRead(['messages:read']) // accepts rk_live_ admin or ot_live_ scoped token?

Open design question:

• Should read-only REST endpoints accept both rk_live_ and ot_live_, where rk_live_ is treated as admin with all read scopes?
• Or should some observer-specific endpoints require ot_live_ only?

My bias: REST read endpoints can accept both rk_live_ and appropriately scoped ot_live_; workspace stream should move to ot_live_ only because it is commonly held open by less-trusted clients.

Error Semantics

Suggested errors:

• 401 invalid_token: token is absent, malformed, unknown, expired, or revoked.
• 403 insufficient_scope: valid observer token lacks the required scope.
• 403 filtered_resource: valid observer token has the scope but its filters exclude the requested channel/conversation/resource.

We should decide whether filtered resources should return 403 or 404 to avoid disclosing existence. For external observer tokens, 404 may be safer for channel/message/conversation reads.

Security Considerations

• Observer tokens must be read-only by construction.
• Store only token hashes.
• Support rotation and revocation.
• Track last_used_at for audit/debugging.
• Optional expiration should be encouraged or required for generated dashboard tokens.
• DM access should be opt-in, not default.
• File download URLs need special care: files:read should not accidentally bypass message/channel filters.
• Search results must enforce the same filters as direct history reads.
• Workspace stream must apply filters before events hit the socket.
• Avoid mixed-case compatibility fallbacks in wire fields; keep JSON snake_case.

Migration Plan

Possible major-upgrade path:

1. Add observer token storage and admin CRUD endpoints.
2. Add observer auth parsing for ot_live_.
3. Teach read-only REST endpoints to accept scoped observer principals.
4. Teach workspace websocket auth to accept ot_live_ with stream:read.
5. Update dashboard to create/use an observer token instead of storing rk_live_ for live stream/read operations.
6. Deprecate or remove rk_live_ support for /v1/ws?scope=workspace.
7. Update README, OpenAPI, and TypeScript/Python/Rust/Swift SDKs.
8. Add conformance tests for scope allow/deny, filters, stream auth, and REST auth.

Acceptance Criteria

• ot_live_ tokens can be minted, listed, rotated, revoked, and expired by a workspace admin.
• /v1/ws?scope=workspace accepts ot_live_ with stream:read.
• /v1/ws?scope=workspace rejects at_live_ and nt_live_.
• The final major-upgrade behavior rejects rk_live_ on workspace stream, or the compatibility behavior is explicitly documented and tested.
• Read-only REST endpoints accept ot_live_ only with the required scopes.
• Write REST endpoints reject ot_live_.
• Channel/conversation/event filters are enforced consistently across REST and websocket stream.
• Search applies observer filters before returning results.
• File reads/downloads respect observer scopes and filters.
• SDKs expose observer token creation and observer-auth clients without encouraging root workspace key use in dashboards.
• README and OpenAPI describe the four-principal model: rk_live_, at_live_, nt_live_, ot_live_.

Open Questions

1. Should rk_live_ be rejected immediately on workspace stream as part of the major upgrade, or deprecated for one release?
2. Should observer tokens be named ot_live_, or should we use a more explicit prefix like obs_live_?
3. Should observer token filters support both allowlists and denylists, or allowlists only?
4. Should DM read access default to false even for dashboard presets?
5. Should observer tokens be able to read delivery status, or should delivery reads remain agent/node/admin only?
6. Should file download require both files:read and the scope for the containing message/conversation?
7. Do we need one-shot or delivery-scoped acknowledgement tokens for pure http_push receivers, separate from nt_live_?
8. Should nt_live_ tokens get any read-only REST capabilities for their bound agents, or only node control APIs?

[willwashburn]

Closing as implemented — validated against current main (v6.0.5, f7e86a3).

The proposal shipped in v5.0.8 (0e1dadb, 2026-06-30) plus follow-ups (#230, #231, #232, #240):

• ot_live_ observer principal exists alongside rk_live_/at_live_/nt_live_ (packages/engine/src/auth/tokenKind.ts), read-only by construction with a dedicated observer_token_forbidden error for write operations.
• observer_tokens table with hashed tokens, scopes, filters, status, and expiry (packages/engine/src/db/schema.ts).
• Workspace stream requires ot_live_ with stream:read — packages/engine/src/engine/wsAuth.ts rejects rk_live_ on /v1/ws?scope=workspace outright ("Observer token required for workspace stream"). The migration question resolved to immediate rejection rather than a deprecation window.
• Scoped read-only REST via requireWorkspaceRead(scopes) middleware with insufficient_scope (403); per-event-type scope enforcement on the stream (packages/engine/src/engine/observerToken.ts).
• Lifecycle API — POST/GET/PATCH/rotate/DELETE /v1/observer-tokens, documented in openapi.yaml with a dedicated observerToken security scheme on ~30 read endpoints.
• Dashboard migrated — packages/observer-dashboard/src/lib/observer-token.ts mints a scoped observer token on login and revokes it on logout, instead of holding the root workspace key for the stream.

---

Generated by Claude Code