Skip to content

[AR-448] Make Relay workspace identity durable across node restarts - #1402

Open
khaliqgant wants to merge 2 commits into
mainfrom
feat/ar-448-durable-workspace-identity
Open

[AR-448] Make Relay workspace identity durable across node restarts#1402
khaliqgant wants to merge 2 commits into
mainfrom
feat/ar-448-durable-workspace-identity

Conversation

@khaliqgant

@khaliqgant khaliqgant commented Jul 30, 2026

Copy link
Copy Markdown
Member

Closes AR-448.

The bug

A node started with no project-pinned workspace fell straight through to the broker, which mints a brand-new messaging-only workspace (startup_single_session_set_from_sources, crates/broker/src/relaycast/auth.rs).

Nothing errored. The node came up, the resident agent registered, everything looked healthy — but the agent was a stranger in a different workspace with a new address, so every message sent to its previous address went nowhere. khaliq-chief became a new process-lifetime agent on each such start.

The fix

node up (and the local up alias) now resolves the workspace in this order:

  1. explicit --workspace-key / RELAY_WORKSPACE_KEY / AGENT_RELAY_WORKSPACE_KEY / RELAY_API_KEY, or a RELAY_NODE_TOKEN that already implies a workspace;
  2. the project pin from a previous start;
  3. the machine-global canonical workspace (agent-relay workspace join|switch) — new;
  4. mint a new workspace — last resort.

Step 3 is the fix. The resolved key is pinned to the project after startup, so later starts take step 2 and no key is ever copied by hand. A machine with no canonical workspace set behaves exactly as before.

Also made the invariant observable rather than assumed:

  • workspace active --json gains a dataPlane object (unified, the shared workspaceId, per-plane IDs, and the names of any that diverge). The human output prints the Relaycast ID it previously omitted. --require-unified turns a divergence into a non-zero exit for setup doctors and supervisors.
  • node status prints the durable Workspace: ID next to the masked workspace key.

Acceptance

workspace active --json proves one data-plane workspace ID — run against the live default workspace:

$ agent-relay workspace active --json --require-unified
{
  "name": "default",
  "key": "rk_live_…9812",
  "cloudWorkspaceId": "50587328-441d-4acb-b8f3-dbe1b3c5de99",
  "relaycastWorkspaceId": "rw_7ccfea89",
  "relaycastApiKey": "rk_live_…9812",
  "relayfileWorkspaceId": "rw_7ccfea89",
  "relayauthWorkspaceId": "rw_7ccfea89",
  ...
  "dataPlane": {
    "unified": true,
    "workspaceId": "rw_7ccfea89",
    "planes": { "relaycast": "rw_7ccfea89", "relayfile": "rw_7ccfea89", "relayauth": "rw_7ccfea89" },
    "divergent": []
  }
}
$ echo $?
0
$ agent-relay workspace active --require-unified
Workspace: default
Cloud workspace ID: 50587328-441d-4acb-b8f3-dbe1b3c5de99
Relaycast workspace ID: rw_7ccfea89
Relayfile workspace ID: rw_7ccfea89
Relayauth workspace ID: rw_7ccfea89
Data-plane workspace ID: rw_7ccfea89 (unified)

node up uses the canonical workspace with no manual key copyingcore.test.tsup falls back to the machine-global canonical workspace when nothing else selects one, and up prefers the project pin over the machine-global canonical workspace.

Stop/start regression testpackages/cli/src/cli/lib/workspace-identity-restart.test.ts. Each start() is a fresh CLI process (new env, new deps) over one persistent checkout and AGENT_RELAY_HOME, which is what a stop/start looks like from the CLI. It models the two behaviors that decide the outcome: the broker joins RELAY_WORKSPACE_KEY when set and otherwise mints, and Relaycast returns the existing agent when a name is re-registered in a workspace it already belongs to.

  • keeps one workspace and one resident address across a restart
  • joins the canonical workspace on a first start with no project pin
  • pins the canonical workspace to the project so later starts resume it
  • explicit --workspace-key still wins
  • a second checkout shares the canonical workspace and the resident address
  • negative control: with no canonical workspace, a second checkout is a different node entirely

Status and logs never print credentialscore.test.tsstatus reports the durable workspace ID without leaking any credential asserts the raw workspace key, node token, ot_live_ tokens, and ?token=-style URLs are all absent. workspace.test.ts asserts the same for workspace active human output. The canonical-workspace fallback logs only its source, never the key.

Documentedspecs/workspace-identity.md covers the invariant, resolution order, how to prove it, secret handling, and migration for existing local nodes (including the one behavior change: a node with no project pin now joins the canonical workspace instead of minting, which moves its resident agents there).

Test evidence

$ npx vitest run packages/cloud/src/workspace-convergence.test.ts \
    packages/cli/src/cli/lib/workspace-identity-restart.test.ts \
    packages/cli/src/cli/commands/workspace.test.ts \
    packages/cli/src/cli/commands/core.test.ts \
    packages/cli/src/cli/commands/node.test.ts \
    packages/cli/src/cli/lib/broker-lifecycle.test.ts

 Test Files  6 passed (6)
      Tests  146 passed (146)

Full packages/cli + packages/cloud: 1095 passed, 20 skipped, 0 failed. npm run typecheck clean; npm run lint 0 errors (no new warnings).

One pre-existing environmental failure, integration-relayfile-contract.test.tsreplaces a stale v1 daemon only when the installed binary can serve v3, reproduces identically on a stashed clean tree — the sandbox can't invoke lsof. Unrelated to this change.

Note

node up's test harness now defaults AGENT_RELAY_HOME to an isolated temp dir. Without that, the new store fallback made tests read (and nearly print) whatever workspace the developer's own machine had active.

Do not merge — the principal owns the merge gate.

🤖 Generated with Claude Code

Review in cubic

A node started with no project-pinned workspace fell straight through to
the broker, which mints a brand-new messaging-only workspace. Nothing
errored: the node came up, the resident agent registered, and it looked
healthy — but it was a stranger in a different workspace with a new
address, so everything sent to its previous address went nowhere.

`node up` now resolves the machine-global canonical workspace
(`agent-relay workspace join|switch`) before letting the broker mint one.
Explicit `--workspace-key`, workspace env vars, and an existing project
pin all still win; the resolved key is pinned to the project afterwards,
so later starts resume it directly. A machine with no canonical
workspace set behaves exactly as before.

Also make the invariant observable:

- `workspace active` emits a `dataPlane` object proving Relaycast,
  Relayfile, and RelayAuth share one data-plane workspace ID, prints the
  Relaycast ID it used to omit, and gains `--require-unified` to turn a
  divergence into a non-zero exit.
- `node status` prints the durable `Workspace:` ID next to the masked
  key, so an operator can compare it across a restart.

Secrets stay out of both paths: the canonical-workspace fallback logs
only its source, never the key.

specs/workspace-identity.md documents the invariant, the resolution
order, and migration behavior for existing local nodes.

Refs AR-448

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@khaliqgant
khaliqgant requested a review from willwashburn as a code owner July 30, 2026 22:56
@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The PR adds workspace convergence reporting to workspace active, introduces strict divergence checking, and resolves startup workspace identity through explicit, project-pinned, or machine-global credentials. It also adds restart, fallback, redaction, CLI, documentation, and changelog coverage.

Changes

Workspace identity and convergence

Layer / File(s) Summary
Data-plane convergence reporting
packages/cloud/src/workspace-convergence.ts, packages/cloud/src/index.ts, packages/cli/src/cli/commands/workspace.ts, packages/cloud/src/workspace-convergence.test.ts, packages/cli/src/cli/commands/workspace.test.ts
Workspace IDs from Relaycast, Relayfile, and RelayAuth are compared, exposed through JSON and text output, and optionally enforced with --require-unified.
Canonical workspace startup and identity
packages/cloud/src/workspace-key.ts, packages/cli/src/cli/lib/broker-lifecycle.ts
Startup resolves explicit credentials first, then project-pinned credentials, then the machine-global canonical workspace; status displays the durable workspace ID while masking credentials.
Restart and precedence validation
packages/cli/src/cli/commands/core.test.ts, packages/cli/src/cli/lib/workspace-identity-restart.test.ts, specs/workspace-identity.md, CHANGELOG.md
Tests and documentation cover restart-stable identity, fallback precedence, explicit overrides, cross-checkout sharing, address stability, and credential redaction.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

Suggested labels: size:XL

Suggested reviewers: willwashburn

Poem

I’m a rabbit with a workspace key,
Keeping three bright planes in harmony.
Pinned or global, the node knows the way,
Secrets stay hidden night and day.
Restart the relay—identity stays! 🐇

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes making Relay workspace identity durable across node restarts.
Description check ✅ Passed The description thoroughly explains the bug, fix, acceptance criteria, testing, documentation, and credential-redaction behavior.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/ar-448-durable-workspace-identity

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 27494c8626

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +53 to +54
const [, reference] = entries[0]!;
const divergent = entries.filter(([, id]) => id !== reference).map(([plane]) => plane);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Select the majority ID before labeling divergent planes

When Relayfile and RelayAuth agree but Relaycast is the outlier, this always uses Relaycast as the reference and reports both agreeing planes as divergent. That makes the new diagnostic identify the wrong service(s) for the exact two-versus-one mismatch it is meant to troubleshoot; choose the majority ID as the reference before constructing divergent.

Useful? React with 👍 / 👎.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/cli/src/cli/lib/broker-lifecycle.ts (1)

1340-1343: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Trim mismatch with the later injection at Line 1407.

Here a whitespace-only --workspace-key is not treated as explicit (so the project pin or canonical key gets applied), but Line 1407 tests raw truthiness and then overwrites RELAY_WORKSPACE_KEY/RELAY_API_KEY with that blank value — the broker then mints a throwaway workspace, the exact drift this change removes. Normalize once and reuse.

🐛 Suggested fix
-  if (options.workspaceKey) {
+  const explicitWorkspaceKey = options.workspaceKey?.trim();
+  if (explicitWorkspaceKey) {
-    deps.env.RELAY_WORKSPACE_KEY = options.workspaceKey;
-    deps.env.RELAY_API_KEY = options.workspaceKey;
+    deps.env.RELAY_WORKSPACE_KEY = explicitWorkspaceKey;
+    deps.env.RELAY_API_KEY = explicitWorkspaceKey;
   }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/cli/lib/broker-lifecycle.ts` around lines 1340 - 1343,
Normalize the workspace key once in the broker lifecycle flow and reuse that
normalized value for both the explicit-key check and the later injection near
the existing workspace/API key assignment. Ensure whitespace-only
options.workspaceKey is treated as absent and cannot overwrite
RELAY_WORKSPACE_KEY or RELAY_API_KEY, while preserving valid explicit keys and
the existing project/canonical fallback behavior.
🧹 Nitpick comments (3)
packages/cloud/src/workspace-convergence.test.ts (1)

33-43: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

This test doesn't exercise its stated claim.

The input contains no cloud workspace ID, so this is identical to the first test and would still pass if cloudWorkspaceId were included in the comparison. To actually pin the exclusion, pass a descriptor carrying a divergent cloudWorkspaceId and assert unified is still true.

♻️ Suggested change
     expect(
       describeDataPlaneConvergence({
+        // Extra field is ignored by the Pick-typed parameter at runtime.
+        cloudWorkspaceId: '50587328-441d-4acb-b8f3-dbe1b3c5de99',
         relaycastWorkspaceId: 'rw_same',
         relayfileWorkspaceId: 'rw_same',
         relayauthWorkspaceId: 'rw_same',
-      }).unified
+      } as Parameters<typeof describeDataPlaneConvergence>[0]).unified
     ).toBe(true);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cloud/src/workspace-convergence.test.ts` around lines 33 - 43,
Update the test case around describeDataPlaneConvergence to include a divergent
cloudWorkspaceId in the descriptor while keeping all data-plane workspace IDs
equal, then assert unified remains true. This must distinguish the cloud
control-plane ID from the data-plane convergence comparison rather than
duplicating the preceding test.
packages/cli/src/cli/commands/workspace.test.ts (1)

4-17: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider spreading ...actual instead of enumerating exports.

The factory returns only six names, so any other @agent-relay/cloud export reached through this module graph resolves to undefined and fails at import time rather than in a readable assertion. Spreading the real module and overriding just the mocked functions keeps the mock from drifting as the barrel grows.

♻️ Suggested change
   const actual = await importOriginal<typeof import('`@agent-relay/cloud`')>();
   return {
-    describeDataPlaneConvergence: actual.describeDataPlaneConvergence,
-    formatDataPlaneDivergence: actual.formatDataPlaneDivergence,
+    ...actual,
     readWorkspaceStore: vi.fn(() => ({ workspaces: {} })),
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/cli/commands/workspace.test.ts` around lines 4 - 17, Update
the `@agent-relay/cloud` mock factory to spread the imported actual module
exports, then override only the workspace-related functions that must remain
mocked, including readWorkspaceStore, resolveActiveWorkspace, setWorkspaceKey,
and switchWorkspace. Preserve the real convergence helper implementations while
ensuring all other barrel exports remain available.
packages/cli/src/cli/lib/workspace-identity-restart.test.ts (1)

287-335: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Duplicated dependency harness.

This deps object is a near-copy of the one inside createMachine.start() (Lines 136-182). Extending createMachine with an options override (e.g. start({ workspaceKey })) would let this test reuse the existing harness and keep the two copies from drifting.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/cli/lib/workspace-identity-restart.test.ts` around lines 287
- 335, Refactor the test dependency setup so the harness created inside
createMachine.start() can accept an options override, including workspaceKey.
Update the test at the shown deps object to call start({ workspaceKey }) or the
equivalent override instead of duplicating the full CoreDependencies object,
while preserving the existing test-specific workspace identity behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@specs/workspace-identity.md`:
- Line 58: Add language identifiers to the fenced code blocks in
workspace-identity.md: use json for the workspace active --json payload and
console or bash for shell transcripts, including the blocks around lines 58,
89-91, 96-102, and 143-148.

---

Outside diff comments:
In `@packages/cli/src/cli/lib/broker-lifecycle.ts`:
- Around line 1340-1343: Normalize the workspace key once in the broker
lifecycle flow and reuse that normalized value for both the explicit-key check
and the later injection near the existing workspace/API key assignment. Ensure
whitespace-only options.workspaceKey is treated as absent and cannot overwrite
RELAY_WORKSPACE_KEY or RELAY_API_KEY, while preserving valid explicit keys and
the existing project/canonical fallback behavior.

---

Nitpick comments:
In `@packages/cli/src/cli/commands/workspace.test.ts`:
- Around line 4-17: Update the `@agent-relay/cloud` mock factory to spread the
imported actual module exports, then override only the workspace-related
functions that must remain mocked, including readWorkspaceStore,
resolveActiveWorkspace, setWorkspaceKey, and switchWorkspace. Preserve the real
convergence helper implementations while ensuring all other barrel exports
remain available.

In `@packages/cli/src/cli/lib/workspace-identity-restart.test.ts`:
- Around line 287-335: Refactor the test dependency setup so the harness created
inside createMachine.start() can accept an options override, including
workspaceKey. Update the test at the shown deps object to call start({
workspaceKey }) or the equivalent override instead of duplicating the full
CoreDependencies object, while preserving the existing test-specific workspace
identity behavior.

In `@packages/cloud/src/workspace-convergence.test.ts`:
- Around line 33-43: Update the test case around describeDataPlaneConvergence to
include a divergent cloudWorkspaceId in the descriptor while keeping all
data-plane workspace IDs equal, then assert unified remains true. This must
distinguish the cloud control-plane ID from the data-plane convergence
comparison rather than duplicating the preceding test.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e854097d-c05b-45ed-86e5-54c53a875e38

📥 Commits

Reviewing files that changed from the base of the PR and between 5677683 and 00dab6e.

📒 Files selected for processing (11)
  • CHANGELOG.md
  • packages/cli/src/cli/commands/core.test.ts
  • packages/cli/src/cli/commands/workspace.test.ts
  • packages/cli/src/cli/commands/workspace.ts
  • packages/cli/src/cli/lib/broker-lifecycle.ts
  • packages/cli/src/cli/lib/workspace-identity-restart.test.ts
  • packages/cloud/src/index.ts
  • packages/cloud/src/workspace-convergence.test.ts
  • packages/cloud/src/workspace-convergence.ts
  • packages/cloud/src/workspace-key.ts
  • specs/workspace-identity.md


## 3. Proving the invariant

```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add languages to the fenced code blocks.

markdownlint flags MD040 on Lines 58, 89, 96, and 143. Use json for the workspace active --json payload and console (or bash) for the shell transcripts; console also satisfies MD014 for the $-prefixed commands.

Also applies to: 89-91, 96-102, 143-148

🧰 Tools
🪛 markdownlint-cli2 (0.23.1)

[warning] 58-58: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@specs/workspace-identity.md` at line 58, Add language identifiers to the
fenced code blocks in workspace-identity.md: use json for the workspace active
--json payload and console or bash for shell transcripts, including the blocks
around lines 58, 89-91, 96-102, and 143-148.

Source: Linters/SAST tools

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant